Wireless communication method, wireless terminal, and program for wireless terminal

Abstract

To relate to a radio communication method for obtaining high communication efficiency in a network including a plurality of radio terminals and a plurality of radio base stations, or the like.SOLUTION: A DQN includes input layers and output layers corresponding to a plurality of radio base stations, and hidden layers including two or more layers and located between them. The DQN functions, for input of a state, as a value function to output values of actions to be performed by radio terminals in the state. A radio communication method includes: (Step 128) limiting parameters to be updated only to the parameters associated with a post stage of the hidden layer of the DQN; (Step 108) providing a connection state Sk(t) to the input layer of the DQN and (Step 110) selecting an optimal action ak(t); and (Step 118) updating the parameters of the DQN on the basis of a reward Γk(t) (Step 114) calculated for the action ak(t). At the time, only the parameters related to the post stage of the hidden layer are updated.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a wireless communication method, a wireless terminal, and a program for a wireless terminal, and in particular to a wireless communication method, a wireless terminal, and a program for a wireless terminal that are suitable for achieving high communication efficiency in a network including multiple wireless terminals and multiple wireless base stations, etc. [Background technology]

[0002] With the development of wireless communication systems, heterogeneous networks using multi-band, multi-access systems are reaching the stage of practical application. In cellular communications, the fifth generation mobile communications (5G) has been put into practical use. 5G communication systems use a wide range of frequencies, from sub-1 GHz bands to millimeter wave bands. In addition, an environment is being developed in which cells of various sizes, from small cells to macro cells, are provided in an overlapping manner.

[0003] Another typical wireless access system, wireless LAN, uses radio frequencies in the 2.4 / 5 / 60 GHz bands, and the use of the 6 GHz band is also being considered. Wireless terminals such as smartphones are generally provided with interfaces (IFs) for accessing both cellular and wireless LAN. Each IF is configured to support multiple frequency bands.

[0004] Wireless terminals generally select a wireless base station or access point (AP) to connect to from multiple frequency bands and access methods to establish communication. With functions such as dual connectivity, one wireless terminal may integrate and use multiple wireless base stations. In such a heterogeneous environment, it is important to control and optimize, across the entire system, which IF a wireless terminal uses and which base station it selects, in order to effectively use system resources.

[0005] Regarding technology for effectively utilizing system resources in a heterogeneous environment, for example, Non-Patent Document 1 and Non-Patent Document 2 below disclose a method for optimizing connections between wireless terminals and APs by reinforcement learning. More specifically, Non-Patent Document 2 discloses a technology for optimizing connections between wireless terminals using multiple IFs and multiple APs by using a method of Deep Q-Learning or Double Deep Q-Learning.

[0006] In addition, the following non-patent document 3 discloses Deep Q-Network (DQN), a method for handling high-dimensional inputs using reinforcement learning, and the following non-patent document 4 discloses Double Deep Q-Network (DDQN). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] “Reinforcement Learning-aided Distributed User-to-Access Points Association in Interfering Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Hirantha Abeysekera, Yasushi Takatori, IEEE Globecom, Waikoloa, Hawaii, USA, Dec. 2019 [Non-Patent Document 2] "Deep Reinforcement Learning-based User Association in Sub6GHz / mmWave Integrated Networks", Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Kenichi Kawamura, Takatsune Moriyama, Hirantha Abeysekera and Yasushi Takatori, in IEEE CCNC, Jan. 2021 [Non-Patent Document 3] "Human-level control through deep reinforcement learning," Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, Nature. 2015 Feb 26; Vol.518(7540): P529-533. doi: 10.1038 / nature14236 [Non-Patent Document 4] "Deep Reinforcement Learning with Double Q-Learning," H. Van Hasselt, A. Guez, and D. Silver, in Thirtieth AAAI conference on artificial intelligence, 2016 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] In a heterogeneous multi-band / multi-access environment, mobile terminals such as smartphones and tablets may be used as wireless terminals. The communication environment surrounding the mobile terminal changes as the mobile terminal moves. In addition, when there are many wireless terminals in the same area, the change in the request from each wireless terminal affects the load of the wireless terminals and wireless base stations belonging to that area. As a result, the communication environment surrounding each wireless terminal changes significantly depending on the state of other wireless terminals. For these reasons, the optimal solution regarding which wireless base station and which IF a wireless terminal placed in a heterogeneous environment should connect to may not be a stable one, but a dynamic one that changes drastically.

[0009] When trying to optimize the connection between a wireless terminal and a wireless base station using reinforcement learning such as DQN or DDQN, if the communication environment changes, it is necessary to update the reinforcement learning parameters to adapt to the changed environment. In an environment where the communication environment changes frequently, such updates need to be performed frequently.

[0010] In an environment in which a wireless terminal is connected to many wireless base stations using multiple IFs, the more the number of wireless base stations and the number of IFs increases, the higher the load caused by the above-mentioned updates.Since the processing capacity of a wireless terminal is limited, it becomes difficult for the wireless terminal to keep up with the high frequency of parameter updates as the number of wireless base stations and the number of available IFs increase.

[0011] The present disclosure has been made in light of the above-mentioned problems, and aims to provide a wireless communication method, a wireless terminal, and a program for a wireless terminal that can reduce the computational load of updating parameters required for optimization in a network that includes multiple wireless terminals and multiple base stations, etc. [Means for solving the problem]

[0012] In order to achieve the above object, a first aspect of the present disclosure is a wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, the method comprising: A step of preparing a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located therebetween, and functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state; detecting a state of connection between the plurality of wireless base stations and the wireless terminal; providing the sensed state to the input layer and determining, according to a result appearing at the output layer, an action to be taken by the wireless terminal under the sensed state; transmitting a request to said plurality of radio base stations corresponding to an action to be taken; obtaining responses to the request from a plurality of wireless base stations; calculating a reward for the behavior based on the response; updating a value for adopting the behavior under the state based on the reward; learning parameters of the value function based on the updated values; limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer; It is preferable that the present invention includes the following:

[0013] A second aspect of the present disclosure is a wireless terminal having a function of connecting to a plurality of wireless base stations, The neural network includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and includes a memory unit storing a neural network that functions as a value function that outputs, in response to an input of a state, a value of each of the actions that the wireless terminal can take under the state, and a CPU; The CPU, A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses to the request from a plurality of wireless base stations; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer; It is advisable to carry out the following:

[0014] A third aspect of the present disclosure is a program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, comprising: The CPU of the wireless terminal has a process of reading from a memory unit a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and that functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state; A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses to the request from a plurality of wireless base stations; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer; It is preferable that the program include a program for executing the above. Effect of the Invention

[0015] According to the present disclosure, in a network including multiple wireless terminals and multiple base stations, the parameter update range of a neural network that determines the behavior of a wireless terminal can be limited to parameters linked to layers subsequent to a hidden layer. Therefore, according to the present disclosure, the calculation load associated with parameter update can be reduced compared to when all parameters are updated. As a result, even if the number of wireless base stations or the number of wireless communication interfaces used therein increases, it becomes possible to update parameters frequently, and it becomes possible to always effectively utilize system resources. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining an overall configuration of a wireless communication system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram for explaining an example of a connection state between a wireless terminal and a wireless base station in the first embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram illustrating a state in which a wireless terminal moves in a heterogeneous environment in the first embodiment of the present disclosure. [Figure 4] FIG. 2 is a block diagram for explaining a functional configuration of a radio base station used in the first embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram for explaining a functional configuration of a wireless terminal used in the first embodiment of the present disclosure. [Figure 6] 2 is a diagram for explaining a hardware configuration of a wireless terminal used in the first embodiment of the present disclosure. FIG. [Figure 7]FIG. 1 is a diagram for explaining an overview of DQN used in the first embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram for explaining the content of a process related to updating a DQN used in the first embodiment of the present disclosure. [Figure 9] FIG. 2 is a diagram for explaining initialization of DQN parameters used in the first embodiment of the present disclosure. [Figure 10] FIG. 2 is a diagram for explaining partial update of parameters of DQN used in the first embodiment of the present disclosure. [Figure 11] 11 is a flowchart illustrating a process flow when a wireless terminal according to the first embodiment of the present disclosure initializes a DQN parameter. [Figure 12] 11 is a flowchart for explaining a processing flow when a wireless terminal according to the first embodiment of the present disclosure partially updates a parameter of DQN. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Embodiment 1 [Overall configuration of the first embodiment] Fig. 1 is a diagram for explaining an overall configuration of a wireless communication system 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the wireless communication system 10 includes a wired communication network 12. A plurality of wireless base stations 16 are connected to the network 12 via a switch 14. Each wireless base station 16 can communicate with another wireless base station 16 via the network 12 and via a wireless path.

[0018] 1 includes a plurality of wireless terminals 18. In this embodiment, the wireless terminals 18 include mobile terminals such as smartphones or tablet terminals.

[0019] Each of the wireless terminals 18 includes a plurality of sets of applications and IFs for wireless communication. For example, the wireless terminals 18 include an application and IF for communication at 1 GHz or less, an application and IF for communication in the millimeter wave band, and an application and IF for communication in the 2.4 / 5 / 60 GHz bands used in wireless LAN.

[0020] The above-mentioned wireless base station 16 also has corresponding IFs, etc. Each wireless terminal 18 can wirelessly communicate with one or more wireless base stations 16 via the above-mentioned applications and IFs. This allows the wireless communication system 10 of the present embodiment to function as a heterogeneous network using multi-band multi-access.

[0021] In addition, the wireless communication system 10 of the present embodiment is compatible with wireless LAN access points (APs) in addition to wireless base stations for cellular communication installed by telecommunications carriers. For the sake of convenience, the following description will be given assuming that the "wireless base station 16" encompasses both.

[0022] [Features of the first embodiment] Next, features of the wireless terminal 18 in this embodiment will be described with reference to Figures 2 to 12. In the following description, it is assumed that the wireless base station 16 and the wireless terminal 18 are provided with an interface IF1 corresponding to communication in a first frequency band and an interface IF2 corresponding to communication in a second frequency band. IF1 is an interface corresponding to a frequency band of, for example, 1 GHz or less, and IF2 is an interface corresponding to frequencies in the millimeter wave band, for example.

[0023] In the following description, when it is necessary to specify a specific wireless terminal 18 among a plurality of wireless terminals, the symbol k will be used to indicate "wireless terminal 18." k Furthermore, when it is necessary to distinguish between the multiple wireless base stations 16, the reference numerals with suffixes, such as 16-1 and 16-2, will be used.

[0024] FIG. 2 shows a wireless terminal 18 k The figure shows that the wireless terminal 18 belongs to two wireless base stations 16-1 and 16-2. k is the upper limit B determined by the hardware constraints in the communication environment surrounding oneself. UB The wireless base stations 16 can be treated as communication targets. BU Of the wireless base stations 16, wireless terminals 18 k is connected to wireless base stations 16-1 and 16-2.

[0025] Wireless terminal 18 k As described above, the wireless base station 16 has two interfaces, IF1 and IF2, which correspond to different bands. In the example shown in FIG. k However, a state is created in which the terminal belongs to the first wireless base station 16-1 at IF1 and belongs to the second wireless base station 16-2 at IF2.

[0026] FIG. 3 shows the results of a multi-band, multi-access heterogeneous environment in which a wireless terminal 18 k First, the wireless terminal 18 is moving. k In this environment, the wireless terminal 18 is located at the leftmost position in the figure. k However, if the state of belonging to the wireless base stations 16-1 and 16-2 is optimal, the wireless terminal 18 k It is desirable for a connection request from to be issued so that this state is realized.

[0027] Here, the wireless terminal 18 k However, if the wireless terminal 18 moves from the leftmost position to the center of the figure, and then to the right side of the figure, the optimal connection combination changes from the initial combination. k This also occurs due to the movement of other wireless terminals 18 (not shown) that belong to the same environment, changes in the requested data rates, etc. k When the communication environment changes, it is desirable to issue a connection request with a combination that is optimal for the changed environment.

[0028] In this embodiment, the wireless terminal 18 determines the wireless base station 16 and IF to which it desires to connect by using a Deep Q-Network (DQN) or a Double Deep Q-Network (DDQN). For this reason, it is desirable that the DQN or DDQN used here quickly converges to an optimal state for a stable communication environment, and quickly follows environmental changes in a dynamic communication environment.

[0029] 4 is a block diagram for functionally explaining an example of wireless base station 16. Wireless base station 16 includes an antenna unit 20. Antenna unit 20 is used by wireless base station 16 to transmit and receive wireless signals to and from other wireless base stations 16 and surrounding wireless terminals 18.

[0030] The antenna unit 20 is connected to the wireless communication unit 22. The wireless communication unit 22 includes an RF (Radio Frequency) unit 24 that processes high-frequency signals, and a MAC (Media Access Control) function unit 26 that has a function of media access control in wireless communication. The wireless communication unit 22 is configured to be able to realize both a function corresponding to a plurality of frequency bands, specifically, a function as an IF1 corresponding to 1 GHz or less, and a function as an IF2 corresponding to millimeter wave band frequencies.

[0031] The wireless base station 16 also includes a quality measurement unit 28. The quality measurement unit 28 includes a wireless environment measurement unit 30 and a traffic measurement unit 32. The wireless environment measurement unit 30 acquires information related to the wireless communication environment surrounding the wireless base station 16. The traffic measurement unit 32 acquires information related to the traffic between the wireless base station 16 and the wireless terminal 18. The quality measurement unit 28 calculates information on the communication quality between the wireless base station 16 and the wireless terminal 18 based on the information. The communication quality information includes, for example, the received power from the wireless terminal 18, the required data rate of the wireless terminal 18, the actual transmission rate, and the band usage rate in the wireless base station 16.

[0032] The wireless base station 16 includes a communication IF unit 34. The communication IF unit 34 is used for wired communication via the network 12. The wireless base station 16 can acquire communication quality information acquired by other wireless base stations 16 through wired communication via the communication IF unit 34 or through wireless communication via the wireless communication unit 22 and the antenna unit 20. The communication quality information acquired from other wireless base stations 16 is managed in a unified manner in the quality measurement unit 28 together with communication quality information acquired by the wireless base station 16 itself.

[0033] The wireless base station 16 includes an information notification unit 36. The information notification unit 36 ​​can read out communication quality information aggregated in the quality measurement unit 28. The information notification unit 36 ​​can provide the read communication quality information to the communication IF unit 34 and the wireless communication unit 22. This allows the wireless base station 16 to provide its own communication quality information to other wireless base stations 16 via a wired or wireless path.

[0034] The wireless base station 16 receives an association request from a wireless terminal 18 located in the vicinity via the wireless communication unit 22 and the antenna unit 20. The association request is processed by a request information evaluation unit 38. The request information evaluation unit 38 determines whether or not association is possible based on the bandwidth and data rate requested by the wireless terminal 18, as well as its own load state, etc.

[0035] The result of the determination by the request information evaluation unit 38 is sent to a request response notification unit 40 and an association information recording unit 42. The request response notification unit 40 provides the result of the determination as to whether association is possible or not to the wireless terminal 18 that issued the association request, via the wireless communication unit 22 and the antenna unit 20. The association information recording unit 42 updates and records information regarding which wireless terminal 18 is associated with the wireless base station 16 based on the result of the determination.

[0036] 5 is a block diagram for functionally explaining an example of wireless terminal 18. Wireless terminal 18 includes an antenna unit 44 and a wireless communication unit 46. Wireless communication unit 46 includes an RF unit 24 and a MAC function unit 50. These functions are substantially the same as the functions of the corresponding blocks included in wireless base station 16, and therefore description thereof will be omitted here.

[0037] The wireless terminal 18 includes a control information calculation unit 52. The control information calculation unit 52 acquires communication quality information from the wireless base station 16 to which the wireless terminal 18 belongs via the wireless communication unit 46 and the antenna unit 44. Then, based on the communication quality information, the control information calculation unit 52 determines an optimal action a to which wireless base station 16 each of the IFs for wireless communication should be connected by DQN or DDQN. k The process determines whether the target object is a target object or not, and performs learning for that purpose.

[0038] The control information calculation unit 52 includes a state calculation unit 54. The state calculation unit 54 calculates a state S to be provided to the input layer of the DQN or DDQN based on the communication quality information. k (t) is calculated.

[0039] The control information calculation unit 52 also includes a reward calculation unit 56. The reward calculation unit 56 calculates the reward for the adopted action a based on the communication quality information. k Reward obtained by k Calculate.

[0040] The control information calculation unit 52 further includes a DQN / DDQN update unit 58. The DQN / DDQN update unit 58 updates a parameter θ of the DQN or DDQN used in reinforcement learning. More specifically, the weighting coefficient and bias value (hereinafter, referred to as "weighting coefficient, etc.") that determine the characteristics of the DQN or DDQN as a value function are updated.

[0041] The DQN or DDQN updated by the DQN / DDQN update unit 58 is used in the attribute calculation unit 60. That is, the attribute calculation unit 60 determines the action a to be taken this time according to the updated DQN or DDQN. k, that is, it calculates which IF should be assigned to which wireless base station 16.

[0042] DQN and DDQN input a state S k Given the above, the possible actions that the wireless terminal 18 can take are k For each of these, the value of adoption is expressed in the output layer. Therefore, if the action with the largest value is selected, the action considered optimal at the current learning stage can be selected. However, the next action a k If is always selected based on the above values, it is possible that one may become trapped in a local solution chosen by chance, and the truly optimal action may never be selected.

[0043] In order to avoid such a situation, the destination calculation unit 60 performs the action a according to the ε-greedy method. k For this reason, the destination calculation unit 60 randomly calculates the next action a with a probability ε regardless of DQN or DDQN. k Also, with probability 1-ε, the action a that DQN or DDQN judges to be optimal is selected. k is selected.

[0044] Furthermore, the destination calculation unit 60 performs a decay process on ε every time learning is repeated. The possibility of falling into a local solution selected by chance is large in the early stages of learning and decreases as learning progresses. For this reason, when the behavior a is randomly selected, k The need to select action a becomes smaller as learning progresses. When ε is decayed, the need to select action a becomes smaller. k Therefore, according to the destination calculation unit 60 of this embodiment, learning can be efficiently carried out and the parameter θ can be converged to an optimal value in a short period of time.

[0045] By the way, transfer learning is known as a method for learning parameters of DQN or DDQN. Transfer learning is a method for learning by replacing the final output layer based on a trained model. In transfer learning, parameters close to the input layer are not changed from the trained values, and only the parameters for the replaced output layer and the parameters of several layers before that are retrained with original data. With transfer learning, only some parameters are subject to update, so the computational load associated with updating can be reduced compared to updating all parameters.

[0046] In the wireless communication system 10 of the present embodiment, as described above, it is desired that the DQN or DDQN quickly follows environmental changes under a dynamic communication environment. To achieve this, it is necessary to frequently update the DQN or DDQN. And, in order to frequently repeat the update, it is necessary to sufficiently reduce the calculation load associated with the update.

[0047] In this embodiment, under the above viewpoint, the idea of ​​transfer learning is applied to the update of DQN or DDQN to perform partial update of parameters. In other words, after the initialization of parameters is completed, among the hidden layers included in DQN or DDQN, the parameters corresponding to the previous stage are fixed, and only the parameters corresponding to the subsequent stage are updated. According to this method, the calculation load associated with the update of DQN or DDQN can be significantly reduced compared to the case where all parameters are to be updated.

[0048] In addition, as described above, the wireless terminal 18 in this embodiment is configured to use the B UB The wireless base stations 16 are treated as communication targets. UB When updating the DQN or DDQN by communicating with the wireless base station 16, the two IFs and B UB The number of possible combinations between the two IFs of each wireless base station 16 is the size of the state space.

[0049] However, B UB Some of these wireless base stations 16 have weak signal strength between them and the wireless terminal 18, and are not actually targets for belonging. If such wireless base stations 16 are excluded from the belonging destinations, the size of the state space to be considered in updating the DQN or DDQN is reduced, and the computational load associated with the update can be reduced.

[0050] To achieve the above functions, the wireless terminal 18 is provided with an environmental change detection unit 62 and an update range control unit 64. The environmental change detection unit 62 performs the following three functions. 1. Based on the signal received from the surrounding wireless base station 16, the received signal strength indicator (RSSI) indicating the channel quality is measured. 2. Select Bmax wireless base stations 16 in descending order of RSSI strength. UB It is a smaller number. 3. When the combination of wireless terminals 18 included in Bmax or the combination of IFs available thereto changes, a change in the environment is recognized.

[0051] The update range control unit 64 performs the following three functions. 1. In the stage of initializing the parameter θ, the DQN / DDQN update unit 58 is instructed to update all of the hidden layers. At the same time, B UB The reward calculation unit 56 and the DQN / DDQN update unit 58 are instructed to treat all of these wireless base stations 6 as active targets. 2. After the initialization is completed, instruct the DQN / DDQN update unit 58 to update only the latter stages of the hidden layer. At the same time, instruct the reward calculation unit 56 and the DQN / DDQN update unit 58 to treat only the Bmax wireless base stations 16 with large RSSI as active targets. 3. When the environmental change detection unit 62 recognizes an environmental change, it instructs the reward calculation unit 56 and the DQN / DDQN update unit 58 to save the learned parameters and change the active targets to the new Bmax wireless base stations 16.

[0052] The reward calculation unit 56 and the DQN / DDQN update unit 58 switch the update method for the parameter θ according to an instruction from the update range control unit 64. As a result, after the initialization is completed, the load associated with updating the parameter θ is sufficiently reduced.

[0053] In this embodiment, the wireless terminal 18 includes a memory unit 66. The memory unit 66 stores the following information: k Store as (t). 1. State S provided to the input layer of DQN or DDQN in this learning k (t), 2. That state S k Action taken in response to (t) k (t), 3. The action a k Reward Γ obtained by (t) k (t), 4. The action a k State S after transition due to (t) k (t+1).

[0054] The wireless terminal 18 also includes an attribute request unit 68. The attribute request unit 68 receives the action a from the attribute calculation unit 60. k Receive the action a k For example, a request is generated to connect the wireless terminal 18 to the wireless base station 16-1 at IF1, or to connect the wireless terminal 18 to the wireless base station 16-2 at IF2. The generated request is transmitted to one or more wireless base stations 16 via the wireless communication unit 46 and the antenna unit 20.

[0055] The wireless terminal 18 includes an associated destination recording unit 70. The information received from the wireless base station 16 includes a response to the request as to whether it is OK or not. For example, the information includes a response to allow a new connection, or a response to allow an update of a connection. The associated destination recording unit 70 updates and records information about the associated wireless base station 16 based on these responses.

[0056] 6 is a diagram for explaining the hardware configuration of the wireless terminal 18. Here, a case will be explained where the wireless terminal 18 is a smartphone. However, the wireless terminal 18 is not limited to a smartphone, and may be a tablet terminal or a general-purpose computer system.

[0057] The wireless terminal 18 includes a CPU (Central Processing Unit) 72. The CPU 72 is connected to various hardware elements included in the wireless terminal 18 via a bus wiring 74. To the bus wiring 74, various memory devices such as a ROM (Read Only Memory) 76, a RAM (Random Access Memory) 78, and a storage 80 are connected.

[0058] The storage 80 includes a non-volatile memory such as a flash ROM. Programs and data required for executing various applications are stored in the storage 80. The memory unit 66 and the destination recording unit 70 described above are realized by the storage 80. The CPU 72 also performs processing according to the programs stored in the storage 80, thereby realizing functions of the control information calculation unit 52, the environmental change detection unit 62, the update range control unit 64, and the like shown in FIG. 5 .

[0059] The wireless terminal 18 includes a communication interface 82. The communication interface 82 corresponds to the antenna unit 44 and the wireless communication unit 46. The wireless terminal 18 further includes an operation unit 84 and a display unit 86. The operation unit 84 includes a touch panel integrated with the display unit 86, in addition to various mechanical switch mechanisms and electronic sensor mechanisms.

[0060] [Overview of DQN] FIG. 7 is a diagram for explaining an overview of DQN. Hereinafter, with reference to FIG. 7, the wireless terminal 18 kThe outline of DQN, which is an example of deep reinforcement learning used by the present embodiment, will be described. The DQN used in this embodiment has an input layer, a hidden layer, and an output layer. In addition, the DQN has a parameter θ i t The characteristic of DQN as a value function is the parameter θ i t is determined by.

[0061] In FIG. 7, the input layer of DQN consists of M nodes S1 to S M These nodes include wireless terminals 18 k Status S k The elements that make up the state S are input. k The elements include, for example, the following elements: 1. Bandwidth utilization rate φb of wireless base station 16-b, 2. Wireless terminal 18 k The required data rate R of the interface f kf , 3. Wireless terminal 18 k and the wireless base station 16-b bkf (1 if it belongs, 0 if not), 4. Wireless terminal 18 k A request for attribution to the wireless base station 16-b of the interface f of bkf (1 if requested, 0 if not), 5. Wireless terminal 18 k The transmission rate r established between the interface f and the wireless base station 16-b bkf .

[0062] The number of hidden layers and the number of nodes are arbitrary. However, in this embodiment, the number of hidden layers is two or more. The output layer is the wireless terminal 18. k But in state S k Actions that can be taken under a i The number of nodes is equal to the total number of actions in the i Some examples are shown below. ·Only IF1 is assigned to radio base station 16-1. · Both IF1 and IF2 are associated with radio base station 16-1. ·Only IF2 is assigned to radio base station 16-1. · IF1 is associated with the wireless base station 16-1, and IF2 is associated with the wireless base station 16-2. · · ·

[0063] Each node in the output layer is connected to a wireless terminal 18. k Possible action a i Q value for each of (i=1~N) = Q(S k ,a i ;θ i t ) is expressed. The Q value is expressed by the parameter θ i t Action a calculated by DQN under i This is the value of adopting it.

[0064] In the method using DQN, the largest Q value is selected from the Q values ​​that appear in the output layer, and the action a that produces that Q value is i (t) is the optimal action at the current time t k According to DQN, in this way, at each learning stage, the optimal action at that time t is determined as a k (t) can be determined.

[0065] FIG. 8 shows the wireless terminal 18 k The control information calculation unit 52 calculates the parameters θ of DQN. i FIG. 8 is a diagram for explaining the flow of updating the action a k For convenience, (t) is shown after DQN / DDQN, but action a k As mentioned above, (t) is determined by DQN or DDQN only when the probability is 1-ε. And, under the probability ε, the action a k (t) is determined randomly. Parameter θ i The experience gained in both cases is k (t) is reflected.

[0066] Wireless terminal 18k So, action a k After (t) is decided, the action a k Reward Γ for (t) k (t) is calculated. Reward Γ k (t) is calculated, for example, by the following formula (1).

[0067]

number

[0068] In the above formula (1), the first term on the right side is a point-adding item, and the second term is a point-deducting item. 1k , ω 2k is a coefficient that determines the allocation ratio for them. Also, ν is an identifier for distinguishing between the sub-1 GHz term that IF1 should handle and the millimeter wave (mW) term that IF2 should handle.

[0069] Furthermore, in the above formula (1), c in the first term on the right side 1k ν (t) is the required data rate R kf A transmission rate r that satisfies bkf This means the amount of points to be added when (t) is obtained. The value is calculated using the following formula (2).

[0070]

number

[0071] The above formula (2) is expressed as follows: bkf (t) bkf (t) / W ν This means that the following is given. However, the W in the last term ν is a factor used to offset and normalize the difference between the (sub) and (mW) frequency bands. 1. Wireless terminal 18 k The interface f of the bkf (t)=1). 2. Wireless terminal 18k The wireless base station 16-b recognizes the ownership of the interface f (x bkf (t)=1), and the required data rate R kf Transmission rate exceeding r bkf (t) is obtained (r bkf (t)>Rkf).

[0072] In the above formula (1), c in the second term on the right side 2k ν (t) refers to the points deducted in the following two cases. 1. A request for attribution is made (a bkf (t)=1), the attribution was allowed, but (x bkf (t)=1), the required data rate R kf A transmission rate r that satisfies bkf If (t) is not obtained (r bkf (t) <R kf ). 2. A request for attribution was made (a bkf (t)=1), when the attribution is dropped from the wireless base station 16 (x bkf ν (t)=0).

[0073] Deduction in case of (sub)c 2k sub (t) can be expressed, for example, as the following formula (3). Also, the deduction point c for (mW) 2k mW (t) can be expressed, for example, as in the following equations (4) and (5).

[0074]

number

[0075]

number

[0076]

number

[0077] Thus, Action a k The required data rate R as a result of (t) kf A transmission rate r that satisfies bkf If (t) is obtained, the reward Γ k On the other hand, if the transmission rate r bkf If (t) is not obtained, reward Γ k (t) becomes a small value.

[0078] Reward Γ k Once (t) is calculated, the current experience e k (t) is stored in memory (step 88). k (t), selected action a k (t), the reward obtained Γ k (t), and the state after the transition S k (t+1) is experience e k It is stored in memory as (t).

[0079] If there is a time correlation in the sample sequence, learning may not proceed properly. To solve this problem, we set the parameter θ i When updating, the experience stored in memory k The memory samples are randomly taken from and processed in batches (step 90).

[0080] A randomly taken memory sample, i.e. (S k , a k , Γ k , S k Experiences that include k Using this, the parameters θ of DQN / DDQN i (Step 92). However, S k ´ is action a k Specifically, the process of step 92 proceeds as follows.

[0081] action a k Reward for Γ kOnce determined, the reward Γ k Based on the state S k Acting under a k The "new Q value" for selecting S k ,a k The new Q value is calculated, for example, as shown in the following formula (6).

[0082] Q(S k ,a k )←Γk+γ*maxQ(S k ´,a k ´) ···(6) where γ is the time discount rate.

[0083] The right-hand side of the above equation (6), Γ k +γ*maxQ(S k ´,a k ´) is (S k ,a k ) is the value that we want the "trained DQN" to calculate as the Q value for each combination. Therefore, the DQN parameter θ i is (S k ,a k ) input, the value is updated so that it appears in the output layer.

[0084] The following equation (7) represents the DQN parameter θ i The loss function L used to update k Shows.

[0085]

number

[0086] However, in the above equation (7), the first term on the right side, Q(S k ,a k ;θ i ) is the parameter θ i The main network with (S k ,a k ) is the Q value calculated for the combination of k ´,a k ´;θi ― ) is the parameter θ i ― If the target network has (S k ´,a k ´) is the Q value calculated for the combination.

[0087] When updating DQN, if the value of the network to be updated is calculated using the network being updated, learning may not be stable. For this reason, when updating DQN, in addition to the main network to be updated, a target network for calculating the update value is usually used. In this embodiment, the second term of equation (7), which corresponds to the new Q value, is calculated using the target network according to the above-mentioned usual method.

[0088] In step 92, the loss function L shown in the above equation (7) is used. k The DQN parameters θ are set so that i , i.e., the main network parameters θ i As a result, the DQN is updated to state S k For input, action a k The Q value of the model is trained to express the value on the right side of equation (6) above.

[0089] The target network is identical to the past main network except immediately after the update. The target network is then updated at a predetermined period (step 94). That is, when the time for updating arrives at the predetermined period, the parameter θ i ― is the parameter θ of the main network. i This completes the update of DQN.

[0090] In addition, the following equation (8) represents the loss function L used in step 92 when the DDQN method is used in this embodiment. k The principle of the update is the same as that of DQN, so we will not explain the details here.

[0091]

number

[0092] [Parameter θ i Initialization Figure 9 shows the parameter θ i This is a schematic diagram of the characteristics of DQN when initializing θ. Here, the number of hidden layers is set to "2". i The parameter θ includes weighting coefficients Wh1a1, etc., for the h1 node in the hidden layer to select the a1 node in the output layer. i When initializing B, as described above, the update range control unit 64 updates all hidden layers. UB This instructs the router to treat all of the wireless base stations 16 as active.

[0093] The input and output layers of DQN and DDQN are UB The number of nodes required to handle the wireless base stations 16 is provided. For example, the output layer includes two IFs and B UB The number of nodes a1 to a|A is equal to the total number of combinations between the IFs of each of the wireless base stations 16. k | is prepared. Parameter θ i In the initialization stage, all of these nodes are set to the active state. Then, all of the weighting coefficients W and other parameters set between the nodes of the DQN or DDQN are updated by learning.

[0094] In this case, the reward Γ k The calculation of (t) is also UB The calculation is performed for each of the wireless base stations 16 as the processing target. That is, as shown in the above formulas (1) to (4), the reward Γ k The calculation of (t) is performed for a set B of wireless base stations 16. Here, the set B is B UB The above equations (1) to (4) are calculated assuming that the wireless base stations 16 are included.

[0095] [Parameter θi [Partial update of Figure 10 shows the parameter θ i 1 is a diagram showing a schematic representation of the characteristics of DQN when the hidden layer is partially updated. After the initialization is completed, as described above, the update range control unit 64 instructs that only the latter hidden layer is to be updated. In response to this instruction, after the initialization is completed, only the latter layer 96 of the two hidden layers is to be updated, and the weighting coefficients W etc. related to the layer before the hidden layer are fixed. When the weighting coefficients W etc. related to the layer before the hidden layer are fixed, the parameters θ to be updated are i Therefore, the number of parameters θ i The calculation load involved in updating is reduced compared to the calculation load involved in the initialization.

[0096] The update range control unit 64 also instructs that, after the initialization is completed, only the Bmax wireless base stations 16 with large RSSI are to be treated as active. Nodes for handling wireless base stations 16 not included in Bmax are prepared in the input and output layers of the DQN and DDQN. At this stage, these nodes are set to an inactive state. For example, if the wireless base station 16-1 is not included in Bmax, the "wireless terminal 18 k IF1 of the wireless terminal 18 to the wireless base station 16-1, or k "connect both IF1 and IF2 of the same to the wireless base station 16-1" k The node for accommodating the request is made inactive.

[0097] In this case, the weighting coefficients W etc. related to the inactive nodes are fixed, and the parameters θ i Only the weighting coefficients W and other parameters related to inactive nodes are updated. If these are fixed, the parameters θ i Therefore, the number of parameters θ i The computational load involved in updating is further reduced compared to the computational load involved in the initialization.

[0098] Reward Γ k For the calculation of (t), the number of elements in set B is BUB By reducing the number of variables from Bmax to Bmax, the number of variables to be handled in the above equations (1) to (4) is reduced. k The computational load of (t) is also significantly reduced compared to the initialization case.

[0099] In this manner, in this embodiment, the parameter θ i After the initialization of the parameter θ i The full update of is switched to a partial update. According to the partial update, the optimal action a k It is possible to significantly reduce the computational load associated with the learning without compromising the characteristics of DQN or DDQN for deriving the above.

[0100] [Processing by the wireless terminal's CPU] 11 is a flowchart for explaining the flow of processing executed by the CPU 72 of the wireless terminal 18 to realize the above-mentioned functions. k This is executed when the application is started in a communication environment for the first time. Note that the following describes the case where DQN is used as the reinforcement learning method.

[0101] In the routine shown in FIG. 11, first, a state S to be provided to the input layer of DQN is calculated based on communication quality information etc. acquired from the surrounding wireless base station 16. k (t) is set (step 100).

[0102] Next, ε to be used in the current learning is set (step 102). For example, a value of about 0.1 is given to ε as an initial value, and thereafter, a decay process is performed each time this routine is repeated.

[0103] Next, a random number is generated that falls within the range of 0 to 1. Furthermore, it is determined whether the random number is equal to or smaller than ε (step 104).

[0104] If the random number ≦ ε is found to be true, the current state S k For (t), the next action is a k(t) is determined randomly (step 106).

[0105] On the other hand, if it is not determined in step 104 that the random number ≦ ε holds, the input layer of the DQN is assigned a state S k (t) is input (step 108). Then, the Q values ​​appearing in the output layer of the DQN are compared, and the action a with the largest Q value is selected. i (t) is the optimal action at the current time a k (t) is selected (step 110).

[0106] Next, the action a selected by the above process k Based on (t), an association request is sent to surrounding wireless base stations 16 (step 112).

[0107] When a response to a request is received, the selected action is taken based on the information contained in the response. k Reward Γ for (t) k (t) is calculated (step 114).

[0108] Next, the experience e acquired in this routine is stored in the RAM 78 constituting the memory unit 66. k (t), i.e., (S k (t), a k (t), Γ k (t), S k (t+1)) is stored (step 116).

[0109] After that, the DQN is updated by batch processing (step 118), and the routine ends. i The total number of is updated. The above process is carried out by using the parameter θ i This is repeated until the initialization is completed.

[0110] Figure 12 shows the parameter θ i12 is a flowchart for explaining the flow of processing executed by the CPU 72 for partial update of the above-mentioned 12. In addition, in Fig. 12, steps that are substantially the same as the steps shown in Fig. 11 are denoted by the same reference numerals, and the description thereof will be omitted or simplified.

[0111] In the routine shown in FIG. 12, first, Bmax wireless base stations 16 with superior RSSI are selected from the surrounding wireless base stations 16 (step 120).

[0112] Next, it is determined whether or not there has been a change in the selection result of the Bmax wireless base stations 16 (step 122). More specifically, it is determined whether or not there has been a change in the combination of communication interfaces available in the selected Bmax wireless base stations 16.

[0113] After the initialization is completed, if Bmax wireless base stations 16 are selected for the first time, it is determined that the above-mentioned "change" has occurred. Also, if the combination of Bmax wireless base stations 16 and available IFs selected in the current cycle is different from the combination selected in the previous cycle, it is determined that a "change" has occurred. In these cases, the parameter θ i are saved (step 124).

[0114] Next, among the nodes in the input / output layers, only the nodes related to possible actions are activated (step 126). More specifically, among the nodes in the input and output layers of the DQN, only those corresponding to possible actions for the selected Bmax wireless base stations 16 are activated, and other input / output nodes are deactivated.

[0115] If it is determined in step 120 that no change has occurred in the Bmax combinations, the processes in steps 124 and 126 are skipped. Then, following these processes, a process is executed in which only the latter layers of the hidden layers of the DQN are subject to update, and the earlier layers are excluded from the update target (step 128).

[0116] Thereafter, the processing from step 100 onwards is executed in the same manner as in the case shown in Fig. 11. In the routine shown in Fig. 12, the attenuated ε may be reset to its initial value in step 102 every time a change is detected in the Bmax combinations. When ε is reset to its initial value, the frequency of random searches is restored, so that the parameter θ i This makes it easier to keep up with changes in the communication environment.

[0117] 12, parameters related to the layer before the hidden layer are held at fixed values ​​in step 118. Then, parameters related to the layer after the hidden layer are updated as learning progresses.

[0118] Furthermore, in the routine shown in FIG. 12, in step 118, the parameter θ i are partially updated. In other words, only the weighting coefficients W etc. located between the active input / output nodes are updated, and other parameters are kept fixed.

[0119] According to the routine shown in FIG. k When the communication environment surrounding the base station 16 changes and the combination of the Bmax wireless base stations 16 or the combination of IFs available therein changes, the learned parameter θ i Then, the optimal action a is stored for each new Bmax wireless base station 16. k Parameter θ for deriving i A partial update is performed.

[0120] According to the above process, the wireless terminal 18 k Even if the number of wireless base stations 16 belonging to the surrounding environment increases and the number of IFs available therein increases, the DQN parameter θ iTherefore, according to this embodiment, it is possible to keep using system resources effectively in a heterogeneous environment with multi-band multi-access.

[0121] Incidentally, in the above-described first embodiment, the wireless terminal 18 and the wireless base station 16 use two IFs for wireless communication, but the present disclosure is not limited to this. For example, the number of IFs for wireless communication may be three or more. Furthermore, the number of IFs for wireless communication may be a mixture of IFs for licensed bands and IFs for unlicensed bands used for wireless LANs and the like. Alternatively, the number of IFs for wireless communication may be only one, and deep reinforcement learning such as DQN may be used to determine which wireless base station 16 the wireless terminal 18 should optimally request a connection to.

[0122] In the above-mentioned first embodiment, DQN or DDQN is used as a value function for determining an action. However, these are merely examples, and the present disclosure is not limited to these. Any function that can determine the value of each action that can be taken under a specific state can be used as a value function in the present disclosure.

[0123] In the above-described first embodiment, a technique for updating only the latter stages of the hidden layer and a technique for treating only Bmax wireless base stations 16 with strong signal strength as active are used in combination, but it is not essential to use them in combination. For example, UB Alternatively, all of the wireless base stations 16 may be treated as active, and only the layers subsequent to the hidden layer may be subject to updating, thereby reducing the calculation load.

[0124] In addition, in the above-mentioned embodiment 1, the case where the hidden layer is two layers is described, but the hidden layer is not limited to two layers, and may be two or more layers as described above. And, when the hidden layer is three or more layers, the layer to be updated is not limited to the last layer. In this case, as long as the layer immediately preceding the last layer is excluded, some layers in the latter stages may be updated. [Explanation of symbols]

[0125] 10. Wireless communication systems 16, 16-1, 16-2, 16-3 Radio base station 18, 18 k Wireless terminal 52 Control information calculation unit 60 Attribution Calculation Department 62 Environmental change detection unit 64 Update range control section Gamma k (t) Remuneration a k (t) action e k (t) Experience S k (t) Status S k (t+1) State after transition R kf Requested Data Rate r bkf Transmission rate θ i Parameters

Claims

1. A wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, comprising: preparing a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located therebetween, and functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state; detecting a state of connection between the plurality of wireless base stations and the wireless terminal; providing the sensed state to the input layer and determining, according to a result appearing at the output layer, a behavior to be adopted by the wireless terminal under the sensed state; transmitting a request to said plurality of radio base stations corresponding to an action to be taken; obtaining responses to the request from a plurality of wireless base stations; calculating a reward for the behavior based on the response; updating a value for adopting the behavior under the state based on the reward; learning parameters of the value function based on the updated values; limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer; The wireless terminal executes the wireless communication method.

2. In the step of initializing the parameters, an update range of the parameters is set to all parameters of the neural network; and storing all of the parameters after the initialization is completed, The wireless communication method according to claim 1 , wherein the learning in a state in which the update range of the parameter is limited is executed after the initialization is completed and after the storage is completed.

3. The neural network is B ＵＢ an input layer and an output layer corresponding to the wireless base stations, B ＵＢ Among the wireless base stations, a wireless base station with excellent communication quality is selected as the B ＵＢ selecting only Bmax pieces of data, the number of which is less than Bmax pieces; further limiting the update range of the parameters to parameters associated with actions related to the Bmax radio base stations; The wireless communication method according to claim 1 or 2, further comprising the step of:

4. The wireless communication method according to claim 1 , wherein the neural network is a Deep Q-Network or a Double Deep Q-Network.

5. A wireless terminal having a function of connecting to a plurality of wireless base stations, The neural network includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and includes a memory unit storing a neural network that functions as a value function that outputs, in response to an input of a state, a value of each of the actions that the wireless terminal can take under the state, and a CPU; The CPU, A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses to the request from a plurality of wireless base stations; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer; A wireless terminal that runs

6. The CPU includes: A process of initializing the parameters by setting the update range of the parameters to all parameters of the neural network; and storing all the parameters after the initialization is completed. The wireless terminal according to claim 5 , wherein the learning in a state in which the update range of the parameter is limited is executed after the initialization is completed and after the storage is completed.

7. The neural network is B ＵＢ an input layer and an output layer corresponding to the wireless base stations, The CPU includes: B ＵＢ Among the wireless base stations, a wireless base station with excellent communication quality is selected as the B ＵＢ A process of selecting only Bmax pieces, which is less than B pieces; further limiting the update range of the parameters to parameters associated with actions related to the Bmax wireless base stations; 7. The wireless terminal according to claim 5 or 6, further comprising:

8. A program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, A CPU provided in the wireless terminal, a process of reading from a memory unit a neural network that includes an input layer and an output layer corresponding to a plurality of wireless base stations, and a hidden layer including two or more layers located between them, and that functions as a value function that outputs, in response to an input of a state, a value of each action that the wireless terminal can take under the state; A process of detecting a state of connection between the plurality of wireless base stations and the wireless terminal; providing the sensed state to the input layer and determining the action to be taken by the wireless terminal under the sensed state according to the result appearing in the output layer; transmitting a request corresponding to an action to be taken to said plurality of wireless base stations; obtaining responses to the request from a plurality of wireless base stations; calculating a reward for the behavior based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A process of limiting the update range of the parameters to parameters associated with one or more layers in the hidden layer, excluding at least the frontmost layer; A program for a wireless terminal including a program for executing the above.

Images