Wireless communication method, wireless terminal, and program for wireless terminal

Abstract

To obtain high communication efficiency in a network including a plurality of radio terminals and a plurality of radio base stations, or the like.SOLUTION: A radio communication method includes: (Step 100) detecting a state Sk(t) of connection between a plurality of radio base stations and a radio terminal, for DL and UL separately; (Step 108) causing a DQN to output values of actions ak(t) to be performed by the radio terminal in the state Sk(t), wherein a radio base station to be accessed is determined, separately for DL and UL, for each of the actions ak(t); (Step 112) requesting an action ak(t) to be adopted to the radio base station; (Step 116, 118) calculating a reward in which evaluation on DL and evaluation on UL are reflected on the basis of a response to the request; and (Step 122) updating a value of the action ak(t) on the basis of the reward ad learning a parameter of the DQN on the basis of the updated value.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a wireless communication method, a wireless terminal, and a program for a wireless terminal, and in particular to a wireless communication method, a wireless terminal, and a program for a wireless terminal that are suitable for achieving high communication efficiency in a network including multiple wireless terminals and multiple wireless base stations, etc. [Background technology]

[0002] With the development of wireless communication systems, heterogeneous networks using multi-band, multi-access systems are reaching the stage of practical application. In cellular communications, the fifth generation mobile communications (5G) has been put into practical use. 5G communication systems use a wide range of frequencies, from sub-1 GHz bands to millimeter wave bands. In addition, an environment is being developed in which cells of various sizes, from small cells to macro cells, are provided in an overlapping manner.

[0003] Another typical wireless access system, wireless LAN, uses radio frequencies in the 2.4 / 5 / 60 GHz bands, and the use of the 6 GHz band is also being considered. Wireless terminals such as smartphones are generally provided with interfaces (IFs) for accessing both cellular and wireless LAN. Each IF is configured to support multiple frequency bands.

[0004] Wireless terminals generally select a wireless base station or access point (AP) to connect to from multiple frequency bands and access methods to establish communication. With functions such as dual connectivity, one wireless terminal may integrate and use multiple wireless base stations. In such a heterogeneous environment, it is important to control and optimize, across the entire system, which IF a wireless terminal uses and which base station it selects, in order to effectively use system resources.

[0005] Regarding technology for effectively utilizing system resources in a heterogeneous environment, for example, Non-Patent Document 1 and Non-Patent Document 2 below disclose a method for optimizing connections between wireless terminals and APs by reinforcement learning. More specifically, Non-Patent Document 2 discloses a technology for optimizing connections between wireless terminals using multiple IFs and multiple APs by using a method of Deep Q-Learning or Double Deep Q-Learning.

[0006] In addition, the following non-patent document 3 discloses Deep Q-Network (DQN), a method for handling high-dimensional inputs using reinforcement learning, and the following non-patent document 4 discloses Double Deep Q-Network (DDQN). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] “Reinforcement Learning-aided Distributed User-to-Access Points Association in Interfering Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Hirantha Abeysekera, Yasushi Takatori, IEEE Globecom, Waikoloa, Hawaii, USA, Dec. 2019 [Non-Patent Document 2] “Deep Reinforcement Learning-based User Association in Sub6GHz / mmWave Integrated Networks”, Thi Ha Ly Dinh, Megumi Kaneko, Keisuke Wakao, Kenichi Kawamura, Takatsune Moriyama, Hirantha Abeysekera and Yasushi Takatori, in IEEE CCNC, Jan. 2021 [Non-Patent Document 3] “Human-level control through deep reinforcement learning,”Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves,Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou,Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, Nature. 2015 Feb 26; Vol.518(7540): P529-533. doi: 10.1038 / nature14236 [Non-Patent Document 4] “Deep Reinforcement Learning with Double Q-Learning,” H. Van Hasselt, A. Guez, and D. Silver, in Thirtieth AAAI conference on artificial intelligence, 2016 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] The above-mentioned non-patent document 1 discloses a method for updating DQN in the following procedure. 1. The wireless terminal determines which wireless base station to connect to by "Action a k (t)". 2. Action a with wireless terminal k If (t) is selected, the reward Γ is calculated based on the transmission data rate obtained from the wireless base station as a result. k (t) is calculated. 3. DQN is in a state S k Actions available under (t)a k For each of (t), this is used to calculate the value of adoption. 4. State S k (t) is an action a k (t) is selected, resulting in reward Γ k When (t) is obtained, the Γ k Based on (t), the state S k (t) under which the action a k The "value" of choosing (t) is updated. 5. Same (S k (t), a k The parameter θ that characterizes DQN is updated so that the “value” calculated by DQN for the combination of (t)) becomes the updated value.

[0009] According to this learning method, the connections between multiple wireless base stations and multiple wireless terminals can be determined so that the transmission rates provided by the multiple wireless base stations to the multiple wireless terminals are optimized. Therefore, the above learning method is useful for effectively utilizing the resources of a wireless communication system.

[0010] However, between a radio base station and a radio terminal, in addition to downlink (DL) communication from the former to the latter, uplink (UL) communication from the latter to the former is also performed. In conventional learning methods, including the techniques described in Non-Patent Documents 1 and 2 above, connection optimization is generally performed by considering only DL communication. However, in a technique for optimization by considering only DL communication, such as the above learning method, it may not be possible to optimize both DL and UL.

[0011] The present disclosure has been made in light of the above-mentioned problems, and aims to provide a wireless communication method, a wireless terminal, and a program for a wireless terminal that can optimize both DL and UL in a network including multiple wireless terminals and multiple base stations, etc. [Means for solving the problem]

[0012] In order to achieve the above object, a first aspect of the present disclosure is a wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, the method comprising: detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication; inputting said state into a value function; a step in which the value function outputs a value of each of actions that the wireless terminal can take under the state, and each of the actions is determined by distinguishing between DL communication and UL communication as to which wireless base station a connection request should be made; transmitting a request to a radio base station corresponding to an action to be taken; obtaining a response to the request from the wireless base station; A reward calculation step of calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response; updating a value for adopting the behavior under the state based on the reward; learning parameters of the value function based on the updated values; It is preferable that the present invention includes the following:

[0013] A second aspect of the present disclosure is a wireless terminal having a function of connecting to a plurality of wireless base stations, Equipped with a CPU, The CPU, detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication; inputting said state into a value function; a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication; transmitting a request to a wireless base station corresponding to the action to be taken; obtaining a response to the request from a wireless base station; A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; It is advisable to carry out the following:

[0014] A third aspect of the present disclosure is a program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, comprising: The CPU of the wireless terminal has detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication; inputting said state into a value function; a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication; transmitting a request to a wireless base station corresponding to the action to be taken; obtaining a response to the request from a wireless base station; A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; It is preferable that the program include a program for executing the above. Effect of the Invention

[0015] According to the present disclosure, both DL evaluation and UL evaluation are reflected in the reward for an action selected by a wireless terminal under a certain state. Then, parameters of an evaluation function indicating an optimal action under a certain state are learned based on the reward. Such learning allows the value function to converge so that both DL and UL are comprehensively optimized. Therefore, according to the present disclosure, bidirectional communication can be comprehensively optimized in a heterogeneous environment where DL and UL coexist. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining an overall configuration of a wireless communication system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram for explaining an example of a connection state between a wireless terminal and a wireless base station in the first embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram showing a state in which the sub-6 GHz band and the millimeter wave band are all used for downlink communication in a heterogeneous environment according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing a state in which a part of the millimeter wave band is used for uplink communication in the environment shown in FIG. [Diagram 5] FIG. 2 is a block diagram for explaining a functional configuration of a radio base station used in the first embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram for explaining a functional configuration of a wireless terminal used in the first embodiment of the present disclosure. [Figure 7] 2 is a diagram for explaining a hardware configuration of a wireless terminal used in the first embodiment of the present disclosure. FIG. [Figure 8] FIG. 1 is a diagram for explaining an overview of DQN used in the first embodiment of the present disclosure. [Figure 9] FIG. 2 is a diagram for explaining the content of a process related to updating a DQN used in the first embodiment of the present disclosure. [Figure 10] 1 is a flowchart illustrating a flow of processing executed by a wireless terminal in the first embodiment of the present disclosure to achieve an optimal connection state. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Embodiment 1 [Overall configuration of the first embodiment] Fig. 1 is a diagram for explaining an overall configuration of a wireless communication system 10 according to a first embodiment of the present disclosure. As shown in Fig. 1, the wireless communication system 10 includes a wired communication network 12. A plurality of wireless base stations 16 are connected to the network 12 via a switch 14. Each wireless base station 16 can communicate with another wireless base station 16 via the network 12 and via a wireless path.

[0018] 1 includes a plurality of wireless terminals 18. In this embodiment, the wireless terminals 18 include mobile terminals such as smartphones or tablet terminals.

[0019] Each of the wireless terminals 18 includes a plurality of sets of applications and IFs for wireless communication. For example, the wireless terminals 18 include an application and IF for communication in a band of 6 GHz or less (hereinafter referred to as "sub6GHz"), and an application and IF for communication in a millimeter wave band (hereinafter referred to as "mmWave"). Furthermore, the wireless terminals 18 may include an application and IF for communication in the 2.4 / 5 / 60 GHz bands used in wireless LANs.

[0020] The above-mentioned wireless base station 16 also has corresponding IFs, etc. Each wireless terminal 18 can wirelessly communicate with one or more wireless base stations 16 via the above-mentioned applications and IFs. This allows the wireless communication system 10 of the present embodiment to function as a heterogeneous network using multi-band multi-access.

[0021] In this embodiment, the wireless terminal 18 manages the connections used for communication for each of the downlink (DL) and the uplink (UL). More specifically, the wireless terminal 18 manages the correspondence between each of a plurality of applications, the IF to be used, and the wireless base station 16 to which the applications belong, by distinguishing between the DL and the UL.

[0022] Meanwhile, the wireless base station 16 judges the DL connection request and the UL connection request from the wireless terminal 18 for each application, and feeds back a response as to whether or not the request is acceptable. Then, the wireless terminal 18 calculates the reward Γ for the currently requested combination of connections based on the received response and taking into account both the results obtained in DL and UL.

[0023] The wireless communication system 10 of the present embodiment may be compatible with wireless LAN access points (APs) in addition to wireless base stations for cellular communication installed by a telecommunications carrier. For the sake of convenience, the following description will be given assuming that the "wireless base station 16" includes both.

[0024] [Problems with the First Embodiment] Next, features of the wireless terminal 18 in this embodiment will be described with reference to Figures 2 to 12. In the following description, it is assumed that the wireless base station 16 and the wireless terminal 18 are provided with an interface IF1 corresponding to communication in a first frequency band and an interface IF2 corresponding to communication in a second frequency band. IF1 is, for example, an interface corresponding to sub6GHz, and IF2 is, for example, an interface corresponding to mmWave.

[0025] In the following description, when it is necessary to specify a specific wireless terminal 18 among multiple wireless terminals, the term "wireless terminal 18" will be used. k In addition, when it is necessary to distinguish between the multiple wireless base stations 16, the reference numerals with suffixes, such as 16-1 and 16-2, are used.

[0026] FIG. 2 shows a wireless terminal 18 k The figure shows that the wireless terminal 18 belongs to two wireless base stations 16-1 and 16-2. k As described above, the wireless base station 16 has two interfaces, IF1 and IF2, which correspond to different bands. In the example shown in FIG. k However, a state is created in which the terminal belongs to the first wireless base station 16-1 at IF1 and belongs to the second wireless base station 16-2 at IF2.

[0027] FIG. 3 shows a heterogeneous environment with multi-band and multi-access, in which four wireless base stations 16-1 to 16-4 and three wireless terminals 18 i , 18 j , 18 k As in the case shown in Figure 2, each of them has IF1 corresponding to sub6GHz and IF2 corresponding to mmWave.

[0028] Sub-6 GHz communication waves have the advantage of being able to get around obstacles and are therefore suitable for securing a wide communication area. In Fig. 3, the elliptical areas surrounding each of wireless base stations 16-1 to 16-4 indicate the communication areas that each of wireless base stations 16-1 to 16-4 achieves by using IF1 corresponding to sub-6 GHz.

[0029] The mmWave communication wave has a tendency to travel in a straight line and is not suitable for covering a wide area like the sub6GHz wave, but is suitable for realizing high-speed and large-capacity communication. kEllipses containing arrows are drawn in three places, including between and. These ellipses represent the direction and width of the beamformed waveform.

[0030] In the example shown in FIG. i , 18 j , 18 k It is assumed that all of the wireless terminals 18 are connected to the wireless base stations 16-1 to 16-4 through DL communication. i , 18 j , 18 k The system capacity can be maximized by determining the connection destination so that each of the base stations 16 can receive the maximum data transmission rate from the base stations 16. However, in the wireless communication system 10 of the present embodiment, UL communication may be used.

[0031] FIG. 4 shows the wireless terminal 18 k 4 shows that IF2 is used for UL communication with the radio base station 16-2. In an environment where DL communication and UL communication coexist as shown in FIG. 4, a situation occurs in which the system capacity cannot be maximized by optimization that evaluates only DL communication. Therefore, in this embodiment, DL communication and UL communication are distinguished, and both are evaluated to optimize the connection.

[0032] [Configuration of wireless base station] 5 is a block diagram for functionally explaining an example of wireless base station 16. Wireless base station 16 includes an antenna unit 20. Antenna unit 20 is used by wireless base station 16 to transmit and receive wireless signals to and from other wireless base stations 16 and surrounding wireless terminals 18.

[0033] The antenna unit 20 is connected to the wireless communication unit 22. The wireless communication unit 22 includes an RF (Radio Frequency) unit 24 that processes high-frequency signals, and a MAC (Media Access Control) function unit 26 that has a function of media access control in wireless communication. The wireless communication unit 22 can realize functions corresponding to a plurality of frequency bands, specifically, a function as an IF1 corresponding to sub6GHz, and a function as an IF2 corresponding to mmWave.

[0034] In this embodiment, both DL communication and UL communication may be mixed in each of the multiple IFs (here, IF1 and IF2) realized by the wireless communication unit 22. However, the optimization process described below is performed assuming that each IF is assigned to only one of DL and UL. Also, this optimization process is performed assuming that IFs of multiple wireless base stations 16 are not assigned simultaneously to one wireless terminal 18 for DL ​​or UL.

[0035] The wireless base station 16 includes a quality measurement unit 28. The quality measurement unit 28 includes a wireless environment measurement unit 30 and a traffic measurement unit 32. The wireless environment measurement unit 30 acquires information related to the wireless communication environment surrounding the wireless base station 16. The traffic measurement unit 32 acquires information related to the traffic between the wireless base station 16 and the wireless terminal 18. The quality measurement unit 28 calculates information on the communication quality between the wireless base station 16 and the wireless terminal 18 based on the information. The communication quality information includes, for example, the received power from the wireless terminal 18, the required data rate of the wireless terminal 18, the actual transmission rate, and the band usage rate in the wireless base station 16.

[0036] The wireless base station 16 includes a communication IF unit 34. The communication IF unit 34 is used for wired communication via the network 12. The wireless base station 16 can acquire communication quality information acquired by other wireless base stations 16 through wired communication via the communication IF unit 34 or through wireless communication via the wireless communication unit 22 and the antenna unit 20. The communication quality information acquired from other wireless base stations 16 is managed in a unified manner in the quality measurement unit 28 together with communication quality information acquired by the wireless base station 16 itself.

[0037] The wireless base station 16 includes an information notification unit 36. The information notification unit 36 ​​can read out communication quality information aggregated in the quality measurement unit 28. The information notification unit 36 ​​can provide the read communication quality information to the communication IF unit 34 and the wireless communication unit 22. This allows the wireless base station 16 to provide its own communication quality information to other wireless base stations 16 via a wired or wireless path.

[0038] The wireless base station 16 receives DL association requests and UL association requests from wireless terminals 18 located in the vicinity via the wireless communication unit 22 and the antenna unit 20. These association requests are processed by a request information evaluation unit 38. The request information evaluation unit 38 determines whether or not to associate based on the bandwidth and data rate requested by the wireless terminal 18, its own load state, and the like.

[0039] The result of the determination by the request information evaluation unit 38 is sent to a request response notification unit 40 and an association information recording unit 42. The request response notification unit 40 provides the determination result on whether or not association is possible for each of DL and UL to the wireless terminal 18 that issued the association request via the wireless communication unit 22 and the antenna unit 20. The association information recording unit 42 updates and records the information of the wireless terminal 18 that is associated with the wireless base station 16 for each DL and UL based on the result of the determination.

[0040] [Wireless terminal configuration] 6 is a block diagram for functionally explaining an example of wireless terminal 18. Wireless terminal 18 includes an antenna unit 44 and a wireless communication unit 46. Wireless communication unit 46 includes an RF unit 48 and a MAC function unit 50. These functions are substantially the same as the functions of the corresponding blocks included in wireless base station 16, so description thereof will be omitted here.

[0041] In this embodiment, the wireless terminal 18 includes a DL / UL management unit 52. The DL / UL management unit 52 executes processes necessary for DL ​​communication in the wireless terminal 18 and processes necessary for UL communication.

[0042] The wireless terminal 18 includes a control information calculation unit 54. The control information calculation unit 54 acquires communication quality information from the wireless base station 16 to which the wireless terminal 18 belongs via the wireless communication unit 46 and the antenna unit 44. The control information calculation unit 54 further performs optimization processing using DQN or DDQN based on the acquired communication quality information. Specifically, the control information calculation unit 54 calculates the optimal behavior a for connecting each of the multiple IFs to which IF of which wireless base station 16 and for which purpose in DL and UL. k The process of determining whether the target is (t) using DQN or DDQN and the process of learning the DQN or DDQN are performed.

[0043] The control information calculation unit 54 includes a state calculation unit 56. The state calculation unit 56 calculates a state S to be provided to the input layer of the DQN or DDQN based on the communication quality information. k (t) is calculated. State S k (t) includes information about which IF of which wireless base station 16 each of its own IFs belongs to for each DL and UL.

[0044] The control information calculation unit 54 also includes a reward calculation unit 58. The reward calculation unit 58 calculates the reward for the adopted action a based on the communication quality information. k Reward Γ obtained by (t) k Calculate (t). Reward Γ k(t) is calculated so as to reflect both the evaluation of DL communication and the evaluation of UL communication. The calculation method will be described in detail later.

[0045] The control information calculation unit 54 further includes a DQN / DDQN update unit 60. The DQN / DDQN update unit 60 updates a parameter θ of the DQN or DDQN used in reinforcement learning. More specifically, the weighting coefficients and bias values ​​(hereinafter referred to as "weighting coefficients, etc.") that determine the characteristics of the DQN or DDQN as a value function are updated.

[0046] The DQN or DDQN updated by the DQN / DDQN update unit 60 is used in the attribute calculation unit 62. That is, the attribute calculation unit 62 determines the action a to be adopted this time according to the updated DQN or DDQN. k (t), that is, it calculates which IF should be assigned to which IF of which wireless base station 16 for which purpose (DL and UL).

[0047] DQN and DDQN input a state S k Given (t), the possible actions that the wireless terminal 18 can take are k For each of (t), the value of adoption is displayed in the output layer. Therefore, if the action with the largest value is selected, the action considered optimal at the current learning stage can be selected. However, the next action a k If (t) is always selected based on the above values, it is possible that we may become trapped in a local solution chosen by chance, and the truly optimal action may never be selected.

[0048] In order to avoid such a situation, the destination calculation unit 62 performs the action a according to the ε-greedy method. k For this reason, the destination calculation unit 62 randomly calculates the next action a with a probability ε regardless of whether it is DQN or DDQN. k (t) is selected. Also, with probability 1-ε, the action a that DQN or DDQN judges to be optimal is k (t) is selected.

[0049] Furthermore, the destination calculation unit 62 performs a decay process on ε each time learning is repeated. The possibility of falling into a local solution selected by chance is large in the early stages of learning, and decreases as learning progresses. For this reason, when the behavior a is randomly selected, k The need to select (t) decreases as learning progresses. When ε is decayed, the action a k It is possible to reduce the frequency with which (t) is randomly selected. Therefore, according to the destination calculation unit 62 of this embodiment, learning can be efficiently carried out and the parameter θ can be converged to an optimal value in a short period of time.

[0050] The wireless terminal 18 includes a memory unit 66. The memory unit 66 stores the following information: k Store as (t). 1. State S provided to the input layer of DQN or DDQN in this learning k (t), 2. That state S k Action taken in response to (t) k (t), 3. The action a k Reward Γ obtained by (t) k (t), 4. The action a k State S after transition due to (t) k (t+1).

[0051] The wireless terminal 18 also includes an attribute request unit 68. The attribute request unit 68 receives the action a from the attribute calculation unit 62. k Receive (t) and take action a k The wireless communication unit 46 generates a request corresponding to (t). For example, a request is generated to connect to the wireless base station 16-1 using IF1 for DL ​​use, or to connect to the wireless base station 16-2 using IF2 for UL use. The generated request is transmitted to one or more wireless base stations 16 via the wireless communication unit 46 and the antenna unit 20.

[0052] The wireless terminal 18 includes an associated destination recording unit 70. The information received from the wireless base station 16 includes a response to the request as to whether it is OK or not. For example, the information includes a response to allow a new connection, or a response to allow an update of a connection. The associated destination recording unit 70 updates and records information about the associated wireless base station 16 based on these responses.

[0053] 7 is a diagram for explaining the hardware configuration of the wireless terminal 18. Here, a case will be explained where the wireless terminal 18 is a smartphone. However, the wireless terminal 18 is not limited to a smartphone, and may be a tablet terminal or a general-purpose computer system.

[0054] The wireless terminal 18 includes a CPU (Central Processing Unit) 72. The CPU 72 is connected to various hardware elements included in the wireless terminal 18 via a bus wiring 74. To the bus wiring 74, various memory devices such as a ROM (Read Only Memory) 76, a RAM (Random Access Memory) 78, and a storage 80 are connected.

[0055] The storage 80 includes a non-volatile memory such as a flash ROM. Programs and data required for executing various applications are stored in the storage 80. The memory unit 66 and the destination recording unit 70 described above are realized by the storage 80. The CPU 72 also performs processing according to the programs stored in the storage 80, thereby realizing functions of the DL / UL management unit 52 and the control information calculation unit 54 shown in FIG. 6 and the like.

[0056] The wireless terminal 18 includes a communication interface 82. The communication interface 82 corresponds to the antenna unit 44 and the wireless communication unit 46. The wireless terminal 18 further includes an operation unit 84 and a display unit 86. The operation unit 84 includes a touch panel integrated with the display unit 86, in addition to various mechanical switch mechanisms and electronic sensor mechanisms.

[0057] [Overview of DQN] FIG. 8 is a diagram for explaining an overview of DQN. Hereinafter, with reference to FIG. 8, the wireless terminal 18 k The outline of DQN, which is an example of deep reinforcement learning used by the present embodiment, will be described. The DQN used in this embodiment has an input layer, a hidden layer, and an output layer. In addition, the DQN has a parameter θ i t The characteristic of DQN as a value function is the parameter θ i t is determined by.

[0058] In FIG. 8, the input layer of DQN consists of M nodes S1 to S M These nodes include wireless terminals 18 k Status S k The elements that make up the state S are input. k The elements of include, for example, the following elements. These are treated separately as those related to DL and those related to UL. 1. Bandwidth utilization rate φb of wireless base station 16-b, 2. Wireless terminal 18 k The required data rate R of the interface f kf , 3. Wireless terminal 18 k and the wireless base station 16-b bkf (1 if it belongs, 0 if not), 4. Wireless terminal 18 k A request for attribution to the wireless base station 16-b of the interface f of bkf (1 if requested, 0 if not), 5. Wireless terminal 18 k The transmission rate r established between the interface f and the wireless base station 16-b bkf .

[0059] The number of hidden layers and the number of nodes are arbitrary. k But in state S k Actions that can be taken under a iThe number of nodes is equal to the total number of actions in the i Some examples are shown below. · IF1 is assigned to radio base station 16-1 for DL ​​use. ·IF1 is assigned to radio base station 16-1 for UL purposes. · IF2 is assigned to wireless base station 16-1 for DL ​​use. · IF2 is assigned to radio base station 16-1 for UL purposes. · IF1 is attributed to the wireless base station 16-1 for DL ​​use, and IF2 is attributed to the wireless base station 16-2 for UL use. · IF1 is attributed to the wireless base station 16-1 for UL use, and IF2 is attributed to the wireless base station 16-2 for DL ​​use. · · ·

[0060] Each node in the output layer is connected to a wireless terminal 18. k Possible action a i Q value for each of (i=1~N) = Q(S k ,a i ;θ i t ) is expressed. The Q value is expressed by the parameter θ i t Action a calculated by DQN under i This is the value of adopting it.

[0061] In the method using DQN, the largest Q value is selected from the Q values ​​that appear in the output layer, and the action a that produces that Q value is i Let a be the optimal action at the current time t. k According to DQN, in this way, at each learning stage, the optimal action at that time t is determined as a k (t) can be determined.

[0062] FIG. 9 shows the wireless terminal 18 k The control information calculation unit 54 calculates the parameters θ of DQN. i FIG. 8 is a diagram for explaining the flow of updating the action a kFor convenience, (t) is shown after DQN / DDQN, but action a k As mentioned above, (t) is determined by DQN or DDQN only when the probability is 1-ε. And, under the probability ε, the action a k (t) is determined randomly. Parameter θ i The experience gained in both cases is k (t) is reflected.

[0063] Wireless terminal 18 k So, action a k After (t) is decided, the action a k Reward Γ for (t) k (t) is calculated. Reward Γ k (t) is calculated, for example, by the following formula (1).

[0064]

number

[0065] In the above formula (1), the first term on the right side is a point-adding item, and the second term is a point-deducting item. 1k , ω 2k is a coefficient that determines the allocation ratio for them. λ is an identifier that indicates each element of the set Λ = {DL, UL}. ν is an identifier that indicates each element of the set L = {sub6GHz, mmWave}.

[0066] Furthermore, in the above formula (1), c in the first term on the right side 1k λ,ν is the desired data rate R at λ (DL or UL) kf λ A transmission rate r that satisfies bkf (t) means the additional points when obtained at ν (sub6GHz or mmWave). The value is calculated by the following formula (2).

[0067]

number

[0068] The above formula (2) is expressed as follows: bkf (t) bkf (t) / W λ,ν This means that the following is given. However, the W in the last term λ,ν is a factor for offsetting and normalizing the difference in frequency bands due to the difference in λ (DL or UL) and ν (sub6GHz or mmWave). 1. The wireless base station 16-b transmits the signal to the wireless terminal 18 k The attribution of interface f is allowed for ν (sub6GHz or mmWave) for λ (DL or UL) applications (x bkf λ,ν =1), and for λ (DL or UL), the required data rate R kf λ A transmission rate of r or more bkf (t) is obtained (r bkf (t) ≧ R kf λ ). 2. The wireless base station 16-b receives the wireless terminal 18 k The interface f of is requesting attribution in ν (sub6GHz or mmWave) for λ (DL or UL) use (a bkf λ,ν (t)=1).

[0069] In the above formula (1), c in the second term on the right side 2k λ,ν is the desired data rate R at λ (DL or UL) kf λ A transmission rate r that satisfies bkf This means the amount of points deducted when (t) is not obtained at ν (sub6GHz or mmWave). The value is calculated, for example, by the following formula (3).

[0070]

number

[0071] The above formula (3) can be expressed as follows in the two cases: k(t) is the required data rate R kf λ and transmission rate r bkf This means that points will be deducted in proportion to the ratio of (t). 1. A request for attribution in ν (sub6GHz or mmWave) has been made for λ (DL or UL) use (a bkf λ,ν (t)=1), attribution was allowed, but (x bkf λ,ν (t)=1), the required data rate R kf λ A transmission rate r that satisfies bkf If (t) is not obtained (r bkf (t) <R kf λ ). 2. A request for attribution in ν (sub6GHz or mmWave) was made for λ (DL or UL) use (a bkf λ,ν (t)=1), if the wireless base station 16 does not permit association with the requested λ (DL or UL) (x bkf λ,ν (t)=0).

[0072] In this way, the reward Γ k (t) is action a k The required data rate R as a result of (t) kf λ A transmission rate r that satisfies bkf On the other hand, if a sufficient transmission rate r bkf If (t) is not obtained, reward Γ k (t) becomes a small value.

[0073] In addition, the transmission rate r included in the above formula (3) bkf (t) can be calculated using the following formula (4) for each of the following four situations. 1. Wireless terminal 18 k However, for DL ​​purposes, it belongs to wireless base station 16-b at sub6GHz. 2. Wireless terminal 18 k However, for UL purposes, it belongs to wireless base station 16-b at sub6GHz. 3. Wireless terminal 18 k However, for DL ​​purposes, it is assigned to wireless base station 16-b via mmWave. 4. Wireless terminal 18 k However, it is assigned to radio base station 16-b via mmWave for UL purposes.

[0074]

number

[0075] In addition, the four right-hand sides of the above equation (4) include γ bkf λ,ν (t) is the SINR (Signal to Interference plus Noise power Ratio) under each condition, which can be calculated using the following equations (5) to (8).

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080] However, in the above formula (5), p bkf DL,sub (t) is the transmission power in DL and sub6GHz. Also, h bk sub (t) is the correlation coefficient between the wireless base station 16-b and the wireless terminal 18k in the sub 6 GHz band. The value becomes large when the distance between the two is short, and is affected by propagation loss and fading. b´ksub (t) and h k´k sub (t) is also the correlation coefficient under the environment indicated by each subscript. Furthermore, the W at the end of the denominator DL,sub represents the bandwidth in DL and sub6GHz and is used to calculate the power of the noise part. bkf λ,ν (t), h bk ν (t) etc., as well as W λ、ν The same applies to.

[0081] As described above, in this embodiment, the wireless terminal 18 k The communication quality between the mobile station and the wireless base station 16 is evaluated for each of DL and UL. k Reward Γ for (t) k (t) reflects both of these.

[0082] In this way, the reward Γ k Once (t) is calculated, the current experience e k (t) is stored in memory (step 88). k (t), selected action a k (t), the reward obtained Γ k (t), and the state after the transition S k (t+1) is experience e k It is stored in memory as (t).

[0083] If there is a time correlation in the sample sequence, learning may not proceed properly. To solve this problem, we set the parameter θ i When updating, the experience stored in memory k The memory samples are randomly taken from and processed in batches (step 90).

[0084] A randomly taken memory sample, i.e. (S k , a k , Γ k , S k Experiences that include kUsing this, the parameters θ of DQN / DDQN i (Step 92). k ´ is action a k Specifically, the process of step 92 proceeds as follows.

[0085] action a k Reward for Γ k Once determined, the reward Γ k Based on the state S k Acting under a k The "new Q value" for selecting S k ,a k The new Q value is calculated, for example, as shown in the following formula (9).

[0086] Q(S k ,a k )←Γ k +γ*maxQ(S k ´,a k ´) ···(9) where γ is the time discount rate.

[0087] The right-hand side of the above equation (9), Γ k +γ*maxQ(S k ´,a k ´) is (S k ,a k ) is the value that we want the "trained DQN" to calculate as the Q value for each combination. Therefore, the DQN parameter θ i is (S k ,a k ) input, the value is updated so that it appears in the output layer.

[0088] The following equation (10) represents the DQN parameter θ i The loss function L used to update k Shows.

[0089]

number

[0090] However, in the above formula (10), the first term on the right side, Q(S k ,a k ;θ i ) is the parameter θ i The main network with (S k ,a k ) is the Q value calculated for the combination of k ´,a k ´;θ i ― ) is the parameter θ i ― If the target network has (S k ´,a k ´) is the Q value calculated for the combination.

[0091] When updating DQN, if the value of the network to be updated is calculated using the network being updated, learning may not be stable. For this reason, when updating DQN, in addition to the main network to be updated, a target network for calculating the update value is usually used. In this embodiment, the second term of equation (10), which corresponds to the new Q value, is calculated using the target network according to the above-mentioned usual method.

[0092] In step 92, the loss function L shown in the above equation (10) is used. k The DQN parameters θ are set so that i , i.e., the main network parameters θ i As a result, the DQN is updated to state S k For input, action a k The Q value of the model is trained to express the value on the right side of equation (9) above.

[0093] The target network is identical to the past main network except immediately after the update. The target network is then updated at a predetermined period (step 94). That is, when the time for updating arrives at the predetermined period, the parameter θ i ― is the parameter θ of the main network.i This completes the update of DQN.

[0094] In addition, the following equation (11) represents the loss function L used in step 92 when the DDQN method is used in this embodiment. k The principle of the update is the same as that of DQN, so we will not explain the details here.

[0095]

number

[0096] [Processing by the wireless terminal's CPU] 10 is a flowchart for explaining the flow of processing executed by the CPU 72 of the wireless terminal 18 to realize the above-mentioned functions. Note that, here, a case will be explained in which DQN is used as the reinforcement learning technique.

[0097] In the routine shown in FIG. 10, first, a state S to be provided to the input layer of DQN is calculated based on communication quality information obtained from the surrounding wireless base stations 16. k (t) is set (step 100). k As described above, (t) includes connection information regarding DL and connection information regarding UL.

[0098] Next, ε to be used in the current learning is set (step 102). For example, a value of about 0.1 is given to ε as an initial value, and thereafter, ε is subjected to a decay process each time this routine is repeated.

[0099] Next, a random number is generated that falls within the range of 0 to 1. Furthermore, it is determined whether the random number is equal to or smaller than ε (step 104).

[0100] If the random number ≦ ε is found to be true, the current state S k For (t), the next action is a k (t) is randomly determined (step 106).k (t) includes information that specifies the wireless base station 16 to which the user is to belong for each of DL and UL.

[0101] On the other hand, if it is not determined in step 104 that the random number ≦ ε holds, the input layer of the DQN is assigned a state S k (t) is input (step 108). Then, the Q values ​​appearing in the output layer of the DQN are compared, and the action a with the largest Q value is selected. i (t) is the optimal action at the current time a k (t) is selected (step 110).

[0102] Next, the action a selected by the above process k Based on (t), an association request is sent to the surrounding wireless base stations 16 (step 112). Specifically, one or more connection requests are sent, specifying DL or UL, sub6GHz or mmWave, and the wireless base station 16-b to which the connection is to be made.

[0103] In the wireless base station 16 that has received the connection request, it is determined whether or not a connection can be established for each of the DL and UL. The determination of whether or not a connection can be established is performed for each of the DL and UL by, for example, a branch and bound method, in the following procedure. 1. All connection requests from the wireless terminal 18 are forwarded to the bkf / φ bkf ) in descending order (where φ bkf The wireless base station 16-b transmits the k Application f with transmission rate r bkf (The load when obtaining the 2. Select candidates for attribution in the order of the above sorting and determine whether they can be accommodated. If they can be accommodated with the remaining capacity, the connection is allowed, otherwise it is rejected. 3. Once all the candidates have been judged, a response including the results is transmitted to the surrounding wireless terminals 18.

[0104] Wireless terminal 18 kWhen the device receives a response to the request, it acquires the communication quality information contained in the response (step 114).

[0105] Next, based on the communication quality information, γ bkf λ,ν (t) is calculated (step 116). This allows the SINR to be calculated for each condition determined by either DL or UL, and either sub6GHz or mmWave.

[0106] Next, γ bkf λ,ν By applying the calculated value of (t) to the above formulas (1) to (4), the current action a k Reward Γ for (t) k (t) is calculated (step 118). This results in a reward Γ k (t) is calculated.

[0107] When the above process is completed, the experience data obtained in this routine is stored in the RAM 78 constituting the memory unit 66. k (t), i.e., (S k (t), a k (t), Γ k (t), S k (t+1)) is stored (step 120).

[0108] Thereafter, the DQN is updated by batch processing (step 122), and the current routine is terminated.

[0109] As described above, the wireless terminal 18 in this embodiment determines the reward Γ k (t) and calculate the reward Γ k Based on the updated Q value based on (t), the parameters θ of DQN or DDQN are iis updated. Therefore, according to this embodiment, in a heterogeneous environment where DL and UL coexist, it is possible to learn DQN or DDQN so as to comprehensively optimize the quality of both DL and UL. Then, connections between multiple wireless terminals 18 and multiple wireless base stations 16 are determined according to the DQN or DDQN, so that bidirectional communications can be comprehensively optimized in a heterogeneous environment where DL and UL coexist.

[0110] Incidentally, in the above-mentioned first embodiment, the wireless terminal 18 and the wireless base station 16 use two IFs for wireless communication, but the present disclosure is not limited to this. For example, the number of IFs for wireless communication may be three or more. Furthermore, the IFs for wireless communication may include a mixture of IFs for licensed bands and IFs for unlicensed bands used for wireless LANs, etc.

[0111] In the above-mentioned first embodiment, DQN or DDQN is used as a value function for determining an action. However, these are merely examples, and the present disclosure is not limited to these. Any function that can determine the value of each action that can be taken under a specific state can be used as a value function in the present disclosure. [Explanation of symbols]

[0112] 10. Wireless communication systems 16, 16-1, 16-2, 16-3 Radio base station 18, 18 i , 18 j , 18 k Wireless terminal 52 DL / UL Management Department 54 Control information calculation unit 60 DQN / DDQN update section Gamma k (t) Remuneration a k (t) action e k (t) Experience S k (t) Status Sk (t+1) State after transition R kf Requested Data Rate r bkf Transmission rate θ i Parameters

Claims

1. A wireless communication method for determining a combination of connections between a plurality of wireless base stations and a wireless terminal, comprising: detecting a state of connection between the plurality of radio base stations and the radio terminal by distinguishing between DL communication and UL communication; inputting said state into a value function; a step in which the value function outputs a value of each of actions that the wireless terminal can take under the state, and each of the actions is determined by distinguishing between DL communication and UL communication and determining to which wireless base station a connection request should be made; transmitting a request to a radio base station corresponding to an action to be taken; obtaining a response to the request from the wireless base station; a reward calculation step of calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response; updating a value for adopting the behavior under the state based on the reward; and learning parameters of the value function based on the updated values; A wireless communication method, wherein the state input to the value function includes the presence or absence of a DL-related association request and the presence or absence of a UL-related association request of the wireless terminal to each of the plurality of wireless base stations.

2. At least some of the plurality of wireless base stations have a plurality of wireless communication interfaces having different specifications, the wireless terminal has the plurality of interfaces for wireless communication, The state includes information regarding which radio base station the radio terminal is connected to, through which interface, and whether DL or UL, the radio terminal is connected to; The wireless communication method according to claim 1 , wherein the action includes information on which interface of the wireless terminal is to be connected to which wireless base station in either DL or UL.

3. The remuneration calculation step includes: calculating a SINR for DL ​​communication requested by the wireless terminal; calculating a transmission rate obtained in DL communication based on the SINR for DL; calculating a SINR in an UL communication requested by the wireless terminal; calculating a transmission rate obtained in an UL communication based on the SINR for the UL; calculating the reward based on the transmission rate for DL ​​and the transmission rate for UL; 3. The wireless communication method according to claim 1, further comprising:

4. The wireless communication method according to claim 1 , wherein the value function is a Deep Q-Network or a Double Deep Q-Network.

5. A wireless terminal having a function of connecting to a plurality of wireless base stations, A CPU is provided, The CPU, detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication; inputting said state into a value function; a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication; transmitting a request to a wireless base station corresponding to the action to be taken; obtaining a response to the request from a wireless base station; A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response; updating a value for taking the action under the state based on the reward; and learning parameters of the value function based on the updated value. The state input to the value function includes the presence or absence of a DL-related association request and the presence or absence of a UL-related association request of the wireless terminal with respect to each of the plurality of wireless base stations.

6. At least some of the plurality of wireless base stations have a plurality of wireless communication interfaces having different specifications, the wireless terminal has the plurality of interfaces for wireless communication, The state includes information regarding which radio base station the radio terminal is connected to, through which interface, and whether DL or UL, the radio terminal is connected to; The wireless terminal according to claim 5 , wherein the action includes information regarding which interface of the wireless terminal is to be connected to which wireless base station in either DL or UL.

7. The reward calculation process includes: A process of calculating an SINR related to DL communication requested by the wireless terminal; Calculating a transmission rate obtained in DL communication based on the SINR for DL; A process of calculating a SINR related to UL communication requested by the wireless terminal; calculating a transmission rate obtained in UL communication based on the SINR for UL; calculating the reward based on the transmission rate for DL ​​and the transmission rate for UL; 7. The wireless terminal according to claim 5 or 6, comprising:

8. A program for a wireless terminal for realizing a function of connecting to a plurality of wireless base stations, A CPU provided in the wireless terminal, detecting a state of connection between the plurality of wireless base stations and the wireless terminal by distinguishing between DL communication and UL communication; inputting said state into a value function; a process of calculating a value of each of actions that the wireless terminal can take under the state by using the value function, and determining to which wireless base station a connection request is to be made for each of the actions by distinguishing between DL communication and UL communication; transmitting a request to a wireless base station corresponding to the action to be taken; obtaining a response to the request from a wireless base station; A reward calculation process for calculating a reward reflecting both the DL evaluation and the UL evaluation based on the response; updating a value for taking the action under the state based on the reward; A process of learning parameters of the value function based on the updated value; A program for executing A program for a wireless terminal, wherein the state input to the value function includes the presence or absence of a DL-related association request and the presence or absence of a UL-related association request of the wireless terminal with respect to each of the plurality of wireless base stations.

Images