Communication control device and communication control method

Abstract

To provide a communication control device and a communication control method for optimizing a bandwidth utilization rate in a communication network.SOLUTION: A communication control device 400 for controlling execution of communication and non-execution of communication of a communication terminal for each time slot of a plurality of time slots includes a reward management unit 420 for managing, for each time slot, a first reward value for the execution of communication by the communication terminal and a second reward value for the non-execution of communication by the communication terminal, a control information creation unit 450 for creating control information indicating whether the communication terminal should execute communication for each time slot, by using a learning model generated by reinforcement learning based on the first reward value and the second reward value, and a transmission unit 460 for transmitting the created control information to the communication terminal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a communication control device and a communication control method. [Background technology]

[0002] In general, communication traffic tends to be concentrated in certain time periods. For example, communication traffic tends to increase during the daytime (e.g., 11:00-13:00) and in the evening (e.g., 17:00-20:00). Conventionally, services have been provided that allow IoT terminals and the like to communicate at night when traffic is low (e.g., see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "IIJ Mobile M2M Access Service", [online], Internet Initiative Japan Inc., [searched September 26, 2023], Internet<https: / / www.iij.ad.jp / biz / mobile-m2m / > Summary of the Invention [Problem to be solved by the invention]

[0004] However, nighttime is not necessarily the only time period with low traffic, and there are cases where traffic is surplus even during times other than nighttime. Thus, currently, optimization of communication band utilization has not yet been sufficiently carried out. [Means for solving the problem]

[0005] According to one aspect of the present invention, there is provided a communication control device that controls whether a communication terminal communicates or not for each time slot of a plurality of time slots, the communication control device comprising: a reward management unit that manages, for each time slot, a first reward value for the communication terminal communicating and a second reward value for the communication terminal not communicating; a control information creation unit that creates control information indicating whether the communication terminal should communicate for each time slot using a learning model generated by reinforcement learning based on the first and second reward values; and a transmission unit that transmits the created control information to the communication terminal.

[0006] According to one aspect of the present invention, the communication terminal may further include a communication status acquisition unit that acquires communication status in a network to which the communication terminal is connected, and the reward management unit may set the first reward value smaller than the second reward value for high traffic time slots, and set the first reward value larger than the second reward value for low traffic time slots, based on the communication status.

[0007] According to one aspect of the present invention, the communication status acquisition unit acquires the communication status at a predetermined time interval, the reward management unit updates the first and second reward values ​​for each time slot according to the communication status acquired at the predetermined time interval, and the learning model may be sequentially updated using the updated first and second reward values.

[0008] Also, according to one aspect of the present invention, the first and second reward values ​​may be preset fixed values.

[0009] According to another aspect of the present invention, the learning model assigns an identifier for identifying the time slot to a state s t and each action a of the communication terminal performing communication or not performing communication is input as t Action value Q(s t ,a t) may be a model configured with a neural network based on deep reinforcement learning.

[0010] According to another aspect of the present invention, the control information creation unit is configured to: t The action value for the communication terminal is an action a t If the action value for the time slot is greater than the action value for the time slot, the control information may be created to instruct that the communication terminal should not carry out communication for the time slot.

[0011] According to one aspect of the present invention, there is provided a communication control method for controlling whether a communication terminal communicates or not for each time slot of a plurality of time slots, the method including the steps of: specifying, for each time slot, a first reward value for the communication terminal communicating and a second reward value for the communication terminal not communicating; creating control information indicating whether the communication terminal should communicate for each time slot using a learning model generated by reinforcement learning based on the first and second reward values; and transmitting the created control information to the communication terminal. Effect of the Invention

[0012] According to the present invention, it is possible to realize optimization and leveling of bandwidth utilization in a communication network. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram showing a schematic configuration of a network system to which a communication control device according to an embodiment of the present invention is applied; [Diagram 2] 1 is a block diagram showing a functional configuration of a communication control device according to an embodiment of the present invention; [Diagram 3] 13 is an example of a management table for a reward management unit to manage reward values. [Figure 4] FIG. 2 is a schematic diagram showing state transitions of a communication terminal; [Diagram 5] FIG. 1 is a diagram showing an example of a learning model (DQN). [Figure 6] FIG. 13 is a diagram showing another example of a learning model (Fixed Target Q-Network). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] FIG. 1 is a diagram showing a schematic configuration of a network system to which a communication control device according to an embodiment of the present invention is applied. The network system 10 includes one or more user terminals 100, one or more communication terminals 200, a gateway device 300, and a communication control device 400. The user terminals 100, the communication terminals 200, the gateway device 300, and the communication control device 400 constitute a communication network 500. Each user terminal 100 and communication terminal 200 can communicate with other devices (not shown) connected to another communication network 600 (e.g., the Internet) via the gateway device 300. Note that the communication network 500 of the network system 10 may include one or more control devices that relay or control communication from / to the user terminals 100 and communication terminals 200 in addition to the gateway device 300 and the communication control device 400, but these are omitted from the illustration in FIG. 1 for the sake of simplicity.

[0016] In this specification, the user terminal 100 refers to a device in which at least a part of communication is performed in response to an operation on the user terminal 100 by a human user. Exemplary user terminals 100 include smartphones, tablet terminals, PCs, game consoles, etc. For example, in response to an operation on the user terminal 100 by a user to view a website or video, data corresponding to the request is transmitted from a predetermined server device or the like on the communication network 600 and received by the user terminal 100.

[0017] The amount of communication data by the user terminal 100 is time-dependent. For example, the amount of communication data by the user terminal 100 tends to increase in the daytime (for example, 11:00 to 13:00) and in the evening (for example, 17:00 to 20:00).

[0018] Meanwhile, in this specification, the communication terminal 200 means a device that automatically performs communication without the operation of a human user. Such a communication terminal 200 is configured to transmit and receive data at a programmed timing, for example. Representative communication terminals 200 include various IoT (Internet of Things) terminals, such as various types of sensor devices configured to measure, generate, and transmit various sensor data. For example, such an IoT terminal (communication terminal 200) may be configured to sequentially generate sensor data at a predetermined period, temporarily store the data until a predetermined transmission timing, and transmit the sensor data stored up to that point to a predetermined server device on the communication network 600 at a predetermined programmed timing. The communication terminal 200 in this specification is not limited to such an IoT terminal, and may include any communication device that automatically transmits and receives data. For example, when a smartphone or a PC (the above-mentioned user terminal 100) transmits and receives data in the background (for example, software update, etc.), these devices may be considered as the communication terminal 200 only for such background communication.

[0019] The communication terminal 200 is a target of communication control by a communication control device 400 according to one embodiment of the present invention. The communication control device 400 controls whether the communication terminal 200 performs communication. In contrast, the user terminal 100 (communication performed in response to a user operation) is not a target of communication control by the communication control device 400.

[0020] The gateway device 300 relays communication between the user terminal 100 and the communication terminal 200 and devices on the communication network 600. The gateway device 300 also has a function of monitoring the communication status (e.g., traffic volume) between the communication network 500 and the communication network 600.

[0021] 2 is a block diagram showing a functional configuration of a communication control device 400 according to an embodiment of the present invention. The communication control device 400 includes a communication status acquisition unit 410, a reward management unit 420, a learning model generation unit 430, a learning model 440 generated by the learning model generation unit 430, a control information creation unit 450, and a control information transmission unit 460. The communication control device 400 can be realized by a computer including a processor and a memory.

[0022] The communication status acquisition unit 410 is configured to acquire information indicating a communication status (e.g., traffic volume) between the communication network 500 and the communication network 600 from the gateway device 300. For example, the communication status acquisition unit 410 may periodically acquire the communication status from the gateway device 300 at a predetermined time interval. In this way, the actual communication status between the communication network 500 and the communication network 600, which changes from moment to moment, is acquired by the communication control device 400.

[0023] The reward management unit 420 is configured to manage reward values ​​used in learning (reinforcement learning) of whether or not to cause the communication terminal 200 to communicate. Specifically, the reward management unit 420 manages (for example, stores in a storage unit such as a database) a reward value (first reward value) corresponding to the case where the communication terminal 200 performs communication and a reward value (second reward value) corresponding to the case where the communication terminal 200 does not perform communication, for each time slot obtained by dividing a predetermined control period (for example, one day) into many short periods each having a short time width.

[0024] Each time slot typically has an equal time duration, but may have unequal time durations. Exemplary time durations of the time slots may be any suitable length, such as a few seconds, tens of seconds, minutes, or tens of minutes.

[0025] The reward value is set so that the first reward value is greater than the second reward value for a time slot in which it is desirable for the communication terminal 200 to communicate, and the second reward value is greater than the first reward value for a time slot in which it is desirable for the communication terminal 200 not to communicate. For example, in an example in which it is known in advance that the amount of communication data increases in the daytime (e.g., 11:00 to 13:00) or in the evening (e.g., 17:00 to 20:00), the first reward value is set to, for example, "1" and the second reward value is set to, for example, "2" for time slots belonging to these time slots in order to advance reinforcement learning so that the communication terminal 200 does not communicate. In this case, the first reward value is set to, for example, "2" and the second reward value is set to, for example, "1" for time slots belonging to time slots other than these time slots in order to advance reinforcement learning so that the communication terminal 200 communicates. Such reward value settings can be appropriately set in advance as fixed values ​​in consideration of known time fluctuations in the amount of communication data.

[0026] 3 is an example of a management table for managing reward values ​​by reward management unit 420. In this example, time slots T1, T4, and T6 belong to a time period in which the amount of communication data is relatively small, and time slots T2, T3, and T5 belong to a time period in which the amount of communication data increases, and the respective reward values ​​are set to the values ​​exemplified above.

[0027] The reward value may be dynamically changed according to the actual communication situation, instead of being set to a fixed value as in the above example. For example, the reward management unit 420 may modify the reward value in an existing management table as shown in FIG. 3 based on the traffic volume of the communication network 500 acquired by the communication situation acquisition unit 410, or may create a new management table. More specifically, the reward management unit 420 may modify or set the first reward value to "1" and the second reward value to "2" for a time slot in which the traffic volume of the communication network 500 is higher than a predetermined threshold, while modifying or setting the first reward value to "2" and the second reward value to "1" for a time slot in which the traffic volume of the communication network 500 is lower than a predetermined threshold. In this way, by using a management table in which each reward value is adjusted to a value reflecting the actual communication situation, the accuracy of learning can be improved and a highly practical learning model 440 can be generated. This allows communication control for the communication terminal 200 to be performed appropriately.

[0028] 4 is a schematic diagram showing state transitions of the communication terminal 200, which is an object of control by the communication control device 400. This state transition diagram is managed (stored) by, for example, the learning model generation unit 430 of the communication control device 400, and is used to generate the learning model 440. The state of the communication terminal 200 changes from state s1 to state s2, from state s2 to state s3, ..., state s t From state s t+1 In this embodiment, the "state" of the communication terminal 200 is the position of a time slot. That is, the state of the communication terminal 200 transitions sequentially from the current time slot to the next time slot in time. In each state (i.e., each time slot), the communication terminal 200 can take either of actions a1 and a2. For example, the communication terminal 200 is in state s t-1 In (a certain time slot), action a2 is performed to create a new state s t (the next time slot), and then state s t In the state s t+1(the next time slot in time). In this embodiment, the behavior a1 of the communication terminal 200 may be that the communication terminal 200 performs communication, and the behavior a2 of the communication terminal 200 may be that the communication terminal 200 does not perform communication.

[0029] Returning to FIG. 2, the learning model generation unit 430 of the communication control device 400 is configured to generate a learning model 440 for determining whether or not the communication terminal 200 should actually perform communication by reinforcement learning based on the state of the communication terminal 200 and a reward value corresponding to each action that the communication terminal 200 can take in that state. For example, as the learning model 440, as shown in FIG. 5, t (identifier that identifies the time slot) is input to the input layer, and state s t The action value Q(s) for each of all actions that the communication terminal 200 can take, i.e., actions a1 and a2, is t ,a t ) from each node of the output layer. Alternatively, as shown in FIG. 6, the learning model 440 may be based on a Fixed Target Q-Network that uses two neural networks (main network and target network) with the same configuration as in FIG. 5. In the DQN in FIG. 5 and the Fixed Target Q-Network in FIG. 6, the action value Q(s t ,a t ) and the reward value provided by the reward management unit 420 to update the parameters (weights and thresholds) of the neural network, thereby training the learning model 440 (i.e., generating the learning model 440).

[0030] In the above example in which the reward value is dynamically changed according to the actual communication situation, the learning model 440 may be successively updated by, for example, repeatedly training the model every time the reward value is changed. This allows communication control for the communication terminal 200 to be performed more appropriately.

[0031] In addition, well-known methods can be applied as a method for learning neural networks in DQN and Fixed Target Q-Network. Therefore, in this specification, a description of the specific learning procedure is omitted.

[0032] The control information creation unit 450 is configured to use the learning model 440 generated by the learning model generation unit 430 to create control information indicating whether or not the communication terminal 200 should perform communication in each time slot. Specifically, the control information creation unit 450 inputs an identifier of each time slot to the input layer of the learning model 440. The learning model 440 calculates, for each time slot, an action value Q(s t ,a t ) from each node of the output layer. The control information creation unit 450 outputs the action value Q(s t ,a t ), and creates control information based on the action corresponding to the selected action value (i.e., either action a1 or a2). For example, if the action value corresponding to action a1 is greater than the action value corresponding to action a2 among the action values ​​output from the learning model 440, the control information creation unit 450 creates control information instructing the communication terminal 200 to perform communication in that time slot, and if the action value corresponding to action a2 is greater than the action value corresponding to action a1, the control information creation unit 450 creates control information instructing the communication terminal 200 not to perform communication in that time slot.

[0033] Control information transmitting unit 460 transmits the control information thus determined to communication terminal 200. Communication terminal 200 operates to perform communication or refrain from performing communication for each time slot in accordance with this control information. This makes it possible to control communication terminal 200 to perform communication in an appropriate time period or time slot, thereby making it possible to realize optimization and equalization of bandwidth usage in the communication network.

[0034] Although the embodiment of the present invention has been described above, the present invention is not limited to this, and various modifications are possible without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0035] 10 Network Systems 100 user terminals 200 Communication terminal 300 Gateway device 400 Communication control device 410 Communication status acquisition unit 420 Compensation Management Department 430 Learning model generation unit 440 Learning Model 450 Control Information Creation Department 460 Control information transmission unit 500 Communication Network 600 Communication Network

Claims

1. A communication control device that controls whether a communication terminal performs communication or not for each time slot among a plurality of time slots, a reward management unit that manages, for each of the time slots, a first reward value for the communication terminal performing communication and a second reward value for the communication terminal not performing communication; a control information generating unit that generates control information indicating whether or not the communication terminal should perform communication for each time slot, using a learning model generated by reinforcement learning based on the first and second reward values; a transmission unit that transmits the created control information to the communication terminal; A communication control device comprising:

2. A communication status acquisition unit that acquires a communication status in a network to which the communication terminal is connected, the reward management unit sets the first reward value smaller than the second reward value for a time slot of high traffic, and sets the first reward value larger than the second reward value for a time slot of low traffic, based on the communication status; The communication control device according to claim 1 .

3. the communication status acquisition unit acquires the communication status at a predetermined time interval; the reward management unit updates the first and second reward values ​​for each time slot in accordance with the communication conditions acquired at the predetermined time intervals; the learning model is successively updated using the updated first and second reward values; The communication control device according to claim 2.

4. The communication control device according to claim 1 , wherein the first and second reward values ​​are preset fixed values.

5. The learning model converts the identifier that identifies the time slot into a state s ｔ and each action a of the communication terminal performing communication or not performing communication is input as ｔ Action value Q(s ｔ , a ｔ The communication control device according to claim 1 , wherein the model is configured by a neural network based on deep reinforcement learning, and outputs a signal representing the state of the communication.

6. The control information creation unit is configured to: ｔ The action value for the communication terminal is an action a ｔ 6. The communication control device according to claim 5, wherein when the action value for a time slot is greater than the action value for a time slot, the control information is generated to instruct that the communication terminal should not carry out communication for the time slot.

7. A communication control method for controlling whether a communication terminal performs communication or not for each time slot among a plurality of time slots, comprising: assigning, for each time slot, a first reward value for the communication terminal to perform communication and a second reward value for the communication terminal not to perform communication; creating control information indicating whether or not the communication terminal should perform communication for each time slot, using a learning model generated by reinforcement learning based on the first and second reward values; transmitting the created control information to the communication terminal; The method includes:

8. The learning model converts the identifier that identifies the time slot into a state s ｔ and each action a of the communication terminal performing communication or not performing communication is input as ｔ Action value Q(s ｔ , a ｔ 8. The method according to claim 7, wherein the model is configured by a neural network based on deep reinforcement learning, which outputs a

Images