Communication management device and communication management method

Abstract

The present invention aims to provide an optimal communication route in a relay network between a gateway of a mobile communication network and a specific external network resource. [Solution] The communication management device 1 includes a first learning unit 12 that applies a reward function to an estimated result of calculating the route selection for the relay devices 50 that a packet from the user terminal 2 should pass through sequentially from the UPF 40 via the relay network 5 to reach the cloud 6, and updates the route selection so as to maximize the reward for the packet from the user terminal 2 to reach the cloud 6, and learns a first route selection strategy for the relay devices 50 that the packet from the user terminal 2 should pass through sequentially from the UPF 40, using a first reinforcement learning model, and a second learning unit 13 that learns the relationship between the node the packet is currently passing through and the first route selection strategy obtained by learning by the first learning unit 12, using a first supervised learning model.

Description

[Technical field]

[0001] The present invention relates to a communication management device and a communication management method. [Background technology]

[0002] Conventionally, in a mobile communication network, MEC (Multi-access Edge Computing) is known as a method for providing cloud computing functions at a location physically close to a user terminal (see Patent Document 1).

[0003] For example, Patent Document 2 discloses a technology that, when a user terminal accesses a specific website, identifies the base station to which the user terminal will connect, and controls communications so that the user terminal communicates with the cloud at the cloud site that is the shortest physical distance from the identified base station.

[0004] However, the techniques described in Patent Documents 1 and 2 do not take into consideration an efficient route of a relay network that exists between a gateway such as a user plane function (UPF) of a core network and a cloud site or an edge site to which a user terminal is connected. For example, in the technique disclosed in Patent Document 2, in order for a user terminal to access a specific website, a route that has a shorter physical distance between the base station in which the user terminal is located, the UPF of the core network, and the cloud site where the specific website is hosted is applied to the user terminal. However, Patent Document 2 does not take into consideration efficient routing of a relay network that exists between the UPF and the cloud site and is composed of a group of relay nodes. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-160813 A [Patent Document 2] Patent No. 7481595 Summary of the Invention [Problem to be solved by the invention]

[0006] In the conventional technology, the optimal communication route of the relay network between the gateway of the mobile communication network and the external network resource is not taken into consideration.

[0007] The present invention has been made to solve the above-mentioned problems, and has an object to provide an optimal communication route in a relay network between a gateway of a mobile communication network and an external network resource. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, a communication management device according to the present invention is a communication management device that manages a communication route from a user terminal to a specific external network resource via a gateway provided in a core network of a mobile communication network and via a relay network to which a plurality of relay nodes are connected, and includes an acquisition unit configured to acquire information on a gateway with which the user terminal is communicating, among a plurality of gateways provided in the core network, and information on a relay node through which a packet from the user terminal passes at each time, as information on a node through which a packet from the user terminal is currently passing; and a calculation unit that calculates a route selection for the relay nodes through which a packet from the user terminal should pass in sequence from the gateway through the relay network to reach the specific external network resource. The network comprises a first learning unit configured to apply a reward function to the estimation result to update so as to maximize a reward for the packet from the user terminal to reach the specific external network resource, and to learn, using a first reinforcement learning model, a first route selection policy for relay nodes that the packet from the user terminal should pass through sequentially from the gateway; a second learning unit configured to learn, using a first supervised learning model, a relationship between the node currently passed by the packet from the user terminal and the first route selection policy for the relay nodes that the packet from the user terminal should pass through sequentially from the gateway, obtained by learning by the first learning unit; and a memory unit configured to store the learned first supervised learning model constructed by the second learning unit.

[0009] In addition, in the communication management device of the present invention, the acquisition unit may be configured to acquire information about the gateway through which the managed user terminal is communicating and information about the relay nodes through which packets from the managed user terminal pass at each time as information about the node through which the packet from the managed user terminal is currently passing, and further to provide the information about the node through which the packet from the managed user terminal is currently passing as an unknown input to the learned first supervised learning model, perform calculations of the learned first supervised learning model, and output the first route selection strategy for the relay nodes through which packets from the managed user terminal should pass sequentially from the gateway, and a communication route management unit configured to notify the gateway and the multiple relay nodes of route information determined based on the first route selection strategy output by the calculation unit.

[0010] In addition, in the communication management device of the present invention, the acquisition unit acquires information about relay nodes through which packets addressed to the user terminal from the specific external network resource pass at each time as information about the node through which the packet addressed to the user terminal is currently passing, the first learning unit applies a reward function to an estimation result obtained by calculating a route selection for the relay nodes through which the packet addressed to the user terminal should pass sequentially before reaching the gateway via the relay network, and updates the route selection so as to maximize the reward for the packet addressed to the user terminal to reach the gateway, and learns a second route selection policy for the relay nodes through which the packet addressed to the user terminal should pass sequentially from the specific external network resource using a second reinforcement learning model, the second learning unit learns the relationship between the node through which the packet addressed to the user terminal is currently passing and the second route selection policy for the relay nodes through which the packet addressed to the user terminal should pass sequentially from the specific external network resource, obtained by learning by the first learning unit, using a second supervised learning model, and the memory unit stores the learned second supervised learning model constructed by the second learning unit.

[0011] In addition, in the communication management device of the present invention, the acquisition unit acquires information about relay nodes through which packets destined for the managed user terminal from the specific external network resource pass at each time as information about the node through which the packets destined for the managed user terminal are currently passing, the calculation unit provides the information about the node through which the packets destined for the managed user terminal are currently passing as an unknown input to the trained second supervised learning model, performs calculations of the trained second supervised learning model, and outputs the second route selection strategy for relay nodes through which packets destined for the managed user terminal should pass sequentially from the specific external network resource, and the communication route management unit may notify the specific external network resource and the multiple relay nodes of the route information determined based on the second route selection strategy output by the calculation unit.

[0012] In addition, in the communication management device of the present invention, the user terminals are multiple user terminals assigned common identification information, and the gateways through which each user terminal communicates include different gateways, the acquisition unit acquires information about nodes through which packets from each user terminal are currently passing, the first learning unit learns the first route selection policy for packets from each user terminal using the first reinforcement learning model, and the second learning unit learns the relationship between the nodes through which packets from each user terminal are currently passing and the first route selection policy for relay nodes through which packets from each user terminal should be sequentially passed from the gateways through which each user terminal communicates learned by the first learning unit using the first supervised learning model.

[0013] In addition, in the communication management device of the present invention, the gateway is a user plane function, and further includes a management information storage unit configured to store management information associating the common identification information, subscriber identification information of the multiple user terminals, and identification information of the specific external network resource, and a setting unit configured to specify the management information based on a location registration request signal from each of the user terminals and instruct the setting of the user plane function of each of the multiple user terminals, wherein the management information storage unit further stores identification information of the user plane function of each of the multiple user terminals set in accordance with an instruction of the setting unit in association with the management information, and the identification information of the user plane function may indicate information of the gateway with which each of the user terminals is communicating.

[0014] In addition, in the communication management device of the present invention, the relay network has a hierarchical structure in which relay node groups with common network addresses are connected, and each of the first route selection strategy and the second route selection strategy may include a probabilistic route selection strategy from any relay node of the relay node group having a first network address through which the packet passes at a first time to each relay node of the relay node group having a second network address through which the packet should pass at the next second time.

[0015] In order to solve the above-mentioned problems, a communication management method according to the present invention is a communication management method for managing a communication route from a user terminal to a specific external network resource via a gateway provided in a core network of a mobile communication network and via a relay network to which a plurality of relay nodes are connected, the method including an acquisition step of acquiring information on a gateway with which the user terminal is communicating, among a plurality of gateways provided in the core network, and information on relay nodes through which packets from the user terminal pass at each time, as information on nodes through which packets from the user terminal are currently passing; and a calculation step of selecting a route for relay nodes through which packets from the user terminal should pass in sequence from the gateway through the relay network to reach the specific external network resource. the first learning step of applying a reward function to the estimated result to update so as to maximize a reward for the packet from the user terminal to reach the specific external network resource, and learning a first route selection strategy for relay nodes that the packet from the user terminal should pass through in sequence from the gateway, using a first reinforcement learning model; a second learning step of learning, using a first supervised learning model, a relationship between the node currently passed by the packet from the user terminal and the first route selection strategy for relay nodes that the packet from the user terminal should pass through in sequence from the gateway, obtained by learning in the first learning step; and a storage step of storing the learned first supervised learning model constructed in the second learning step in a storage unit.

[0016] In addition, in the communication management method of the present invention, the acquisition step may include a calculation step of acquiring information about a gateway through which a managed user terminal is communicating and information about relay nodes through which packets from the managed user terminal pass at each time as information about the node through which the packet from the managed user terminal is currently passing, and further providing the information about the node through which the packet from the managed user terminal is currently passing as an unknown input to the trained first supervised learning model, and performing calculations of the trained first supervised learning model to output the first route selection strategy for the relay nodes through which the packet from the managed user terminal should pass sequentially from the gateway, and a communication route management step of notifying the gateway and the multiple relay nodes of route information determined based on the first route selection strategy output in the calculation step.

[0017] In addition, in the communication management method according to the present invention, the acquisition step acquires information on relay nodes through which packets addressed to the user terminal from the specific external network resource pass at each time as information on nodes through which packets addressed to the user terminal are currently passing, and the first learning step applies a reward function to an estimation result obtained by calculating a route selection for relay nodes through which the packets addressed to the user terminal should pass in sequence before reaching the gateway via the relay network, and updates the route selection so as to maximize a reward for the packets addressed to the user terminal to reach the gateway, The second learning step may learn a second route selection policy for relay nodes that a packet destined for the user terminal should pass through in sequence from the specific external network resource, using a second reinforcement learning model; the second learning step may learn a relationship between the node through which a packet destined for the user terminal is currently passing and the second route selection policy for relay nodes that a packet destined for the user terminal should pass through in sequence from the specific external network resource, obtained by learning in the first learning step, using a second supervised learning model; and the storage step may store the learned second supervised learning model constructed in the second learning step in the storage unit.

[0018] In addition, in the communication management method of the present invention, the acquisition step may acquire information about relay nodes through which packets destined for the managed user terminal from the specific external network resource pass at each time as information about the node through which the packets destined for the managed user terminal are currently passing, the calculation step may provide the information about the node through which the packets destined for the managed user terminal are currently passing as an unknown input to the trained second supervised learning model, perform calculations of the trained second supervised learning model, and output the second route selection strategy for relay nodes through which packets destined for the managed user terminal should pass sequentially from the specific external network resource, and the communication route management step may notify the specific external network resource and the multiple relay nodes of route information determined based on the second route selection strategy output in the calculation step.

[0019] Furthermore, in the communication management method of the present invention, the user terminals may be a plurality of user terminals assigned with common identification information, and the gateways through which each user terminal communicates may include gateways that are different from one another, the acquisition step may acquire information on nodes through which packets from each of the user terminals are currently passing, the first learning step may learn the first route selection policy for packets from each of the user terminals using the first reinforcement learning model, and the second learning step may learn, using the first supervised learning model, a relationship between the nodes through which packets from each of the user terminals are currently passing and the first route selection policy for relay nodes through which packets from each of the user terminals should be sequentially passed from the gateways through which each of the user terminals communicates learned in the first learning step.

[0020] Furthermore, in the communication management method according to the present invention, the gateway is a user plane function, and the method further includes a management information storage step of storing, in a management information storage unit, management information that associates the common identification information, subscriber identification information of the multiple user terminals, and identification information of the specific external network resource, and a setting step of specifying the management information based on a location registration request signal from each of the user terminals and instructing setting of the user plane function of each of the multiple user terminals, wherein the management information storage step further associates identification information of the user plane function of each of the multiple user terminals that is set in accordance with the instruction in the setting step with the management information and stores it in the management information storage unit, and the identification information of the user plane function may indicate information of the gateway with which each of the user terminals is communicating. Effect of the Invention

[0021] According to the present invention, a relationship between a node through which a packet from a user terminal is currently passing and a first route selection policy for a relay node through which the packet from the user terminal should pass from a gateway is learned using a first supervised learning model, so that an optimal communication route of a relay network between a gateway of a mobile communication network and an external network resource can be provided. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram showing a configuration of a communication management system including a communication management device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a communication management system including a communication management device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram for explaining the structure of the first storage unit included in the communication management device according to the present embodiment. [Figure 4] FIG. 4 is a diagram for explaining the structure of the first storage unit included in the communication management device according to the present embodiment. [Diagram 5]FIG. 5 is a diagram for explaining the learning process by the first learning unit included in the communication management device according to the present embodiment. [Figure 6] FIG. 6 is a block diagram showing a configuration of a first learning unit included in the communication management device according to the present embodiment. [Figure 7] FIG. 7 is a diagram for explaining the learning process by the second learning unit included in the communication management device according to the present embodiment. [Figure 8] FIG. 8 is a block diagram showing a hardware configuration of the communication management device according to the present embodiment. [Figure 9] FIG. 9 is a sequence diagram showing the operation of the communication management system according to the present embodiment. [Figure 10] FIG. 10 is a flowchart showing the operation of the communication management device according to the present embodiment. [Figure 11] FIG. 11 is a sequence diagram showing the operation of the communication management system according to the present embodiment. [Figure 12] FIG. 12 is a flowchart showing the first learning process of the communication management device according to the present embodiment. [Figure 13] FIG. 13 is a flowchart showing the first learning process of the communication management device according to the present embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of the communication management device according to the present embodiment. [Figure 15] FIG. 15 is a sequence diagram showing the operation of the communication management system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] BEST MODE FOR CARRYING OUT THE PRESENT DISCLOSURE Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS.

[0024] [Communication management system configuration] First, with reference to FIG. 1, an overview of a communication management system including a communication management device 1 according to an embodiment of the present invention will be described.

[0025] The communication management system according to the present embodiment includes a communication management device 1, a user terminal 2, a base station 3, a core network 4, a relay network 5, and a cloud (external network resource) 6. The communication management system is compliant with a 5G mobile communication network, and manages uplink and downlink communication routes from the user terminal 2 through a UPF 40 (gateway) included in the core network 4 of the mobile communication network, and through the relay network 5 to which multiple relay devices 50 (relay nodes) are connected, to a specific cloud 6.

[0026] The communication management device 1 is connected to a core network 4, a plurality of relay devices 50 included in a relay network 5, and a cloud 6 via a network NW.

[0027] The user terminal 2 is realized as a mobile communication terminal such as a smartphone, a tablet computer, a laptop computer, etc. The user terminal 2 is equipped with a SIM, and the contract profile of the SIM stores the subscriber identification information of the user, and includes identifier information such as a subscriber identification number (IMSI: International Mobile Subscriber Identity) assigned to a mobile phone line contract, a telephone number of the subscriber user (MSISDN: Mobile Subscriber International Subscriber Directory Number), and a SIM card number (ICCID: Integrated Circuit Card Identifier). The user terminal 2 is uniquely identified by the IMSI.

[0028] The user terminal 2 is also assigned a terminal IP address that uniquely identifies the terminal. The IP address is assigned to the user terminal 2 through the SMF 41 after a session is established. In this embodiment, there are N user terminals 2 (N is a positive integer equal to or greater than 2), and each user terminal 2 is present in a communication area of ​​a different base station 3. The user terminals 2 access a specific cloud 6 via the relay network 5 from a UPF 40 designated according to the base station 3 and cloud 6 in which the user terminal 2 is present. Furthermore, the user terminal 2 receives a data packet from the specific cloud 6 via the relay network 5 and the UPF 40. In this embodiment, a plurality of user terminals 2 are grouped in advance based on a predetermined attribute, and a group ID (common identification information) is assigned.

[0029] The base station 3 is composed of a wireless base station compatible with the 5G system, and relays communications between the user terminal 2 present in the communication area and the core network 4. The base station 3 is connected to the core network 4 via a network such as a backhaul link. N base stations 3 (N is an integer equal to or greater than 2) are provided.

[0030] The core network 4 is connected to the communication management device 1 via a network NW such as a LAN or WAN. The core network 4 includes multiple UPFs (User Plane Functions) 40 in the U-plane. The core network 4 also includes an AMF (Access and Mobility Management Function) 41, a SMF (Session Management Function) 42, a PCF (Policy Control Function) 43, and a UDM (Unified Data Management) / UDR (Unified Data Repository) 44, which are nodes in the C-plane. Functional nodes in the U-plane and C-plane other than those described above that the core network 4 includes are omitted in the illustration.

[0031] The UPF 40 is a user plane function that processes packets between the base station 3 and a data network such as the Internet. The UPF 40 functions as a gateway between the core network 4 and an external data network. A plurality of UPFs 40 are provided in the core network 4. In this embodiment, each UPF 40 transmits packets from the user terminal 2 to the cloud 6 via the relay network 5. Also, each UPF 40 forwards packets transmitted from the cloud 6 and passed through the relay network 5 to the user terminal 2. The user terminal 2 communicates with a UPF 40 designated according to the base station 3 in its coverage area, the cloud 6 to be accessed, etc.

[0032] The UPF 40 has an IP address, which allows the UPF 40 to be uniquely identified. In this embodiment, as shown in Fig. 2, the multiple UPFs 40 provided at the ends of the core network 4, i.e., the U-plane of the mobile communication network, have one common network address. Each UPF 40 is connected to a relay device 50 that is a relay node constituting a relay network 5. Specifically, the UPF 40 has a connection configuration in which it is connected to each of the relay devices 50 at the first stage of the relay network 5 in a so-called full mesh.

[0033] The AMF 41 is an access and mobility management device, and manages the registration and wireless connection of the user terminal 2 that moves to each communication area.

[0034] The SMF 42 is a session management function that performs the establishment, modification, release, etc. of PDU (Packet Data Unit) sessions between the user terminal 2 and a data network such as the Internet. The SMF 42 sets an appropriate communication path for data communication between the user terminal 2 and the UPF 40 based on a PCC (Policy and Charging Control) policy from the PCF 43.

[0035] The PCF 43 determines QoS and policies and provides them to the SMF 42. The PCF 43 applies PCC rules according to the specifications of 3GPP (registered trademark), and in response to an instruction from the communication management device 1, creates a PCC policy for setting a communication path of the UPF 40 through which the user terminal 2 communicates.

[0036] The UDM / UDR 44 manages subscriber profiles, performs authentication, and manages mobility. In this embodiment, the UDM / UDR 44 stores a group ID assigned to each IMSI in the subscriber profile. Note that, although this embodiment illustrates an example in which the UDM / UDR 44 is configured as a single device, the UDM / UDR 44 may be a device in which the UDM and UDR are arranged separately.

[0037] The relay network 5 is a relay node group to which multiple relay nodes are connected. The relay network 5 is provided between the UPF 40 and an external cloud 6, and relays packets from the user terminal 2 and the cloud 6. The relay node is composed of relay devices 50, and each relay device 50 is uniquely identified by an IP address. Each relay device 50 is also connected to the communication management device 1 via the network NW. The relay device 50 includes a CPU, a memory, a communication interface (WAN / LAN), a routing table, etc., and receives, routes, and forwards packets from the user terminal 2 and the cloud 6 based on route information from the communication management device 1.

[0038] As shown in FIG. 2, the relay network 5 has a hierarchical structure in which relay devices 50 having a common network address are connected to each other. For example, in the example of FIG. 2, a plurality of UPFs 40 provided at the end of the U-plane of the core network 4 have a network (NW) address 1. The first stage relay device 50 group connected to the UPF 40 has a NW address 2. Furthermore, the relay device 50 group with the NW address 2 is connected to the next second stage relay device 50 group, and finally connected to the relay device 50 group with the NW address m in the mth stage. Furthermore, the relay device 50 group with the NW address m is connected to the cloud 6. The relay device 50 of each network address is connected to each of the relay devices 50 with the next network address.

[0039] Also, for example, a packet from a user terminal 2 in an uplink passes through a serving base station 3, and starts at a UPF 40 having a NW address 1, then passes through one of the n relay devices 50 at a NW address 2 that is the next NW address of the NW address 1. After that, the packet further passes through one of the n relay devices 50 at a subsequent NW address 3. In this way, for each network address, one of the relay devices 50 having the network address of the next hop is selected as the relay device 50 to pass through.

[0040] The communication management system manages the relay device 50 as the destination for each NW address from which a packet from the user terminal 2 is forwarded from the UPF 40 group of NW address 1 to NW addresses 2 to m in that order until the packet reaches the cloud 6. The communication management system also manages the relay device 50 as the destination for each NW address from which a packet from the cloud 6 is forwarded from the UPF 40 group of NW address m to NW addresses m-1 to 2 in that order until the packet reaches the UPF 40 with which the user terminal 2 is communicating.

[0041] The cloud 6 provides a predetermined website, a web application, and the like. The cloud 6 has a cloud base that is a geographical location or area where physical devices constituting the website, etc. are located. The cloud 6 can be an edge server such as a server, a data center, or an MEC server. In this embodiment, a plurality of user terminals 2 access a specific cloud 6 from the base station 3 and UPF 40 in which each of them is located. Furthermore, packets from the cloud 6 are transmitted to each of a plurality of user terminals 2 that are connected to different base stations 3 and UPFs 40.

[0042] The communication management system learns, for each network address, a route selection policy indicating which relay device 50 of the next network address the packet should pass through, by reinforcement learning, based on information about nodes such as the relay device 50 through which the packet is currently passing. Furthermore, using the route selection policy for each user terminal 2 obtained by reinforcement learning as teacher data, the communication management system learns the relationship between information about the node through which the packet is currently passing and the route selection policy for the relay device 50 through which the packet should next pass through, by using a supervised learning model.

[0043] Furthermore, the communication management system performs calculations of a learned supervised learning model using information on the node through which the packet is currently passing as unknown input for each of the uplink and downlink, and outputs a policy for selecting a route through the relay device 50 through which the packet should be passed in sequence. Then, the communication management system notifies the UPF 40, relay device 50, or cloud 6 in communication of route information determined based on the output route selection policy. The UPF 40, relay device 50, and cloud 6 forward the packet to the next relay device 50 based on the notified route information. Thus, the user terminal 2 can access the cloud 6 via the optimal communication route and receive data from the cloud 6.

[0044] [Function block of communication management device] 1, the communication management device 1 includes a first storage unit (management information storage unit) 10A, a setting unit 10B, an acquisition unit 11, a first learning unit 12, a second learning unit 13, a second storage unit (storage unit) 14, a third storage unit 15, a calculation unit 16, a decision unit 17, and a communication route management unit 18. The communication management device 1 manages the communication route of the relay network 5 in the uplink, from the user terminal 2 until the packet reaches the cloud 6 from the UPF 40, and the communication route of the relay network 5 in the downlink, from the cloud 6 until the packet reaches the UPF 40.

[0045] The first storage unit 10A stores management information that associates the group ID of the user terminal 2, the subscriber identification information of the user terminal 2, and the identification information of a specific cloud 6. FIGS. 3 and 4 are diagrams showing a table 10a of management information stored in the first storage unit 10A. The table 10a holds the values ​​of "group ID," "IMSI," "cloud IP address," and "UPF IP address." The group ID is an ID assigned as common identification information to a plurality of user terminals 2 that have been grouped in advance. The IMSI of each of the user terminals 2 that belong to the group can be identified by the group ID. In addition, the cloud 6 to which the user terminal 2 is to connect is specified in advance for each group ID.

[0046] 3, the value of "UPF IP address" is "null" because the communication path of UPF 40 for the data communication of the user terminal 2 has not yet been set. On the other hand, in table 10a in Fig. 4, the setting unit 10B has already set the communication path of UPF 40 for the data communication of the user terminal 2, so the IP address of UPF 40 during communication is held as the value of "UPF IP address".

[0047] The setting unit 10B designates management information based on a location registration request signal from each user terminal 2, and instructs the setting of the UPF 40 of each user terminal 2. Specifically, the setting unit 10B instructs the setting of a user plane function for data communication of each user terminal 2 based on a group ID associated with the IMSI (subscriber identification information) included in the location registration request signal. In detail, the setting unit 10B receives a location registration request signal from the user terminal 2 via the core network 4, and refers to the table 10a shown in FIG. 3. The location registration request signal includes the IMSI of the user terminal 2. The location registration request signal is transmitted to the core network 4 via the base station 3 when the user terminal 2 moves within a communication area, as a periodic location update, and when the power is turned on.

[0048] The setting unit 10B requests the PCF 43 in the core network 4 to create a PCC policy by specifying the group ID, the IMSI, and the IP address of the communication management device 1. In response to the creation request, the PCF 43, the SMF 42, and the UDM / UDR 44 in the core network 4 work together to set an appropriate communication path of the UPF 40 for data communication of the user terminal 2. When the communication path of the UPF 40 is set for data communication of the user terminal 2, the IP address of the UPF 40 related to the setting of the communication path is stored for each IMSI in the table 10a of the first storage unit 10A, as described in FIG.

[0049] The acquisition unit 11 acquires information on the UPF 40 with which the user terminal 2 is communicating, among multiple UPFs 40 (multiple gateways) included in the core network 4, and information on the relay device 50 (relay node) through which the packet from the user terminal 2 passes at each time, as information on the node through which the packet from the user terminal 2 is currently passing in the uplink. The node through which the packet is currently passing is a node that has received the packet but has not yet forwarded it to the next destination node.

[0050] Specifically, the acquisition unit 11 acquires the IP address of the UPF 40 or relay device 50 that is the source of the inquiry as information on the currently passed node from the inquiry about the next destination node transmitted from the UPF 40 or relay device 50 that received the packet from the user terminal 2. The acquisition unit 11 can acquire information on which relay device 50 in which group of relay devices 50 the packet is passed through from among the multiple network addresses shown in FIG.

[0051] In managing the downlink communication route, the acquisition unit 11 acquires the IP address of the relay device 50 through which a packet addressed to the user terminal 2 from the cloud 6 passes at each time, as information on the node through which the packet addressed to the user terminal 2 is currently passing. From an inquiry about the next destination node transmitted from the relay device 50 that received the packet addressed to the user terminal 2, the acquisition unit 11 acquires the IP address of the relay device 50 that sent the inquiry, as information on the node through which the packet is currently passing. Note that, at the time step of the first route selection in which the packet is forwarded from the cloud 6 to the next hop relay device 50, the acquisition unit 11 acquires the IP address of the cloud 6.

[0052] The first learning unit 12 applies a reward function to the estimated result of calculating the selection of the route of the relay devices 50 that a packet from the user terminal 2 should pass through sequentially from the UPF 40 through the relay network 5 to reach the cloud 6, and updates it so as to maximize the reward for the packet from the user terminal 2 to reach the cloud 6, and learns a first route selection strategy for the relay nodes that the packet from the user terminal 2 should pass through sequentially from the UPF 40 using a first reinforcement learning model.

[0053] The first learning unit 12 further applies a reward function to an estimation result obtained by calculating the selection of a route of the relay devices 50 through which a packet addressed to the user terminal 2 should pass in sequence before reaching the UPF 40 via the relay network 5, updates the route so as to maximize the reward for the packet addressed to the user terminal 2 to reach the UPF 40, and learns a second route selection policy for the relay devices 50 through which the packet should pass in sequence from the cloud 6, using a second reinforcement learning model. Note that, hereinafter, the first reinforcement learning model and the first route selection policy related to the uplink communication route, and the second reinforcement learning model and the second route selection policy related to the downlink may be collectively referred to as the reinforcement learning model and the route selection policy, respectively.

[0054] In this embodiment, as a route selection policy, an action a of routing a packet from the UPF 40, the relay device 50, or the cloud 6 through which the packet has just passed to each of the n relay devices 50 having the network address of the next hop is performed. n A packet is transferred from one relay device 50 among a group of relay devices 50 having the same network address to one of a group of relay devices 50 having a different network address that is connected downstream along the link direction.

[0055] Furthermore, the route selection policy includes a policy of probabilistic route selection from a relay device 50 having a first network address through which the packet passes at a first time to each of n relay devices 50 having a second network address through which the packet should pass at a next second time. Further, the policy includes a policy of probabilistic route selection from the relay device 50 having the selected second network address to each of a group of relay devices 50 having a third network address through which the packet should pass at a next third time.

[0056] The first learning unit 12 uses a neural network model including an input layer s, a hidden layer h, and an output layer q as the first reinforcement learning model and the second reinforcement learning model as shown in Fig. 5. In addition, as the neural network model, a state s t, and all the action value functions Q(s t ,a1), Q(s t ,a2), Q(s t ,a3), ,Q(s t ,a n-1 ), Q(s t ,a n We adopt Deep Q-Network (DQN), a neural network that outputs

[0057] More specifically, the first learning unit 12 provides the IP address of the node through which the packet is currently passing as an input to the neural network model, and performs calculations on the neural network model. Then, the first learning unit 12 performs behavior a of routing (selecting a route) the packet to each of the n relay devices 50 related to the network address of the next hop as a route selection from the node through which the packet is currently passing (UPF 40, relay device 50, or cloud 6) to the next relay device 50 through which the packet should pass. n The first estimate Q1 of the action value function that represents the expected cumulative value of future rewards obtained if

[0058] The reward is a state s, which is the IP address of the node (UPF 40, relay node 50, or cloud 6) through which the packet is currently routed, and an action a, which is the action of routing the packet to a specific relay node 50 among the group of relay nodes 50 that are the next hop. n , and the reward function r=r(s, a, s') for the next state s', which is the IP address of the next relay device 50 through which the packet will pass.

[0059] In this embodiment, the reward function in the first reinforcement learning model includes, as a variable, the degree of arrival of a packet from the UPF 40 with which the user terminal 2 communicates to the cloud 6. For example, when a packet of the user terminal 2 reaches the cloud 6 in the shortest distance by an action of being routed to a specific relay device 50, the reward, which is a scalar quantity, is set to a larger value. On the other hand, when a packet of the user terminal 2 moves away from the physical location of the cloud 6, a negative reward value (e.g., r=-1) can be given. Similarly, the reward function in the second reinforcement learning model includes, as a variable, the degree of arrival of a packet from the cloud 6 to the UPF 40 with which the user terminal 2 communicates.

[0060] Furthermore, the first learning unit 12 provides the IP address of the node through which the packet is currently passing as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function. The first learning unit 12 learns weight parameters of the neural network model so that the first estimated value Q1 becomes a target value calculated from the second estimated value Q2.

[0061] If the weight parameter of the neural network model is θ and the action value function is Q(s, a; θ), the minimization loss function for learning is given by the following equation (1). L(θ)=1 / 2{r+γmax a’ Q(s',a';θ)-Q(s,a;θ)} 2 (1)

[0062] In the above formula (1), r is the reward (immediate reward) and γ is the discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value at the state s' one step forward, that is, the second estimated value Q2. The target value is r+γmax a’ It is expressed as Q(s',a';θ).

[0063] The first learning unit 12 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above formula (1).

[0064] More specifically, the first learning unit 12 may employ a Fixed Target Q-Network that uses two neural networks, a main QN121 and a target QN123, as shown in FIG. 6. The main QN121 selects an optimal action and updates an action-value function Q. Meanwhile, the target QN123 estimates and evaluates the value of an action a' to be taken in the next state s' of the result of the action. The main QN121 and the target QN123 have neural networks with the same layer structure, but the parameter of the main QN121 is "θ" and the parameter of the target QN123 is "θ". - " is given by

[0065] The main QN 121 receives, as state s, the IP address of the node through which the packet is currently passing from the environment 120. The environment 120 is a mobile communication network system in which the user terminal 2 is located, and in this environment 120, by taking action a for routing to any of the relay devices 50 of each network address, the packet is forwarded to any of the relay devices 50 of the network address at the subsequent stage, transitioning to the next state s' and simultaneously acquiring a reward r from the environment 120.

[0066] The first learning unit 12 inputs the state s related to the IP address of the node through which the packet is currently passing to the main QN 121, and obtains the action value function Q(s, a; θ). The first learning unit 12 calculates the action a using, for example, the ε-greedy method, or calculates the optimal action argmax a Q(s, a; θ) is calculated. In the environment 120, the action argmax related to the optimal route selection at the current time is a Q(s, a; θ). The environment 120 routes packets with the action argmax aAs a result of Q(s, a; θ), the IP address of the routing destination relay device 50 is observed as the next state s', and a reward r is output. The experience data 124 stores the experience (s, a, r, s') output from the environment 120.

[0067] The first learning unit 12 calculates a loss function L in a DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L.

[0068] The first learning unit 12 periodically copies the weights of the main QN 121 to the target QN 123 to perform synchronization. The synchronization of the target QN 123 is performed less frequently than the update frequency of the weights of the main QN 121. The first learning unit 12 extracts experience from the experience data 124, inputs past states to the target QN 123, and calculates the estimated value max a’ Q(s', a';θ - The first learning unit 12 outputs the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ - ) is used to learn the weights of the main QN 121 in the DQN loss calculation 122.

[0069] The trained first reinforcement learning models for the uplink obtained by the training of the first training unit 12 are N trained first reinforcement learning models constructed for each of the N user terminals 2 that connect from multiple UPFs 40 that are different initial points to a specific cloud 6 that is the same final point. In addition, a trained second reinforcement learning model for the downlink is also constructed for each of the N user terminals 2. These N trained second reinforcement learning models indicate a route selection policy for an optimal communication route from the cloud 6 that is the same initial point to multiple UPFs 40 that are different final points. These trained first reinforcement learning models and trained second reinforcement learning models are used as supervised data for supervised learning by the second training unit 13.

[0070] The route selection strategy, the action value function Q(s t ,a1), Q(s t ,a2), Q(s t ,a3), ,Q(s t ,a n-1 ), Q(s t ,a n ) in which actions a1~a n represents the value of the cumulative reward expected when each of n relay devices 50 having the same network address is selected as the forwarding destination. As shown in FIG. 2, an uplink packet is routed from the UPF 40 (IP address: IP1) of NW address 1 to one of n relay devices 50 (IP addresses: IP21 to IP2n) of NW address 2. For example, a1=0.1, a2=0.1, a3=0.1, . . ., a5=0.6, . . ., a n = 0.2, the relay device 50 of the IP 26 with the largest probability value is selected.

[0071] The second learning unit 13 uses a first supervised learning model to learn the relationship between the node through which the packet from the user terminal 2 is currently passing and the first route selection policy for the relay node through which the packet should sequentially pass from the UPF 40, obtained by the learning of the first learning unit 12. In the case of the downlink, the second learning unit 13 uses a second supervised learning model to learn the relationship between the node through which the packet addressed to the user terminal 2 is currently passing and the second route selection policy for the relay device 50 through which the packet addressed to the user terminal 2 should sequentially pass from the cloud 6, obtained by the learning of the first learning unit 12. In the following, the first supervised learning model for the uplink and the second supervised learning model for the downlink may be collectively referred to as the supervised learning model.

[0072] FIG. 7 shows the structure of a neural network model adopted as an example of the first supervised learning model and the second supervised learning model used by the second learning unit 13. The neural network model includes an input layer x, a hidden layer h, and an output layer y. The second learning unit 13 provides the IP address of the node (UPF 40, relay device 50, or cloud 6) through which the packet is currently passing, and the IP address of the relay device 50 through which the packet will pass at each subsequent time t, to the input layer of the neural network model, applies an activation function to the weighted sum of the inputs, and passes the output determined by threshold processing to the output layer. Each output node of the output layer outputs a predicted output of the model corresponding to n action value functions Q that route the packet to each of n relay devices 50 having a common network address.

[0073] The second learning unit 13 introduces an objective function E shown in the following equation (2) to learn parameters of the neural network model so that a route selection policy for a relay device 50 through which a packet of the user terminal 2 should sequentially pass, which is a predicted value from the neural network model for the IP address of the node through which the packet is currently passing, becomes the value of an optimized route selection policy for the relay device 50 through which the packet should sequentially pass, which is reinforcement learned by the first learning unit 12.

[0074]

number

[0075] In the above formula (2), y1, y2, , y n indicates the predicted output value of each output node. Also, Y1, Y2, . . . , Y n is a correct label included in the teacher data, and here is n optimized action value functions Q for the n relay devices 50 through which the packet should pass next, for the IP address of the node through which the packet is currently passing, obtained by reinforcement learning by the first learning unit 12. Furthermore, as described above, the first learning unit 12 performs reinforcement learning for each of the N user terminals 2.

[0076] In the uplink, N trained first reinforcement learning models (first route selection policies) are constructed from N different initial point UPFs 40 to a specific cloud 6. Each of these N first reinforcement learning models further optimizes n action value functions Q(s t ,a1), Q(s t ,a2), Q(s t ,a3), ,Q(s t ,a n-1 ), Q(s t ,a n ) is given. In addition, in the downlink, N trained second reinforcement learning models (second route selection policies) are constructed from the cloud 6 relating to the same initial point to N UPFs 40 at different final points. Similarly, n further action value functions Q are given to each of the N trained second reinforcement learning models. The second learning unit 13 uses all of these action value functions Q as supervised data in supervised learning for each of the uplink and downlink.

[0077] In the configuration example of the neural network model in FIG. 7, training data is shown to the right of each output node that outputs a predicted output value of the model. n shows n action value functions Q. The second learning unit 13 adjusts weight parameters of the neural network related to the supervised learning model so that the objective function E is minimized, that is, becomes 0. The second learning unit 13 can optimize the objective function E by using a backpropagation method or the like.

[0078] The second learning unit 13 can learn the first supervised learning model related to the uplink communication route using one neural network model. In addition, in learning the second supervised learning model related to the downlink, the second learning unit 13 can learn using one neural network model by providing the input node with the IP address of the node through which the packet is currently passing and the IP address of the UPF 40 at the final point.

[0079] The second storage unit 14 stores a trained first reinforcement learning model and a trained second reinforcement learning model constructed by reinforcement learning by the first learning unit 12. In the present embodiment, the second storage unit 14 stores N trained first reinforcement learning models and N trained second reinforcement learning models.

[0080] The third storage unit 15 stores a trained first supervised learning model and a trained second supervised learning model constructed by supervised learning by the second learning unit 13. In the present embodiment, the third storage unit 15 stores one trained first supervised learning model and one trained second supervised learning model.

[0081] The calculation unit 16 provides information on nodes currently passing through packets from a user terminal (a user terminal to be managed) 2 as unknown input to the trained first supervised learning model, performs calculations on the trained first supervised learning model, and outputs a first uplink route selection policy for relay nodes through which packets from the user terminal 2 should pass in sequence from the UPF 40. The user terminal 2 to be managed means the user terminal 2 to be inferred.

[0082] In addition, the calculation unit 16 provides information about the nodes through which packets destined for the user terminal 2 are currently passing as unknown input to the trained second supervised learning model, performs calculations on the trained second supervised learning model, and outputs a second downlink route selection strategy for the relay nodes through which packets destined for the user terminal 2 should sequentially pass from the cloud 6.

[0083] The information on the node through which packets from the managed user terminal 2 are currently passing, and the information on the node through which packets destined for the managed user terminal 2 are currently passing, are the IP addresses of the UPF 40, the relay device 50, or the cloud 6 acquired by the acquisition unit 11.

[0084] The determination unit 17 determines the relay device 50 through which the packet should pass next from the node through which the packet is currently passing, based on the first route selection policy for the uplink output by the calculation unit 16. More specifically, based on the optimal first route selection policy output by the calculation unit 16, the determination unit 17 sets the IP address of the currently communicating UPF 40 or relay device 50 in state s t For each state s t In step 3, the relay device 50 for which the action value function Q has a maximum value is selected, thereby determining the relay devices 50 through which the action should be passed in sequence.

[0085] The determination unit 17 further determines, based on the second route selection policy for the downlink, the next relay device 50 through which the packet destined for the managed user terminal 2 should pass, from the node (cloud 6 or relay device 50) through which the packet is currently passing.

[0086] The communication route management unit 18 notifies the UPF 40 and the relay device 50 of the route information determined by the determination unit 17 based on the first uplink route selection policy output by the calculation unit 16. In addition, the communication route management unit 18 notifies the cloud 6 and the relay device 50 of the route information determined by the determination unit 17 based on the second downlink route selection policy output by the calculation unit 16.

[0087] More specifically, the communication route management unit 18 notifies the UPF 40, the relay device 50, or the cloud 6 through which the packet is currently passing of route information indicating the next destination relay device 50 via the network NW. The notified UPF 40, the relay device 50, or the cloud 6 updates the routing table and forwards the received packet to the next hop relay device 50.

[0088] [Hardware configuration of communication management device] Next, an example of a hardware configuration for realizing the communication management device 1 having the above-mentioned functions will be described with reference to FIG.

[0089] 8, the communication management device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output I / O 106 connected via a bus 101, and a program that controls these hardware resources. Furthermore, the communication management device 1 can include a display device 107 connected via the bus 101.

[0090] The processor 102 is realized by a CPU, a GPU, an FPGA, an ASIC, or the like.

[0091] The main memory device 103 prestores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 implement the functions of the communication management device 1, such as the setting unit 10B, the acquisition unit 11, the first learning unit 12, the second learning unit 13, the calculation unit 16, the determination unit 17, and the communication route management unit 18 shown in FIG.

[0092] The communication interface 104 is an interface circuit for connecting the communication management device 1 to various external electronic devices via a network.

[0093] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or a flash memory as the storage medium.

[0094] The auxiliary storage device 105 has a program storage area for storing the communication management program executed by the communication management device 1. It also has a program storage area for storing the reinforcement learning program executed by the communication management device 1. Furthermore, the auxiliary storage device 105 has an area for storing the supervised learning program. The first storage unit 10A, the second storage unit 14, and the third storage unit 15 described in FIG. 1 are realized by the auxiliary storage device 105. Furthermore, for example, it may have a backup area for backing up the above-mentioned data, programs, etc.

[0095] The input / output I / O 106 is an input / output device that inputs signals from an external device and outputs signals to an external device.

[0096] The display device 107 is configured with an organic EL display, a liquid crystal display, etc. The display device 107 can display information related to management of the communication route of the user terminal 2.

[0097] [Operation of the communication management system] Next, the operation of the communication management system including the communication management device 1 having the above-described configuration will be described with reference to the sequence of FIG.

[0098] FIG. 9 is a sequence diagram showing a process of setting a communication path for data communication between the user terminal 2 and the UPF 40 by the communication management system.

[0099] First, the cloud 6 transmits information in which a group ID, an IMSI, and an IP address of the cloud 6 are associated with each other to the communication management device 1 (step S100). Information related to grouping of the user terminals 2 is stored in advance in the cloud 6. Next, the first storage unit 10A of the communication management device 1 creates a table 10a (FIG. 3) based on the received group ID, IMSI, and IP address of the cloud 6 (step S101). After that, the setting unit 10B of the communication management device 1 transmits an instruction to the UDM / UDR 44 to provision a group ID using the IMSI as a key (step S102).

[0100] Thereafter, when starting data communication with the cloud 6, the user terminal 2 transmits a location registration request signal to the core network 4 (step S103). The location registration request signal includes the IMSI of the user terminal 2. When the UDM / UDR 44 receives the location registration request signal, the UDM / UDR 44 associates a group ID with the IMSI included in the received location registration request signal based on the information in the table 10a (FIG. 3) created in step S101, and transfers the location registration request signal to the communication management device 1 (step S104).

[0101] Next, the setting unit 10B of the communication management device 1 requests the PCF 43 to create a PCC policy by specifying the IMSI, group ID, and further the IP address of the device itself, contained in the location registration request signal received in step S104 (step S105). Then, the PCF 42 creates a PCC policy based on the specified requirements (step S106). The PCF 43 creates a PCC policy including information about which UPF 40 the data communication of the user terminal 2 passes through. The PCC policy specifies the optimal UPF 40 as a communication path for data communication from the user terminal 2 to the cloud 6.

[0102] Next, the PCF 43 transmits the created PCC policy to the SMF 42 (step S107). After that, the SMF 42 transmits setting information related to the user plane function, such as setting of a communication path defined by the PCC policy, to the UPF 40 specified by the PCC policy (step S108). Next, the UPF 40 registers the received setting information in a memory (step S109).

[0103] Next, UPF 40 transmits an ACK to SMF 42 to notify that the communication path setting has been applied (step S110). Furthermore, SMF 42 transmits an ACK to the communications management device 1 to notify that the communication path setting has been applied (step S111). After that, the communications management device 1 transmits an ACK to the user terminal 2 (step S112). Further after that, a communications path for data communication between the user terminal 2 and UPF 40 is established (step S113).

[0104] Next, the communication management device 1 executes a communication management process for managing the communication route of the relay network 5 in the uplink until the user terminal 2 accesses the cloud 6 from the UPF 40, and the communication route of the relay network 5 in the downlink until the data from the cloud 6 is received via the UPF 40 (step S114). After that, a data transfer process is performed between the UPF 40 and the cloud 6 via the optimal route based on the route information from the communication management device 1 (step S115).

[0105] [Operation of the communication management device] Next, the communication management process executed by the communication management device 1 will be described in detail with reference to Fig. 10 and Fig. 11. Fig. 10 is a sequence showing a learning process of a communication route from the UPF 40 until an uplink packet reaches the cloud 6. Fig. 11 is a sequence showing an optimal route selection process of an uplink packet using a trained first supervised learning model. Below, a case will be described in which each user terminal 2 accesses a specific cloud 6 from a different UPF 40.

[0106] First, in FIG. 10, when the UPF 40 with which the user terminal 2 is currently communicating notifies the communication management device 1 of the group ID, IMSI, and IP address of the UPF (own device) 40 of the user terminal 2, the communication management device 1 executes the following process.

[0107] First, the first storage unit 10A associates the IP address of the UPF 40 received from the UPF 40 with the IMSI in the table 10a (FIG. 3) and updates the table 10a (FIG. 4) (step S1). Next, the acquisition unit 11 acquires the IP address of the node through which the packet is currently passing (step S2). The acquisition unit 11 acquires the IP address of the UPF 40 held in the table 10a updated in step S1 as the pass-through node at the first time. In addition, as the IP address of the pass-through node at each subsequent time, the acquisition unit 11 acquires the IP address of the relay device 50 that received the packet and sent the inquiry for the next pass-through node.

[0108] Next, the first learning unit 12 applies a reward function to an estimation result obtained by calculating the selection of a route for the relay devices 50 through which a packet from the user terminal 2 should pass in sequence from the UPF 40 through the relay network 5 to reach the cloud 6, updates the route so as to maximize the reward for the packet from the user terminal 2 to reach the cloud 6, and learns a first route selection policy for the relay nodes through which the packet should pass in sequence from the UPF 40 using a first reinforcement learning model (first learning process) (step S3). In step S3, the learning process is performed for each of the grouped multiple user terminals 2. Details of the first learning process will be described later.

[0109] Thereafter, the second storage unit 14 stores the learned first reinforcement learning model obtained in step S3 (step S4). Next, the second learning unit 13 learns the relationship between the node through which the packet of the user terminal 2 is currently passing and the first route selection policy for the relay node through which the packet should pass from the UPF 40 in sequence, obtained by learning by the first learning unit 12, using a supervised learning model (second learning process) (step S5).

[0110] Specifically, when the IP address of the node through which the packet from the user terminal 2 is currently passing, i.e., the current state, is given as an input value to the first supervised learning model, the second learning unit 13 repeatedly adjusts and updates parameters such as weights and thresholds, and determines the values ​​of these parameters so that the error between the predicted output value of the first route selection strategy for the relay device 50 through which the packet should sequentially pass and the supervised data minimizes the objective function E in the above equation (2).

[0111] In step S5, the second learning unit 13 can determine parameters that minimize the objective function E by using the backpropagation method or the like. Since the packet passes through the UPF 40 at the initial time, the IP address of the UPF 40 is input to the first supervised learning model. Furthermore, the IP addresses of the relay devices 50 that the packet passes through at each subsequent time are input to the first supervised learning model, and supervised learning is performed.

[0112] Next, the third storage unit 15 stores the trained first supervised learning model constructed in step S5 (step S6). Through the above processing, the learning of the uplink communication route is completed.

[0113] Next, with reference to the sequence of FIG. 11, a process of selecting an optimal route for uplink packets by the communication management device 1 using the trained first supervised learning model will be described.

[0114] First, the user terminal 2 transmits a packet to the UPF 40 (step S200). The packet from the user terminal 2 includes the IMSI, the source IP address of the user terminal 2, and the destination IP address of the UPF 40. After that, the UPF 40, which receives the packet, transmits the group ID of the user terminal 2, the IMSI, and the IP address of the UPF 40 to the communication management device 1 (step S201). Next, the first storage unit 10A of the communication management device 1 associates the IP address of the UPF 40 received in step S201 with the IMSI in the table 10a, stores it, and updates it (step S202).

[0115] Next, the acquisition unit 11 of the communication management device 1 acquires the IP address of the node through which the packet is currently passing (step S203). The acquisition unit 11 acquires the IP address of the UPF 40 held in the table 10a updated in step S202 as the first time-based route node.

[0116] Thereafter, the calculation unit 16 provides the information on the nodes through which the packet from the user terminal 2 is currently passing, obtained in step S203, as an unknown input to the trained first supervised learning model, performs calculations on the trained first supervised learning model, and outputs a first route selection strategy for the relay nodes through which the packet from the user terminal 2 should pass sequentially from the UPF 40 (step S204).

[0117] Next, the decision unit 17 selects the relay device 50 that takes the action a with the maximum value of the n action value functions Q output in step S204, thereby deciding the relay device 50 through which the packet should be sequentially routed (step S205). Next, the communication route management unit 18 notifies the UPF 40 of the route information decided in step S205 (step S206). Specifically, the communication route management unit 18 transmits the IP address (IP21) of the next hop relay device 50 as the route information to the UPF 40.

[0118] Upon receiving the notification of the route information, UPF 40 forwards the packet to the specified destination relay device 50 (step S207). The forwarded packet contains the group ID and IMSI of the user terminal 2, the source IP address of the user terminal 2, and the IP address (IP 21) of the destination relay device 50. Next, the relay device 50 (IP 21) that received the packet from UPF 40 transmits the group ID and IMSI of the user terminal 2 to the communication management device 1 to inquire about the next relay node (step S208). After that, the process proceeds to step S203.

[0119] The acquisition unit 11 of the communication management device 1 acquires the IP address (IP 21) of the relay device 50 that sent the inquiry including the group ID and IMSI in step S208 as the IP address of the node through which the packet is currently passing (step S203). Furthermore, after the calculation process in step S204 and the route determination process in step S205, the communication route management unit 18 notifies the relay device 50 (IP 21) of the IP address (IP 31) of the next hop relay device 50, which is the determined route information (step S209).

[0120] Thereafter, the relay device 50 (IP 21) transfers the packet to the next relay device 50 (IP 31) (step S210). The packet contains the group ID and IMSI of the user terminal 2, the IP address of the source user terminal 2, and the IP address (IP 31) of the destination relay device 50. In this way, the packets from the user terminal 2 are transferred in sequence to the next hop relay device 50 for each network address.

[0121] Thereafter, when the last relay device 50 (IP N1) receives the packet, it transmits the group ID and IMSI of the user terminal 2 to the communication management device 1 to inquire about the next node to be passed through (step S211). Then, the processes from step S203 to step S205 are executed, and the IP address of the cloud 6 is notified to the relay device 50 (IP N1) as route information (step S212). Then, the relay device 50 (IP N1) forwards the packet to the cloud 6 (step S213). With the above processes, management of the uplink communication route is completed.

[0122] Next, the first learning process (step S3 in FIG. 10) by the communication management device 1 will be described with reference to the flowcharts in FIG. 12 and FIG. 13. First, as shown in FIG. 12, the above-mentioned step S2 (FIG. 10) is executed. After that, the first learning unit 12 inputs the IP address of the node through which the packet from the user terminal 2 is currently passing, which is the current state acquired in step S2, to the neural network model, performs calculations on the neural network model, and outputs a first estimated value Q1 of the action value function that represents the expected value of the cumulative value of future rewards obtained when each action is taken to route the packet from the node through which the packet from the user terminal 2 is currently passing to each of the n relay devices 50 as the relay devices 50 through which the packet should next pass (step S20).

[0123] Next, the acquisition unit 10 acquires the IP address of the relay device 50 through which the packet passed at the next time t as the next state s' (step S21). The IP address of the relay device 50 through which the packet passed next is based on information acquired by the acquisition unit 11 for each time step. Furthermore, the first learning unit 12 provides the IP address of the relay device 50 through which the packet passed next, acquired in step S21, as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function (step S22).

[0124] Next, the first learning unit 12 calculates a target value from the second estimated value Q2 (step S23). Then, the first learning unit 12 learns weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S24). Specifically, the first learning unit 12 updates the weight parameters of the neural network model so as to minimize the loss function of the above formula (1).

[0125] Thereafter, the processes from step S3 to step S24 are repeated until a trained reinforcement learning model is constructed for all of the plurality of user terminals 2 (step S25: NO). If learning is then performed for all of the plurality of user terminals 2 (step S25: YES), the second storage unit 14 stores the trained first reinforcement learning model obtained in step S24 (step S6).

[0126] Next, with reference to FIG. 13, the first learning process by the first learning unit 12 in the case where a Fixed Target Q-Network using two neural networks, a main QN 121 and a target QN 123, is adopted will be described.

[0127] The process of step S2 is similar to the process of the first learning process described in Fig. 10. After that, the first learning unit 12 provides the main QN 121 with the IP address of the node through which the packet from the user terminal 2 is currently passing, acquired in step S2, as an input, performs calculations of the neural network model, outputs the action value function Q, and calculates the route selection a of the relay device 50 through which the packet should next pass (step S120).

[0128] Next, the first learning unit 12 returns the action to the environment 120 according to the route selection a obtained in step S120, and obtains the IP address of the relay device 50 to which the packet has been forwarded, which is the next state s', and the reward r (step S121).

[0129] The first learning unit 12 stores the experience (s, a, r, a') obtained in step S121 in the experience data 124 (step S122). Next, the first learning unit 12 calculates a loss function L in a DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L (step S123). The first learning unit 12 repeats the processes from step S120 to step S123 a set number of times.

[0130] After that, the first learning unit 12 periodically copies the weights of the main QN 121 to the target QN 123 to synchronize them (step S124). The synchronization of the target QN 123 is performed less frequently than the update frequency of the weights of the main QN 121. Next, the first learning unit 12 extracts experience from the experience data 124, inputs past states to the target QN 123, and calculates the estimated value max a’ Q(s', a';θ - ) is output (step S126).

[0131] Next, the first learning unit 12 calculates the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ -) (step S127). Next, the first learning unit 12 calculates a loss function L in the DQN loss calculation 122 using the target value calculated in step S127 (step S128). Next, the first learning unit 12 learns the weights of the main QN 121 so as to minimize the loss given by the loss function L (step S129). After that, the first reinforcement learning models that have been trained for each of the multiple user terminals 2 are stored in the second storage unit 14 (step S6).

[0132] Here, the management of downlink communication routes by the communication management device 1 will be described with reference to the sequences of Fig. 14 and Fig. 15. Note that the following description will focus on processes that are different from the operation of the communication management device 1 in the management of uplink communication routes described in Fig. 10 and Fig. 11.

[0133] 14 is a flowchart showing the learning process of a downlink communication route. First, when the cloud 6 transmits the group ID, IMSI, and IP address of the cloud 6 of the user terminal 2 to the communication management device 1, the following process is executed. First, the setting unit 10B identifies the IP address of the UPF 40 with which the user terminal 2 is communicating based on the information received from the cloud 6 (step S30). Specifically, the IP address of the UPF 40 is identified by referring to the table 10a (FIG. 4).

[0134] Next, the acquisition unit 11 acquires the IP address of the node through which the packet is currently passing (step S31). The acquisition unit 11 acquires the IP address of the cloud 6 that transmitted the group ID and IMSI of the user terminal 2 as the IP address of the node at the initial time, and acquires the IP address of the node at each subsequent time from the relay device 50 (step S31).

[0135] Next, the first learning unit 12 performs a first learning process (step S32). After that, the second storage unit 14 stores the learned second reinforcement learning model (step S33). Next, the second learning unit 13 performs a second learning process (step S34). Then, the third storage unit 15 stores the learned second supervised learning model (step S35). The first learning process in step S32 can be executed as reinforcement learning using DQN and Fixed Target Q-Network, similar to the learning process of the uplink communication route described in FIG. 12 and FIG. 13.

[0136] Here, Fig. 15 is a sequence showing a route selection process for a downlink communication route using the trained second supervised learning model. As shown in Fig. 15, the cloud 6 transmits the group ID, IMSI, and IP address of the cloud 6 of the user terminal 2 to be managed to the communication management device 1 (step S300). Next, the setting unit 10B of the communication management device 1 identifies the IP address of the UPF 40 with which the user terminal 2 is communicating based on the information received in step S300 (step S301).

[0137] Next, the acquisition unit 11 acquires the IP address of the node through which the packet addressed to the user terminal 2 to be managed is currently passing (step S302). After that, the calculation unit 16 performs calculations of the learned second supervised learning model (step S303). Next, the determination unit 17 determines the relay device 50 to which the packet is to be forwarded based on the second route selection policy obtained in step S303 (step S304). Then, the communication route management unit 18 notifies the cloud 6, which is the current node of the packet, of route information indicating the relay device 50 to which the packet is to be forwarded determined in step S304 (step S305). The route information includes the IP address (IP N1) of the next hop relay device 50.

[0138] The cloud 6 that has received the route information transfers the packet to the designated relay device 50 (step S306). The packet includes the group ID and IMSI of the user terminal 2, the IP address of the cloud 6 that is the sender, and the IP address (IP N1) of the destination relay device 50. After that, the relay device 50 (IP N1) that has received the packet transmits the group ID and IMSI of the user terminal 2 to the communication management device 1 to inquire about the next relay node (step S307). Then, the processes from step S302 to step S304 are repeatedly executed. After that, the communication route management unit 18 of the communication management device 1 notifies the relay device 50 (IP N1) of the IP address of the next hop relay device 50, which is the route information (step S308). In this manner, the packets are transferred in sequence to the relay device 50 of the next hop.

[0139] Thereafter, the relay device 50 (IP 31) transmits the packet to the next hop relay device 50 (IP 21) according to the route information (step S309). The packet contains the group ID and IMSI of the user terminal 2, the source IP address of the cloud 6, and the destination IP address (IP 21) of the relay device 50. The relay device 50 (IP 21) that receives the packet transmits the group ID and IMSI of the user terminal 2 to the communication management device 1 to inquire about the next node to pass through (step S310). Thereafter, the process proceeds from step S302 to step S304.

[0140] Next, the communication route management unit 18 notifies the relay device 50 (IP 21) of the IP address of the next hop UPF 40, which is the route information (step S311). After that, the relay device 50 (IP 21) forwards the packet to the UPF 40 (step S310). Further after that, the UPF 40 transmits the packet to the user terminal 2 (step S313). Through the above processing, the downlink packet reaches the user terminal 2 from the cloud 6.

[0141] As described above, the communication management device 1 according to this embodiment learns, by reinforcement learning, an optimal route selection policy for the relay device 50 through which uplink and downlink packets should pass in sequence, and uses the route selection policy obtained by reinforcement learning as teaching data to learn the relationship between the IP address of the node through which the packet from the user terminal 2 is currently passing and the route selection policy for the relay device 50 through which the packet should pass in sequence, using a supervised learning model. Therefore, it is possible to provide an optimal communication route of the relay network between the gateway of the mobile communication network and a specific external network resource.

[0142] Furthermore, according to the communication management device 1 of this embodiment, when a plurality of user terminals 2 to which a group ID is assigned transmit packets to a specific cloud 6 from different UPFs 40 with which they are communicating, an optimal route selection policy for the relay network 5 is learned by reinforcement learning. Furthermore, using all of the route selection policies from the different UPFs 40 to the specific cloud 6 as teacher data, the relationship between information on the node through which the packet from the user terminal 2 is currently passing and the route selection policy is learned by supervised learning. Therefore, after grouping a plurality of user terminals 2, an optimal communication route can be provided for each group.

[0143] In the embodiment described above, the reinforcement learning model used by the first learning unit 12 is exemplified as DQN related to Fixed Target Q-Network composed of a multi-layer neural network, but the reinforcement learning model can be CNN, multi-layer perceptron, etc. In addition to DQN exemplified as a reinforcement learning model, Double DQN, Dueling DQN, Actor-Critic (AC) method, Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Q-learning, etc. can be used.

[0144] In the embodiment described above, a multilayer neural network is used as the supervised learning model used by the second learning unit 13. However, the supervised learning model may be a multilayer perceptron, a decision tree-based model such as a random forest, a support vector machine, or the like.

[0145] The above describes embodiments of the communication management device and communication management method of the present invention, but the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can make within the scope of the invention described in the claims are possible. [Explanation of symbols]

[0146] 1...communication management device, 10A...first memory unit, 10B...setting unit, 11...acquisition unit, 12...first learning unit, 13...second learning unit, 14...second memory unit, 15...third memory unit, 16...calculation unit, 17...determination unit, 18...communication route management unit, 2...user terminal, 101, 201...bus, 102...processor, 103...main memory unit, 104...communication interface, 105...auxiliary memory unit, 106...input / output I / O, 107...display device, 120...environment, 121...main QN, 122...DQN loss calculation, 123...target QN, 124...empirical data, NW...network.

Claims

1. A communication management device that manages a communication route from a user terminal to a specific external network resource via a gateway provided in a core network of a mobile communication network and a relay network to which a plurality of relay nodes are connected, comprising: an acquisition unit configured to acquire, among a plurality of gateways included in the core network, information on a gateway with which the user terminal is communicating, and information on the relay node through which a packet from the user terminal passes at each time, as information on a node through which a packet from the user terminal is currently passing; a first learning unit configured to apply a reward function to an estimation result obtained by calculating a route selection for the relay nodes through which a packet from the user terminal should pass in sequence from the gateway through the relay network to reach the specific external network resource, and to update the route selection so as to maximize a reward for the packet from the user terminal to reach the specific external network resource, and to learn, using a first reinforcement learning model, a first route selection policy for the relay nodes through which a packet from the user terminal should pass in sequence from the gateway; a second learning unit configured to learn, using a first supervised learning model, a relationship between the node through which a packet from the user terminal is currently passing and the first route selection policy for the relay node through which the packet from the user terminal should sequentially pass from the gateway, the relationship being obtained by learning by the first learning unit; a storage unit configured to store the trained first supervised learning model constructed by the second learning unit; A communication management device comprising:

2. 2. The communication management device according to claim 1, the acquisition unit acquires, as information on a node through which a packet from the user terminal to be managed is currently passing, information on a gateway with which the user terminal to be managed is communicating and information on the relay node through which a packet from the user terminal to be managed passes at each time; a calculation unit configured to provide information on the node currently being passed by a packet from the managed user terminal as an unknown input to the trained first supervised learning model, perform calculations on the trained first supervised learning model, and output the first route selection policy for the relay node through which the packet from the managed user terminal should be passed in sequence from the gateway; a communication route management unit configured to notify the gateway and the plurality of relay nodes of route information determined based on the first route selection policy output by the calculation unit; A communication management device comprising:

3. 3. The communication management device according to claim 2, the acquisition unit acquires information on the relay node through which a packet addressed to the user terminal from the specific external network resource passes at each time as information on a node through which a packet addressed to the user terminal is currently passing; the first learning unit applies a reward function to an estimation result obtained by calculating a route selection for the relay nodes through which a packet addressed to the user terminal should pass in sequence before reaching the gateway via the relay network, and updates the route selection so as to maximize a reward for the packet addressed to the user terminal to reach the gateway; and learns a second route selection policy for the relay nodes through which a packet addressed to the user terminal should pass in sequence from the specific external network resource, using a second reinforcement learning model; the second learning unit learns, using a second supervised learning model, a relationship between the node through which the packet destined for the user terminal is currently passing and the second route selection policy for the relay node through which the packet destined for the user terminal should sequentially pass from the specific external network resource, obtained by the learning by the first learning unit; The storage unit stores a trained second supervised learning model constructed by the second learning unit. A communication management device comprising:

4. 4. The communication management device according to claim 3, the acquiring unit acquires information on the relay node through which a packet addressed to the managed user terminal from the specific external network resource passes at each time as information on a node through which a packet addressed to the managed user terminal is currently passing; the calculation unit provides information on the node through which the packet addressed to the managed user terminal is currently passing as an unknown input to the trained second supervised learning model, performs calculations on the trained second supervised learning model, and outputs the second route selection policy for the relay node through which the packet addressed to the managed user terminal should sequentially pass from the specific external network resource; The communication route management unit notifies the specific external network resource and the plurality of relay nodes of route information determined based on the second route selection policy output by the calculation unit. A communication management device comprising:

5. 2. The communication management device according to claim 1, The user terminals are a plurality of user terminals to which common identification information is assigned, and gateways through which the respective user terminals communicate include gateways different from each other; The acquisition unit acquires information about a node through which a packet from each of the user terminals is currently passing, The first learning unit learns the first route selection policy for packets from each of the user terminals by using the first reinforcement learning model; The second learning unit learns, using the first supervised learning model, a relationship between the node through which the packet from each of the user terminals is currently passing and the first route selection policy for the relay node through which the packet from each of the user terminals should be sequentially passed from the gateway with which each of the user terminals communicates, which is learned by the first learning unit. A communication management device comprising:

6. 6. The communication management device according to claim 5, The gateway is a user plane function; a management information storage unit configured to store management information that associates the common identification information, subscriber identification information of the plurality of user terminals, and identification information of the specific external network resource; a setting unit configured to specify the management information based on a location registration request signal from each of the user terminals and to instruct setting of a user plane function of each of the user terminals; Equipped with The management information storage unit further stores identification information of a user plane function of each of the user terminals, the identification information being set in accordance with an instruction from the setting unit, in association with the management information; The identification information of the user plane function stored in the management information storage unit indicates information of a gateway with which each of the user terminals is communicating. A communication management device comprising:

7. 4. The communication management device according to claim 3, the relay network has a hierarchical structure in which relay node groups each having a common network address are connected to each other, Each of the first route selection policy and the second route selection policy includes a probabilistic route selection policy from any one of a group of relay nodes having a first network address through which a packet passes at a first time to each of the relay nodes having a second network address through which a packet should pass at a next second time. A communication management device comprising:

8. A communication management method for managing a communication route from a user terminal to a specific external network resource via a gateway provided in a core network of a mobile communication network and a relay network to which a plurality of relay nodes are connected, comprising: an acquisition step of acquiring information on a gateway with which the user terminal is communicating, among a plurality of gateways included in the core network, and information on the relay node through which a packet from the user terminal passes at each time, as information on a node through which a packet from the user terminal is currently passing; a first learning step of applying a reward function to an estimation result of calculating a route selection for the relay nodes through which a packet from the user terminal should pass in sequence from the gateway through the relay network to reach the specific external network resource, and updating the route selection so as to maximize a reward for the packet from the user terminal to reach the specific external network resource, and learning a first route selection policy for the relay nodes through which a packet from the user terminal should pass in sequence from the gateway using a first reinforcement learning model; a second learning step of learning, using a first supervised learning model, a relationship between the node through which the packet from the user terminal is currently passing and the first route selection policy for the relay node through which the packet from the user terminal should sequentially pass from the gateway, the relationship being obtained by learning in the first learning step; a storage step of storing the trained first supervised learning model constructed in the second learning step in a storage unit; A communication management method comprising:

9. 9. The method of claim 8, further comprising: The acquiring step acquires information on a gateway with which a managed user terminal is communicating and information on the relay node through which a packet from the managed user terminal passes at each time as information on a node through which a packet from the managed user terminal is currently passing; and a calculation step of providing information on the node through which the packet from the managed user terminal is currently passing as an unknown input to the trained first supervised learning model, and performing a calculation of the trained first supervised learning model to output the first route selection policy for the relay node through which the packet from the managed user terminal should sequentially pass from the gateway. a communication route management step of notifying the gateway and the plurality of relay nodes of route information determined based on the first route selection policy output in the calculation step; A communication management method comprising:

10. 10. The method of claim 9, further comprising: The acquiring step acquires information on the relay node through which a packet addressed to the user terminal from the specific external network resource passes at each time as information on a node through which the packet addressed to the user terminal is currently passing; The first learning step applies a reward function to an estimation result obtained by calculating a route selection for the relay nodes through which a packet addressed to the user terminal should pass in sequence before reaching the gateway via the relay network, thereby updating the route selection so as to maximize a reward for the packet addressed to the user terminal to reach the gateway; and learns a second route selection policy for the relay nodes through which a packet addressed to the user terminal should pass in sequence from the specific external network resource using a second reinforcement learning model; The second learning step learns, using a second supervised learning model, a relationship between the node through which the packet addressed to the user terminal is currently passing and the second route selection policy for the relay node through which the packet addressed to the user terminal should sequentially pass from the specific external network resource, which is obtained by learning in the first learning step; The storage step stores the trained second supervised learning model constructed in the second learning step in the storage unit. A communication management method comprising:

11. The method of managing communications according to claim 10, The acquiring step acquires information on the relay node through which a packet addressed to the managed user terminal from the specific external network resource passes at each time as information on a node through which a packet addressed to the managed user terminal is currently passing; The calculation step provides information on the node currently being passed through by a packet destined for the managed user terminal as an unknown input to the trained second supervised learning model, performs calculations on the trained second supervised learning model, and outputs the second route selection policy for the relay node through which the packet destined for the managed user terminal should be sequentially passed from the specific external network resource; The communication route management step notifies the specific external network resource and the plurality of relay nodes of route information determined based on the second route selection policy outputted in the calculation step. A communication management method comprising:

12. 9. The method of claim 8, further comprising: The user terminals are a plurality of user terminals to which common identification information is assigned, and gateways through which the respective user terminals communicate include gateways different from each other; The acquiring step acquires information about a node through which a packet from each of the user terminals is currently passing, the first learning step includes learning the first route selection policy for packets from each of the user terminals by using the first reinforcement learning model; The second learning step learns, using the first supervised learning model, a relationship between the node through which the packet from each of the user terminals is currently passing and the first route selection policy for the relay node through which the packet from each of the user terminals should sequentially pass from the gateway with which each of the user terminals communicates learned in the first learning step. A communication management method comprising:

13. 13. The method of claim 12, further comprising: The gateway is a user plane function; a management information storage step of storing management information in a management information storage unit, the management information being associated with the common identification information, the subscriber identification information of the plurality of user terminals, and the identification information of the specific external network resource; a setting step of specifying the management information based on a location registration request signal from each of the user terminals and instructing setting of a user plane function of each of the user terminals; Equipped with The management information storage step further includes storing, in the management information storage unit, identification information of a user plane function of each of the user terminals, which is set in accordance with the instruction in the setting step, in association with the management information; The identification information of the user plane function stored in the management information storage unit indicates information of a gateway with which each of the user terminals is communicating. A communication management method comprising:

Images