Route management system and route management method

Abstract

The objective is to allow optimal actions to be shared among multiple devices with a simpler configuration. [Solution] The route management system includes a first learning unit 20 that uses a first machine learning model to learn an optimal travel route within the communication area 30A that indicates the course that the first mobile terminal 2a should take sequentially from its position at each time based on the position information of the first mobile terminal within the communication area 30A at each time; a second learning unit 12 that uses a second machine learning model to learn the relationship between the current position of the first mobile terminal 2a and the optimal travel route within the communication area 30A obtained by learning by the first learning unit 20; and a notification unit 17 that notifies the second mobile terminal 2b of the learned second machine learning model as route management information for the communication area 30A in response to receiving a location registration request signal transmitted by the second mobile terminal 2b when it arrives in the communication area 30A.

Description

[Technical field]

[0001] The present invention relates to a route management system and a route management method. [Background technology]

[0002] Conventionally, there has been known a technique for learning the optimal behavior of a device such as a robot by using machine learning (see Patent Document 1). However, the technique described in Patent Document 1 does not take into consideration sharing the optimal behavior of one device with other devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-173643 A Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, it has been difficult to share optimal actions across multiple devices using a simpler configuration.

[0005] The present invention has been made to solve the above-mentioned problems, and has an object to allow a plurality of devices to share optimal actions with a simpler configuration. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the route management system of the present invention comprises a first learning unit configured to learn, using a first machine learning model, an optimal movement route within a first communication area that indicates a course that the first mobile terminal should take sequentially from its position at each time based on position information of the first mobile terminal within the first communication area at each time; a second learning unit configured to learn, using a second machine learning model, the relationship between the current position of the first mobile terminal and the optimal movement route within the first communication area obtained by learning by the first learning unit; a memory unit configured to store the learned second machine learning model constructed by learning by the second learning unit; and a notification unit configured to notify the second mobile terminal of the learned second machine learning model as route management information of the first communication area in response to receiving a second position registration request signal transmitted by the second mobile terminal when it arrives at the first communication area.

[0007] In addition, the route management system of the present invention may further include a route management device having the second learning unit and the notification unit, and the first mobile terminal may be equipped with the first learning unit, and the first mobile terminal may transmit the optimal travel route for the first communication area, obtained by learning by the first learning unit, to the route management device in response to a first location registration request signal emitted upon reaching a second communication area adjacent to the first communication area.

[0008] In addition, in the route management system of the present invention, the first location registration request signal transmitted by the first mobile terminal in the first communication area is associated with a first group identifier which is an identifier of a group to which the first mobile terminal belongs, the memory unit associates the first group identifier with the learned second machine learning model and stores it, and the notification unit may notify the second mobile terminal of the learned second machine learning model as the route management information when the identifier of the group associated with the second location registration request signal transmitted by the second mobile terminal when it reaches the first communication area is the first group identifier.

[0009] In addition, in the route management system of the present invention, a route management device may further include the first learning unit, the second learning unit, and the notification unit, and the route management device may further include a first acquisition unit configured to acquire location information of the first mobile terminal added to a first location registration request signal that the first mobile terminal transmits at each time in the first communication area, and the first learning unit may perform learning based on the location information of the first mobile terminal acquired by the first acquisition unit.

[0010] In addition, in the route management system of the present invention, the second mobile terminal may include a second acquisition unit configured to acquire the route management information notified by the notification unit, a third acquisition unit configured to acquire the current position of the terminal, a calculation unit configured to provide the current position of the terminal acquired by the third acquisition unit as an unknown input to the trained second machine learning model, perform calculations of the trained second machine learning model, and output the optimal movement route within the first communication area indicating a course to be taken sequentially from the current position of the terminal, and a movement control unit configured to control the movement of the terminal based on the optimal movement route within the first communication area output by the calculation unit.

[0011] In addition, in the route management system of the present invention, the first learning unit may learn the optimal movement route within the first communication area by reinforcement learning, which indicates the path that the first mobile terminal should take sequentially from the position of the first mobile terminal at each time from the position where it arrives at the first communication area to the position where it leaves the first communication area.

[0012] In order to solve the above-mentioned problems, the route management method of the present invention includes a first learning step of using a first machine learning model to learn an optimal movement route within a first communication area, which indicates a course that the first mobile terminal should take sequentially from its position at each time, based on position information of the first mobile terminal within the first communication area at each time; a second learning step of using a second machine learning model to learn the relationship between the current position of the first mobile terminal and the optimal movement route within the first communication area obtained by learning in the first learning step; a storage step of storing the trained second machine learning model constructed by learning in the second learning step in a memory unit; and a notification step of notifying the second mobile terminal of the trained second machine learning model as route management information of the first communication area in response to receiving a second position registration request signal transmitted by the second mobile terminal when it reaches the first communication area.

[0013] In addition, in the route management method of the present invention, a route management device may execute the second learning step and the notification step, the first mobile terminal may execute the first learning step, and the first mobile terminal may transmit the optimal movement route for the first communication area obtained by learning in the first learning step to the route management device in response to a first location registration request signal that is emitted upon reaching a second communication area adjacent to the first communication area.

[0014] Furthermore, in the route management method of the present invention, the first location registration request signal transmitted by the first mobile terminal in the first communication area is associated with a first group identifier which is an identifier of a group to which the first mobile terminal belongs, the storage step may associate the first group identifier with the learned second machine learning model and store it in the memory unit, and the notifying step may notify the second mobile terminal of the learned second machine learning model as the route management information if the identifier of the group associated with the second location registration request signal transmitted by the second mobile terminal when it reaches the first communication area is the first group identifier.

[0015] In addition, in the route management method of the present invention, the route management device executes the first learning step, the second learning step, and the notification step, and further includes a first acquisition step executed by the route management device to acquire location information of the first mobile terminal added to a first location registration request signal that the first mobile terminal transmits at each time in the first communication area, and the first learning step may perform learning based on the location information of the first mobile terminal acquired in the first acquisition step.

[0016] In addition, the route management method of the present invention may further include a second acquisition step executed by the second mobile terminal to acquire the route management information notified in the notification step, a third acquisition step to acquire the current position of the terminal, a calculation step to provide the current position of the terminal acquired in the third acquisition step as an unknown input to the trained second machine learning model, perform calculations of the trained second machine learning model, and output the optimal movement route within the first communication area indicating a course to be taken sequentially from the current position of the terminal, and a movement control step to control movement of the terminal based on the optimal movement route within the first communication area output in the calculation step.

[0017] In addition, in the route management method of the present invention, the first learning step may involve learning, by reinforcement learning, the optimal movement route within the first communication area, which indicates the path that the first mobile terminal should take sequentially from the position of the first mobile terminal at each time from the position where it arrives at the first communication area to the position where it leaves the first communication area. Effect of the Invention

[0018] According to the present invention, in response to receiving a second location registration request signal transmitted when the second mobile terminal arrives in the first communication area, the second machine learning model that has been trained is notified to the second mobile terminal as route management information within the first communication area. Therefore, with a simpler configuration, optimal actions can be shared among multiple devices. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram showing a configuration of a route management system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining an overview of the route management system according to the present embodiment. [Diagram 3] FIG. 3 is a diagram for explaining the structure of a setting information table provided in the route management system according to the present embodiment. [Figure 4] FIG. 4 is a block diagram showing a configuration of a mobile terminal provided in the route management system according to the present embodiment. [Diagram 5] FIG. 5 is a diagram for explaining an optimal travel route stored in the route management system according to the present embodiment. [Figure 6] FIG. 6 is a schematic diagram showing the configuration of the first learning unit according to the present embodiment. [Figure 7] FIG. 7 is a block diagram showing a configuration of the first learning unit according to the present embodiment. [Figure 8] FIG. 8 is a schematic diagram for explaining the configuration of the second learning process according to the present embodiment. [Figure 9] FIG. 9 is a block diagram showing a hardware configuration of the path management device according to the present embodiment. [Figure 10] FIG. 10 is a block diagram showing a hardware configuration of a mobile terminal according to this embodiment. [Figure 11] FIG. 11 is a sequence diagram showing the operation of the route management system according to the present embodiment. [Figure 12] FIG. 12 is a sequence diagram showing the operation of the route management system according to the present embodiment. [Figure 13] FIG. 13 is a sequence diagram showing the operation of the route management system according to the present embodiment. [Figure 14] FIG. 14 is a flowchart showing the first learning process of the route management system according to the present embodiment. [Figure 15]FIG. 15 is a flowchart showing the first learning process of the route management system according to the present embodiment. [Figure 16] FIG. 16 is a block diagram showing a configuration of a route management device included in a route management system according to a modified example of the present embodiment. [Figure 17] FIG. 17 is a block diagram showing a configuration of a mobile terminal provided in a route management system according to a modified example of the present embodiment. [Figure 18] FIG. 18 is a sequence diagram showing the operation of a route management system according to a modification of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] BEST MODE FOR CARRYING OUT THE PRESENT DISCLOSURE Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS.

[0021] [Route management system configuration] First, with reference to FIG. 1, an overview of a route management system including a route management device 1 and a mobile terminal 2 according to an embodiment of the present invention will be described.

[0022] The route management system according to the present embodiment includes a route management device 1 and a plurality of mobile terminals 2, and the route management device 1 and the mobile terminals 2 are communicably connected to each other via a wireless communication network NW that complies with a predetermined communication standard such as LTE / 4G, 5G, 6G, etc. In response to a location registration request signal transmitted by the mobile terminal 2 upon reaching the communication area 30A, the route management system notifies the mobile terminal 2 of route management information that indicates an optimal travel route within the communication area 30A that has been learned in advance.

[0023] The mobile terminal 2 includes an autonomous flying object such as a drone or an unmanned aerial vehicle, an automatic driving vehicle, a mobile robot, an excavator, a ship, etc. The mobile space in which the mobile terminal 2 moves is each communication area 30. In this embodiment, a first mobile terminal 2a and a plurality of second mobile terminals 2b are provided. The first mobile terminal 2a moves through the communication area 30A ahead of the other mobile terminals 2. The second mobile terminal 2b arrives at the communication area 30A after the first mobile terminal 2a. In the following, when the first mobile terminal 2a and the plurality of second mobile terminals 2b are not distinguished from each other, they are collectively referred to as "mobile terminals 2".

[0024] A group ID (group identifier) ​​is assigned to a plurality of mobile terminals 2 including a first mobile terminal 2a and a plurality of second mobile terminals 2b based on a predetermined attribute. The group ID can be assigned to terminals that have common attributes related to communication and services. The mobility management process of this embodiment is performed for each of a plurality of mobile terminals 2 having the same group ID.

[0025] In the following, an example will be described in which the mobile terminal 2 is a mobile robot. The mobile terminal 2 controls its autonomous movement using a controller that processes information from a sensor 208 (described later) and controls the rotation speed of a motor 209 and a drive mechanism 210. The mobile terminal 2 also acquires its own GPS position using a GPS receiver 207. The mobile terminal 2 is configured as an IoT terminal having an IP address, and each IP address can uniquely identify the mobile terminal 2.

[0026] Moreover, the mobile terminal 2 according to this embodiment includes a SIM and has an International Mobile Subscriber Identity (IMSI) stored in the SIM. The mobile terminal 2 is uniquely identified by the IMSI. When the mobile terminal 2 crosses a communication area 30 of a destination and at regular intervals, the mobile terminal 2 transmits a location registration request signal to the base station 3 and the core network 4. The location registration request signal includes the IMSI of the mobile terminal 2. The mobile terminal 2 also transmits the location registration request signal by adding its own terminal location information to it. Each of the mobile terminals 2 has the same configuration. The functional blocks and hardware configuration of the mobile terminal 2 will be described in detail later.

[0027] 1, a mobile terminal 2 moves so as to cross in order L communication areas 30A (first communication area) and 30B (second communication area) covered by L base stations 3A and 3B (L is an integer equal to or greater than 2). The communication area 30A is the communication area 30 immediately preceding the communication area 30B, which are any adjacent communication areas 30 among the L communication areas 30.

[0028] FIG. 2 shows the mobile space of the mobile terminal 2 corresponding to the communication area 30A. The mobile space is a three-dimensional matrix-like space composed of unit spaces divided into a plurality of spaces. Each unit space constituting the mobile space has the same volume. Furthermore, each unit space has a node ID, and each unit space is represented by one position (x, y, z). The position information can be three-dimensional GPS position coordinates consisting of latitude, longitude, and altitude. For example, a representative value such as the center position of the unit space can be used as the position of the unit space.

[0029] Also, as shown in FIG. 2, the mobile terminal 2 moves from a position in the unit space corresponding to the position of the initial point S where the communication area 30A is reached to a position in the unit space of the final point G where the communication area 30A is left, with each unit space as a waypoint. As an example, the final point G of each communication area 30A, 30B is set in advance. Furthermore, when the mobile terminal 2 leaves the communication area 30A and reaches the adjacent communication area 30B, it similarly moves to a position away from the position where the communication area 30B is reached. Thus, in this embodiment, the optimal movement route for each communication area 30 is learned in advance and shared by multiple mobile terminals 2 that arrive at each communication area 30. Also, the optimal movement route is learned by the first mobile terminal 2a, which moves through the communication area 30 first among the multiple mobile terminals 2, and is transmitted to the route management device 1 via the network NW.

[0030] The base station 3 is composed of a wireless base station compatible with the 5G communication standard, and relays communication between a mobile terminal 2 present in a communication area 30 and a core network 4. In this embodiment, the base station 3 has L base stations 3A and 3B (L is an integer of 2 or more), each of which covers a communication area 30A and 30B. The base station 3 is connected to the core network 4 via a backhaul link. In this embodiment, one base station 3 covers one communication area 30, and the base station 3 and the communication area 30 are identified by a base station ID. Hereinafter, when the base stations 3A and 3B and the communication areas 30A and 30B are not distinguished from each other, they may be collectively referred to as the "base station 3" and the "communication area 30", respectively.

[0031] The core network 4 is connected to the route management device 1 via a network NW such as a LAN, a WAN, or the Internet. The core network 4 includes an AMF (Access and Mobility Management Function) 40, a UDM (Unified Data Management) / UDR (Unified Data Repository) 41, a SMF (Session Management Function) 42, and a PCF (Policy Control Function) 43, which are nodes in the C-plane. The core network 4 also includes a plurality of UPFs (User Plane Functions) 44 in the U-plane. Functional nodes in the U-plane and C-plane other than those described above that the core network 4 includes are omitted from the illustration.

[0032] The AMF 40 is a node that provides a mobility control function and performs mobility control such as location registration, paging, and handover. The location information of the mobile terminal 2 added to the location registration request signal received by the AMF 40 is transmitted to the UDM / UDR 41.

[0033] The UDM / UDR 41 manages subscriber profiles, performs authentication, and manages mobility. In this embodiment, the UDM / UDR 41 adds a field for location information to the subscriber profile. The UDM / UDR 41 according to this embodiment includes a communication interface 41a for communicating with the route management device 1. The UDM / UDR 41 stores location information using the IMSI included in the location registration request signal as a key. The transmission timestamp of the location registration request signal identifies what time and at what GPS position the mobile terminal 2 identified by the IMSI is located.

[0034] In addition, the UDM / UDR 41 creates and stores a setting information table T1 in which the IMSIs, group IDs, base station IDs, and communication paths of the UPF 44 set in data communication of the mobile terminal 2 of multiple mobile terminals 2 are associated with each other, as shown in FIG. 3, in response to an instruction from the route management device 1. The instruction from the route management device 1 includes grouping information in which multiple IMSIs and group IDs of the management targets that are specified in advance are associated with each other. The value of the IP address of the UPF 44 in the setting information table T1 is registered after a communication path is set by the SMF 42 and PCF 43 described later. The UDM / UDR 41 can configure the setting information table T1 by adding a group ID field to the subscriber profile. The setting information table T1 managed by the UDM / UDR 41 is also synchronized with the route management device 1, and the same contents are stored (fourth storage unit 16).

[0035] The UDM / UDR 41 may be configured such that the UDM and UDR are integrated into one device, or the UDM / UDR 41 may be configured such that the UDM and UDR are separately arranged.

[0036] The SMF 42 is a session management function that performs the establishment, modification, release, etc. of PDU (Packet Data Unit) sessions between the mobile terminal 2 and a data network such as the Internet. The SMF 42 sets an appropriate communication path for data communication between the mobile terminal 2 and the UPF 44 based on a PCC (Policy and Charging Control) policy from the PCF 43.

[0037] The PCF 43 determines QoS and policies and provides them to the SMF 42. The PCF 43 applies PCC rules according to the 3GPP (registered trademark) specifications, and in response to an instruction from the route management device 1, creates a PCC policy for setting a communication path of the UPF 44 through which the mobile terminal 2 communicates.

[0038] The UPF 44 is a user plane function that processes data between the base station 3 and a data network (DN) 5 such as the Internet. In this embodiment, the UPF 44 includes a communication interface 44a for communicating with the path management device 1.

[0039] The data network 5 is an external network such as the Internet or a cloud network.

[0040] In the route management system according to the present embodiment, a first mobile terminal 2a, which moves in advance through a communication area 30A, learns an optimal travel route within the communication area 30A through reinforcement learning. Furthermore, the optimal travel route within the communication area 30A obtained through reinforcement learning is transmitted to the route management device 1 in response to a location registration request signal transmitted by the first mobile terminal 2a when the first mobile terminal 2a reaches the next communication area 30B. The route management device 1 learns the relationship between the current location of the mobile terminal 2 and the optimal travel route within the communication area 30A learned by the first mobile terminal 2a, using a supervised learning model. Furthermore, the learned supervised learning model is notified to the second mobile terminal 2b as route management information for managing the route of the following second mobile terminal 2b.

[0041] Based on the notified route management information, the second mobile terminal 2b performs calculations of a trained supervised learning model with the current unit space position of the terminal as an unknown input, and outputs a course to be followed sequentially. Then, based on the output course to be followed sequentially, the second mobile terminal 2b determines a course from the initial point to the final point in the communication area 30A, and controls movement within the communication area 30A.

[0042] The second mobile terminal 2b, which has received the route management information, changes its course according to the determined course as shown by the arrows in FIG. 2, and moves in the direction to proceed for each unit space. The course can include various courses, i.e., movement directions. In FIG. 2, the movement space is described on a two-dimensional plane, but the course of the mobile terminal 2 can be a three-dimensional course. Here, the course refers to movement from the position of each unit space to the position of each adjacent unit space. Also, the route includes the entire route from the position of the initial point to the position of the final point.

[0043] [Route management device functional block] 1, the route management device 1 includes a setting unit 10, a route acquisition unit 11, a second learning unit 12, a first memory unit 13, a second memory unit (memory unit) 14, a third memory unit 15, a fourth memory unit 16, and a notification unit 17. The route management device 1 manages the route of a mobile terminal 2 for each communication area 30, which is a movement space.

[0044] The setting unit 10 designates an IMSI and a group ID based on a location registration request signal from each mobile terminal 2, and instructs the setting of the UPF 44 of each mobile terminal 2. Specifically, the setting unit 10 transmits to the UDM / UDR 41 grouping information in which the IMSIs and group IDs of the management targets registered in advance are associated with each other. Based on the grouping information from the setting unit 10, the UDM / UDR 41 creates a setting information table T1 (FIG. 3). Furthermore, the setting unit 10 instructs the setting of a user plane function for data communication of each mobile terminal 2 based on the group ID associated with the IMSI included in the location registration request signal from each mobile terminal 2.

[0045] The setting unit 10 requests the PCF 43 of the core network 4 to create a PCC policy by specifying the group ID, the IMSI, and the IP address of the route management device 1. In response to the creation request, the PCF 43, the SMF 42, and the UDM / UDR 41 of the core network 4 cooperate to set an appropriate communication path of the UPF 44 for data communication of the mobile terminal 2. When the communication path of the UPF 44 for data communication of the mobile terminal 2 is set, the setting information table T1 updated in the UDM / UDR 41 is synchronized with, for example, the fourth storage unit 16, and the updated setting information table T1 is stored in the fourth storage unit 16.

[0046] The route acquisition unit 11 acquires the optimal moving route in the communication area 30A learned by the first mobile terminal 2a. As described above, the first mobile terminal 2a is a terminal that moves through the communication areas 30A and 30B ahead of the other mobile terminals 2. The first mobile terminal 2a learns the optimal moving route in the communication area 30A while moving ahead of the other mobile terminals 2. The route acquisition unit 11 acquires the optimal moving route in the previous communication area 30A from the first mobile terminal 2a at the timing when the first mobile terminal 2a transmits a location registration request signal (first location registration request signal) when it arrives at the next communication area 30B. A group ID is associated with the location registration request signal transmitted by the first mobile terminal 2a.

[0047] The second learning unit 12 learns the relationship between the current position of the first mobile terminal 2a and the optimal movement route within the communication area 30A acquired by the route acquisition unit 11, using a supervised learning model (second machine learning model). The second learning unit 12 will be described in detail later.

[0048] The first storage unit 13 stores the optimal travel route in the communication area 30A learned and constructed by the preceding first mobile terminal 2a, acquired by the route acquisition unit 11. The first storage unit 13 stores the optimal travel route in association with a group ID. The first storage unit 13 may also store the optimal travel route in association with a base station ID.

[0049] The second storage unit 14 (storage unit) stores a trained supervised learning model constructed by supervised learning by the second learning unit 12. The second storage unit 14 stores the trained supervised learning model in association with a group ID. The second storage unit 14 may further store the trained supervised learning model in association with a base station ID.

[0050] The third storage unit 15 stores the final point of each of the communication areas 30A and 30B. The third storage unit 15 also stores position information of unit spaces constituting the movement space of the communication areas 30A and 30B, and identification information of the mobile terminal 2, such as the IMSI.

[0051] The fourth storage unit 16 stores grouping information that associates IMSIs with group IDs, and a setting information table T1 stored in the UDM / UDR 41.

[0052] In response to receiving a location registration request signal (second location registration request signal) transmitted by the second mobile terminal 2b upon reaching the communication area 30A, the notification unit 17 notifies the second mobile terminal 2b of the trained supervised learning model as route management information of the communication area 30A. More specifically, when the group ID associated with the location registration request signal transmitted by the second mobile terminal 2b upon reaching the communication area 30A is "1", the notification unit 17 notifies the second mobile terminal 2b of the trained supervised learning model as route management information. The notification unit 17 can transmit the route management information to the second mobile terminal 2b via the network NW.

[0053] [Mobile terminal functional block] Next, the functional blocks of the mobile terminal 2 will be described. Fig. 3 is a block diagram showing the configuration of the mobile terminal 2. The first mobile terminal 2a and the second mobile terminal 2b can have the same configuration. In this case, the mobile terminal 2 functions as the first mobile terminal 2a when it is preceding in the communication area 30, and functions as the second mobile terminal 2b when it is following.

[0054] The mobile terminal 2 includes a first learning unit 20, a movement route storage unit 21, a sending unit 22, a second acquisition unit 23, a third acquisition unit 24, a calculation unit 25, a decision unit 26, a movement control unit 27, and a fifth storage unit 28. When the mobile terminal 2 functions as the preceding first mobile terminal 2a, it learns an optimal movement route in the communication areas 30A and 30B and transmits it to the route management device 1. When the mobile terminal 2 functions as the following second mobile terminal 2b, it determines the next route that the mobile terminal should take based on the route management information notified from the route management device 1, and controls the movement of the mobile terminal until it reaches the final point position within the communication area 30.

[0055] The first learning unit 20 uses a first machine learning model to learn an optimal movement route within the communication area 30A, which indicates a course that the terminal should take from its position at each time, based on the position information of the terminal at each time in the communication area 30A. As an example, the first learning unit 20 learns, by reinforcement learning, an optimal movement route within the communication area 30A, which indicates a course that the terminal should take from its position at each time, from a position where the terminal arrives in the communication area 30A to a position where the terminal leaves the communication area 30A. The position of the terminal at each time is acquired by a third acquisition unit 24, which will be described later.

[0056] The first learning unit 20 applies a reward function to the estimated results of calculating the sequential routes that the terminal should take from the initial position in the communication area 30 to the final position, and updates the results so as to maximize the reward for the terminal to reach the final position, and learns the route that the terminal should take from its current position using a reinforcement learning model.

[0057] In this embodiment, as a course of action for the mobile terminal 2 to move sequentially from the position of each unit space, the mobile terminal 2 selects actions a1 related to movement in a predetermined n (n is an integer of 2 or more) directions with respect to the moving direction. n The travel direction is a direction based on the position of the unit space where the mobile terminal 2 was located immediately before.

[0058] The first learning unit 20 uses a neural network model including an input layer s, a hidden layer h, and an output layer q as shown in Fig. 6 as a reinforcement learning model. In addition, as the neural network model, a state s t , and all the action value functions Q(s t ,a 1 ), Q(s t ,a 2 ), Q(s t ,a 3 ), , Q(s t ,a n-1 ), Q(s t ,a n We adopt Deep Q-Network (DQN), a neural network that outputs

[0059] More specifically, the first learning unit 20 provides the current position of the unit space, which indicates the current position of the terminal itself, as an input to the neural network model, performs calculations on the neural network model, and learns actions a related to each of the movements in n directions as a course to be taken by the mobile terminal 2 from the current position in the unit space. n The first estimate Q1 of the action value function that represents the expected cumulative value of future rewards obtained if

[0060] The reward is a state s indicating the current location of the mobile terminal 2, and an action a of the mobile terminal 2 moving in a given direction. n , and the reward function r=r(s, a, s') for the next position of the mobile terminal 2, i.e., the next state s'. In this embodiment, the reward function includes, as a variable, the degree of arrival at a position in a unit space related to the final point of the mobile terminal 2. In addition, it may include, as a variable, the degree of arrival at a position in a unit space corresponding to a space containing obstacles such as high-rise buildings and steel towers. For example, when the action of moving the mobile terminal 2 in a specified direction brings it closer to the final point or when it reaches the final point in the shortest distance, the reward, which is a scalar quantity, is set to a larger value.

[0061] On the other hand, when the mobile terminal 2 moves away from the final point or reaches a unit space where an obstacle exists, a negative reward value (e.g., r=-1) can be given. In this way, by setting the reward for a unit space where an obstacle exists as a negative value, the mobile terminal 2 can reach the destination point while avoiding these points.

[0062] Furthermore, the first learning unit 20 provides the next position in the unit space that the terminal reaches as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function. The first learning unit 20 learns weight parameters of the neural network model so that the first estimated value Q1 becomes a target value calculated from the second estimated value Q2.

[0063] If the weight parameter of the neural network model is θ and the action value function is Q(s, a; θ), the minimization loss function for learning is given by the following equation (1). L(θ)=1 / 2{r+γmax a’ Q(s',a';θ)-Q(s,a;θ)} 2 (1)

[0064] In the above formula (1), r is the reward (immediate reward) and γ is the discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value at the state s' one step forward, that is, the second estimated value Q2. The target value is r+γmax a’ It is expressed as Q(s',a';θ).

[0065] The first learning unit 20 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above formula (1).

[0066] More specifically, the first learning unit 20 may employ a Fixed Target Q-Network that uses two neural networks, a main QN221 and a target QN223, as shown in FIG. 7. The main QN221 selects an optimal action and updates an action-value function Q. Meanwhile, the target QN223 estimates and evaluates the value of an action a' to be taken in the next state s' of the result of the action. The main QN221 and the target QN223 have neural networks with the same layer structure, but the parameter of the main QN221 is "θ" and the parameter of the target QN223 is "θ". - " is given by

[0067] The main QN 221 receives the current position of the mobile terminal 2 as state s from the environment 220. The environment 220 is a system of moving spaces in which the mobile terminal 2 is placed, that is, the communication area 30. In this environment 220, the mobile terminal 2 moves to another unit space by taking an action a related to movement in a predetermined direction, and transitions to the next state s', while simultaneously acquiring a reward r from the environment 220.

[0068] The first learning unit 20 inputs the state s related to the current position of the mobile terminal 2 to the main QN 221 and obtains the action value function Q(s, a; θ). The first learning unit 20 calculates the action a using, for example, the ε-greedy method, or calculates the optimal action argmax a Q(s, a; θ) is calculated. In the environment 220, the mobile terminal 2 performs the action argmax a Q(s, a; θ). The environment 220 is a place where the mobile terminal 2 takes action argmax a As a result of performing Q(s, a; θ), the position in the unit space to which the robot has moved is observed as the next state s', and a reward r is output. The experience data 224 stores the experience (s, a, r, s') output from the environment 220.

[0069] The first learning unit 20 calculates a loss function L in a DQN loss calculation 222, and updates the weight of the main QN 221 with the gradient of the loss function L.

[0070] The first learning unit 20 periodically copies the weights of the main QN 221 to the target QN 223 to perform synchronization. The synchronization of the target QN 223 is performed less frequently than the update frequency of the weights of the main QN 221. The first learning unit 20 extracts experience from the experience data 224, inputs past states to the target QN 223, and calculates the estimated value max a’ Q(s', a';θ - The first learning unit 20 outputs the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ - ) is used to learn the weights of the main QN 221 in the DQN loss calculation 222.

[0071] The movement route storage unit 21 stores an optimal movement route that indicates a course to be taken sequentially from the position of the initial point in the communication area 30A learned by the first learning unit 20 to the position of the final point, that is, a learned reinforcement learning model. FIG. 5 is a diagram showing an optimal movement route stored in the movement route storage unit 21. The time stamp [t] in FIG. 5 indicates the timing when the third acquisition unit 24 acquires the GPS position of the terminal. In this way, the optimal movement route is information in which the GPS position of the terminal at each time is associated with the action value function Q at that position.

[0072] The sending unit 22 transmits the optimal movement route in the communication area 30A obtained by the first learning unit 20 to the route management device 1 at the timing when the terminal itself transmits a location registration request signal upon reaching the communication area 30B adjacent to the communication area 30. Specifically, the sending unit 22 transmits the optimal movement route after the data communication path of the terminal itself is set in response to the location registration request signal.

[0073] The above-described first learning unit 20 and movement route storage unit 21 are components necessary when the mobile terminal 2 functions as the preceding first mobile terminal 2a. On the other hand, the following second acquisition unit 23, calculation unit 25, decision unit 26, movement control unit 27, and fifth storage unit 28 are components necessary when the mobile terminal 2 functions as the following second mobile terminal 2b.

[0074] The second acquisition unit 23 acquires the route management information notified by the notification unit 17 of the route management device 1.

[0075] The third acquisition unit 24 acquires the current position of the terminal. More specifically, the third acquisition unit 24 acquires the position of the unit space in which the terminal is located for each time t based on the GPS position of the terminal. The third acquisition unit 24 refers to the fifth storage unit 28 and acquires the position of the unit space corresponding to the current GPS position received by the GPS receiver 207 as the current position of the terminal.

[0076] The calculation unit 25 provides the current position of the terminal acquired by the third acquisition unit 24 as an unknown input to the trained supervised learning model, performs calculations on the trained supervised learning model, and outputs an optimal movement route within the communication area 30A indicating the course that the terminal should take sequentially from its current position.

[0077] The determination unit 26 determines the next route to be taken from the current position of the terminal based on the optimal moving route within the communication area 30A output by the calculation unit 25. More specifically, based on the course plan output by the calculation unit 25, the determination unit 26 determines the current position in the unit space as the state s t For each state s t The path to be taken is determined by selecting the path that is action a with the maximum value of the action value function Q.

[0078] The movement control unit 27 controls the movement of the terminal itself based on the optimal movement route within the communication area 30A output by the calculation unit 25. Specifically, the movement control unit 27 controls the movement of the terminal itself based on the next route to be taken determined by the determination unit 26. The movement control unit 27 can calculate a control command for the next route to be taken from the current position, and transmit the control command value to the motor 209.

[0079] The fifth storage unit 28 stores map data including the position coordinates of each communication area 30 which is the moving space, and information associating the position coordinates of unit spaces constituting the moving space with the node IDs of the unit spaces.

[0080] Here, the second learning unit 12 included in the above-mentioned route management device 1 will be described in more detail with reference to FIG.

[0081] FIG. 8 shows the structure of a neural network model employed as an example of a supervised learning model used by the second learning unit 12. The neural network model includes an input layer x, a hidden layer h, and an output layer y. The second learning unit 12 provides the current unit space position of the mobile terminal 2, i.e., the unit space position corresponding to the GPS position of the mobile terminal 2 at each time t, to the input layer of the neural network model, applies an activation function to the weighted sum of the input, and passes the output determined by threshold processing to the output layer. Each output node of the output layer outputs a predicted output of the model corresponding to n action value functions Q for each time t in one communication area 30.

[0082] The second learning unit 12 introduces an objective function E shown in the following equation (2) to learn parameters of the neural network model so that the route to be taken sequentially from the current position within the communication area 30, which is a predicted value from the neural network model for the current position of the mobile terminal 2, becomes an optimized value of the route to be taken sequentially from the current position of the mobile terminal 2, which is reinforcement learned by the first learning unit 20 of the mobile terminal 2.

[0083]

number

[0084] In the above equation (2), y 1 ,y 2 ,···,y n indicates the predicted output value of each output node. Also, Y 1 ,Y 2 ,···,Y n is a label of the teacher data, and in this case, is n optimized action value functions Q for the current position within the communication area 30 obtained by reinforcement learning by the first learning unit 20 of the mobile terminal 2. Furthermore, as described above, the first learning unit 20 of the mobile terminal 2 calculates the time t 1 Position (X 1 ,Y 1 ,Z 1 ) Action value function Q 1 (s 1 ,a 1)~Q 1 (s 1 ,a n ), from time t m Position (X m ,Y m ,Z m ) Action value function Q m (s m ,a 1 )~Q m (s m ,a n ) are used as training data.

[0085] The objective function E in the above formula (2) is further expressed as an action value function Q 1 ~Q m In this case, the first time t 1 Position (X 1 ,Y 1 ,Z 1 ) label Y 1 Y in this case 1 The optimized n action value functions Q 1 (s 1 ,a 1 ), Q 1 (s 1 ,a 2 ), Q 1 (s 1 ,a 3 ), , Q 1 (s 1 ,a n-1 ), Q 1 (s 1 ,a n ), the second time t 2 Position (X 2 ,Y 2 ,Z 2 ) label Y 2 Y in this case 2 The optimized n action value functions Q 2 (s 2 ,a 1 ), Q 2 (s 2 ,a 2 ), Q 2 (s 2 ,a 3), , Q 2 (s 2 ,a n-1 ), Q 2 (s 2 ,a n ), , , the mth time t m Position (X m ,Y m ,Z m ) label Y m The optimized n action value functions Q m (s m ,a 1 ), Q m (s m ,a 2 ), Q m (s m ,a 3 ), , Q m (s m ,a n-1 ), Q m (s m ,a n ) is used as training data.

[0086] In this case, the above formula (2) is further expressed as the following formula (3).

number

[0087] In the objective function E in the above equation (3), y 1_1 ,y 1_2 ,···,y 1-n ,···,y 2_1 ,y 2_2 ,···,y 2-n ,···,y m_1 ,y m_2 ,···,y m-n is n(n action value functions Q) × m(time t 1 ~t m The predicted output values ​​of the m output nodes are 1_1 ~Y 1_n is the time t 1 The optimized action value function Q 1 Corresponding to Y 2_1 ~Y 2_n is the time t2 The optimized action value function Q 2 Corresponding to Y m_1 ~Y m_n is the time t m The optimized action value function Q m This is the training data corresponding to

[0088] In this way, all of the optimized action value functions Q at each time t obtained by the trained reinforcement learning model are used as training data. In the example of the neural network model configuration in Figure 8, training data is shown to the right of each output node that outputs the predicted output value of the model. Starting from the top output node, the training data Y 1_1 ~Y 1_n is time t 1 n action value functions Q 1 is shown, but the training data Y 2 Y m For each, the action value function Q 2 ,···,Q m In the figure, the n values ​​corresponding to the above are omitted.

[0089] The value of the objective function E in the above formula (3) is the above input value x of the supervised learning model, and the output value y for the position in the unit space corresponding to the GPS position of the mobile terminal 2 at time t. 1_1 ,y 1_2 ,···,y 1-n ,···,y 2_1 ,y 2_2 ,···,y 2-n ,···,y m_1 ,y m_2 ,···,y m-n is the target output of the training data Y 1_1 ~Y 1_n ,Y 2_1 ~Y 2_n ,···,Y m_1 ~Y m_nThe second learning unit 12 adjusts weight parameters of the neural network related to the supervised learning model so that the objective function E is minimized, that is, becomes 0. The second learning unit 12 can optimize the objective function E by using a backpropagation method or the like.

[0090] [Hardware configuration of route management device] Next, an example of a hardware configuration for realizing the route management device 1 having the above-mentioned functions will be described with reference to FIG.

[0091] 9, the path management device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output I / O 106, which are connected via a bus 101, and a program that controls these hardware resources. Furthermore, the path management device 1 can include a display device 107 connected via the bus 101.

[0092] The processor 102 is realized by a CPU, a GPU, an FPGA, an ASIC, or the like.

[0093] The main memory device 103 prestores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 realize the functions of the route management device 1, such as the setting unit 10, the route acquisition unit 11, the second learning unit 12, and the notification unit 17 shown in FIG.

[0094] The communication interface 104 is an interface circuit for connecting the route management device 1 to various external electronic devices via a network.

[0095] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or a flash memory as the storage medium.

[0096] The auxiliary storage device 105 has a program storage area for storing the route management program executed by the route management device 1. Furthermore, the auxiliary storage device 105 has an area for storing the supervised learning program. The first storage unit 13, the second storage unit 14, the third storage unit 15, and the fourth storage unit 16 described in FIG. 1 are realized by the auxiliary storage device 105. Specifically, the auxiliary storage device 105 has an area for storing the final point for each communication area 30 of the mobile terminal 2. Furthermore, the auxiliary storage device 105 has an area for storing the position coordinates of the moving space and the position coordinates of the unit space. Furthermore, the auxiliary storage device 105 has an area for storing identification information such as the IP address of the mobile terminal 2. Furthermore, for example, it may have a backup area for backing up the above-mentioned data, programs, etc.

[0097] The input / output I / O 106 is an input / output device that inputs signals from an external device and outputs signals to an external device.

[0098] The display device 107 is configured with an organic EL display, a liquid crystal display, etc. The display device 107 can display a map display of the moving space, the current positions and progress of the multiple mobile terminals 2 for each group, and the position information of the final point for each communication area 30.

[0099] [Mobile terminal hardware configuration] Next, an example of a hardware configuration for realizing the mobile terminal 2 having the above-mentioned functions will be described with reference to FIG.

[0100] 10, the mobile terminal 2 can be realized by, for example, a microcomputer equipped with a processor 202, a main memory device 203, a communication interface 204, an auxiliary memory device 205, and an input / output I / O 206, which are connected via a bus 201, a program for controlling these hardware resources, a GPS receiver 207, a sensor 208, a motor 209, a drive mechanism 210, and a battery 211. A controller for controlling the autonomous movement of the mobile terminal 2 is realized by a computer such as a microcomputer and a program.

[0101] Programs for the processor 202 to perform movement control and calculations are stored in advance in the main memory device 203. The processor 202 and the main memory device 203 realize the functions of the mobile terminal 2, such as the first learning unit 20, the second acquisition unit 23, the calculation unit 25, the decision unit 26, and the movement control unit 27, shown in FIG.

[0102] The communication interface 204 is an interface circuit for network connection between the mobile terminal 2 and the route management device 1. The communication interface 204 realizes the transmission unit 22 described in FIG.

[0103] The auxiliary storage device 205 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from the storage medium. The auxiliary storage device 205 can use a semiconductor memory such as a hard disk or a flash memory as the storage medium.

[0104] The auxiliary storage device 205 has a program storage area for storing a movement control program executed by the mobile terminal 2. It also has a program storage area for storing a reinforcement learning program executed by the mobile terminal 2. The auxiliary storage device 205 also has an area for storing a calculation program for performing calculations of a learned supervised learning model. The auxiliary storage device 205 realizes the movement route storage unit 21 and the fifth storage unit 28 described in FIG. 4. The auxiliary storage device 205 also has an area for storing identification information such as an IP address of the mobile terminal 2. The auxiliary storage device 205 can also have an area for storing an application that notifies the route management device 1 of a GPS position. Furthermore, for example, it may have a backup area for backing up the above-mentioned data, programs, and the like.

[0105] The input / output I / O 206 is an input / output device that inputs signals from an external device and outputs signals to an external device.

[0106] The GPS receiver 207 has an antenna for receiving GPS signals built in. The GPS receiver 207 realizes the third acquisition unit 24 in FIG.

[0107] The sensor 208 is composed of various sensors such as an altitude sensor, an attitude sensor, a camera, LiDAR, RADAR, etc. In addition to the GPS receiver 207, the altitude sensor realizes the third acquisition unit 24 in Fig. 4. Furthermore, the controller controls the movement of the mobile terminal 2 based on the various sensor data measured by the sensor 208.

[0108] The motor 209 rotates by the rotary drive, and drives a drive mechanism 210 attached to the rotary shaft of the motor 209 .

[0109] The battery 211 is a lithium ion battery or the like, and supplies power to the components of the mobile terminal 2 .

[0110] [Operation of the route management system] Next, the operation of the route management system including the route management device 1 and the mobile terminal 2 having the above-described configuration will be described with reference to the sequences of FIG. 11 and FIG.

[0111] 11 is a sequence diagram showing grouping of mobile terminals 2 by the route management system and setting of communication paths for data communication. In the following, a grouping and communication path setting process is described which is performed when a first mobile terminal 2a moving in advance in a communication area 30A reaches a communication area 30B of a base station 3B as a next destination after performing reinforcement learning of an optimal moving route in the communication area 30A.

[0112] First, the route management device 1 transmits grouping information associated with a group ID and an IMSI to the UDM / UDR 41 (step S100). The grouping information is stored in advance in the route management device 1. Next, the UDM / UDR 41 creates a setting information table T1 (FIG. 3) based on the received grouping information (step S101). In step S101, the values ​​of "group ID" and "IMSI" in the setting information table T1 are registered, and there are no entries for the values ​​of the other items.

[0113] After that, the first mobile terminal 2a, which has moved from the communication area 30A to the communication area 30B, transmits a location registration request signal from the base station 3B to the core network 4 (step S102). The location registration request signal includes the IMSI of the mobile terminal 2 and the base station ID of the base station 3B of the present communication area 30B. When the location registration request signal is received by the UDM / UDR 41 via the AMF 40, the UDM / UDR 41 associates a group ID with the base station ID and IMSI included in the received location registration request signal based on the information in the setting information table T1 (FIG. 3) created in step S101, and further transfers the location registration request signal to the route management device 1 (step S103). Note that in step S103, the value of the base station ID in the setting information table T1 is set.

[0114] Next, the setting unit 10 of the route management device 1 requests the PCF 43 to create a PCC policy by specifying the IMSI, base station ID, group ID, and further the IP address of the route management device 1, which are included in the location registration request signal received in step S103 (step S104). Then, the PCF 43 creates a PCC policy based on the specified requirements (step S105). The PCF 43 creates a PCC policy including information on which UPF 44 the data communication of the mobile terminal 2 will pass through. The PCC policy specifies the optimal UPF 44 as a communication path for data communication from the mobile terminal 2 to the data network 5.

[0115] Next, the PCF 43 transmits the created PCC policy to the SMF 42 (step S106). After that, the SMF 42 transmits data communication path setting information related to the user plane function, such as setting of a communication path specified by the PCC policy, to the UPF 44 specified by the PCC policy (step S107). Next, the UPF 44 registers the received data communication path setting information in a memory (step S108).

[0116] Next, the UPF 44 transmits an ACK to the SMF 42 to notify it of the IP address of the UPF 44 (step S109). Furthermore, the SMF 42 transmits an ACK to the route management device 1 to notify it of the IP address of the UPF 44 (step S110). After that, the route management device 1 notifies the UDM / UDR 41 of the IP address of the UPF 44, and transmits an ACK to the first mobile terminal 2a (step S111). At that time, the IP address of the UPF 44 that communicates with the first mobile terminal 2a is set in the setting information table T1. Moreover, the setting information table T1 updated by the UDM / UDR 41 is also stored in the fourth storage unit 16 of the route management device 1. Further, after that, a communication path for data communication between the first mobile terminal 2a and the UPF 44 is established (step S112).

[0117] Thereafter, the sending unit 22 of the first mobile terminal 2a sends the optimal movement route in the communication area 30A of the movement source, which has been reinforced learned by the first learning unit 20, via the UPF 44 (step S113) to the route management device 1 (step S114). The sending unit 22 sends the optimal movement route, together with the base station ID of the communication area 30A, to the route management device 1. Thereafter, the route management device 1 executes the second learning process and an accounting management process including a notification process (step S115).

[0118] FIG. 12 is a sequence diagram showing grouping of mobile terminals 2 by the route management system and setting of a communication path for data communication when the subsequent second mobile terminal 2b reaches the communication area 30A after the process of FIG. 11. The process from step S100 to step S111 of FIG. 12 is the same as that described in FIG. 11. The difference from FIG. 11 is that in step S112′, when the route management device 1 receives a signal indicating that a data communication path is established, the route management device 1 notifies the second mobile terminal 2b of the route management information of the communication area 30A (step S213). The route management device 1 designates the group ID and IMSI and notifies the second mobile terminal 2b of the route management information for the communication area 30A via the UPF 44 (step S214).

[0119] Next, the operations of the first mobile terminal 2a, the second mobile terminal 2b, and the route management device 1 will be described with reference to the sequence of FIG. 13. First, the first mobile terminal 2a with group ID "1" moves first in the communication area 30A. The third acquisition unit 24 of the first mobile terminal 2a acquires the current position of the unit space from the GPS position of the first mobile terminal 2a (step S1). Next, the first learning unit 20 of the first mobile terminal 2a performs a first learning process (step S2). In the first learning process, the first learning unit 20 applies a reward function to an estimated result of calculating a course that the first mobile terminal 2a should take from the initial point position of the communication area 30A to the final point position, updates the course so as to maximize the reward for reaching the final point position, and learns a course policy that the first mobile terminal should take from the current position using a reinforcement learning model. The details of the first learning process will be described later.

[0120] Thereafter, the first mobile terminal 2a reaches the next destination communication area 30B from the source communication area 30A (step S3). The sending unit 22 of the first mobile terminal 2a transmits the optimal movement route in the communication area 30A learned in step S2 to the route management device 1 at the timing of transmitting a location registration request signal to the route management device 1 via the core network 4 (step S4). Specifically, step S4 is executed after the grouping and establishment of the data communication path described in FIG. 11 (steps S100 to S112 in FIG. 11). The route acquisition unit 11 of the route management device 1 acquires the optimal movement route in the communication area 30A from the first mobile terminal 2a.

[0121] Next, the second learning unit 12 of the route management device 1 uses the optimal travel route in the communication area 30A (Figure 3) acquired from the first mobile terminal 2a in step S4 as teacher data, and learns the relationship between the current position of the first mobile terminal 2a and the optimal travel route within the communication area 30A using a supervised learning model (step S5).

[0122] Specifically, the second learning unit 12 repeatedly adjusts and updates parameters such as weights and thresholds, and determines the values ​​of these parameters so that the error between the predicted output value of the policy for the sequential course to be taken when the position in the unit space corresponding to the current GPS position of the first mobile terminal 2a, i.e., the current state, is given as an input value to the supervised learning model and the teacher data minimizes the objective function E in the above formula (3). In step S5, the second learning unit 12 can determine the parameters that minimize the objective function E by using an error backpropagation method or the like.

[0123] Next, the second storage unit 14 stores the trained supervised learning model constructed in step S5 (step S6). In step S6, the trained supervised learning model and the group ID are stored in association with each other. Furthermore, the second storage unit 14 can store the trained supervised learning model in association with the base station ID.

[0124] Thereafter, when the second mobile terminal 2b arrives in the communication area 30A, it transmits a location registration request signal to the route management device 1 via the core network 4 (step S7). The location registration request signal transmitted by the second mobile terminal 2b includes an IMSI and is assigned a group ID. Next, the notification unit 17 of the route management device 1 determines whether the second mobile terminal 2b is a subsequent mobile terminal 2 belonging to group ID "1" based on the group ID assigned to the location registration request signal received in step S7 (step S8). Furthermore, it is possible to determine which communication area 30 the route management information of which is to be notified is based on the base station ID assigned to the location registration request signal. Thereafter, when it is determined that the second mobile terminal 2b is a subsequent second mobile terminal 2b of group ID "1", the notification unit 17 notifies the second mobile terminal 2b of the route management information of the communication area 30A (step S9). The second acquisition unit 23 of the second mobile terminal 2b acquires the route management information.

[0125] Next, the third acquisition unit 24 of the second mobile terminal 2b acquires the current unit space position based on the GPS position of the terminal (step S11). Specifically, the unit space position corresponding to the GPS position received by the GPS receiver 207 can be acquired as the current position of the terminal. Next, the calculation unit 25 uses the route management information acquired in step S9 to provide the current unit space position of the terminal acquired in step S11 as an unknown input, performs calculations of the learned supervised learning model, and outputs a course plan to be taken from the current unit space position (step S12).

[0126] For example, in the moving space corresponding to the communication area 30A shown in FIG. 2, when the position of the initial point in the communication area 30A of the second mobile terminal 2b is input as the current position at time t=1 to the trained supervised learning model, n action value functions Q to be followed from the initial point at time t=1 are output. More specifically, when the current position of the second mobile terminal 2b is given as an unknown input, the values ​​of the specific n course measures corresponding to the current position of the mobile terminal 2 from among the optimized course measures for positions corresponding to m times are output as the calculation results of the trained supervised learning model. The values ​​of the other n course measures not corresponding to the current position of the mobile terminal 2 are 0.

[0127] Next, the decision unit 26 of the second mobile terminal 2b selects a route along which the action a having the maximum value of the n action-value functions Q output in step S12 is taken, thereby deciding a route to be taken in sequence (step S13). Next, the movement control unit 27 controls the movement of the terminal itself based on the route to be taken next decided in step S13 (step S14). More specifically, the movement control unit 27 can calculate a control command for the route to be taken next from the current position, and transmit the control command value to the motor 209.

[0128] The mobile terminal 2 repeats the process from step S10 to step S14 from the initial point in the communication area 30A until it reaches the unit space of the final point (step S15: NO). If it reaches the unit space of the final point (step S15: YES), the process ends. In this way, the route management system executes the process from step S1 to step S15 for each communication area 30, thereby allowing multiple mobile terminals 2 with a common group ID to move through the L communication areas 30 via an optimal route.

[0129] Next, the first learning process of the first mobile terminal 2a will be described in detail with reference to the flowcharts of Fig. 14 and Fig. 15. First, step S1 described in Fig. 13 is executed. First, the first learning unit 20 inputs the position of the unit space where the terminal is currently located, which is the current state of the terminal acquired in step S1, to the neural network model, performs calculations on the neural network model, and outputs a first estimated value Q1 of an action value function that represents an expected value of a cumulative value of future rewards obtained when the terminal takes each action related to movement in a predetermined direction relative to the traveling direction as a course to be followed next from the current position in the unit space (step S20).

[0130] Next, the third acquisition unit 24 acquires the position of the terminal in the unit space at the next time t as the next state s' (step S21). The position of the unit space where the terminal reaches next is determined based on the GPS position of the terminal acquired by the third acquisition unit 24 for each time step. Furthermore, the first learning unit 20 provides the position of the unit space where the terminal reaches next, acquired in step S21, as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function (step S22).

[0131] Next, the first learning unit 20 calculates a target value from the second estimated value Q2 (step S23). Then, the first learning unit 20 learns weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S24). Specifically, the first learning unit 20 updates the weight parameters of the neural network model so as to minimize the loss function of the above formula (1). The first learning unit 20 repeats the processes from step S1 to step S24 a set number of times.

[0132] Thereafter, the movement route storage unit 21 stores the trained reinforcement learning model obtained in step S24 (step S25).

[0133] Next, the first learning process by the first learning unit 20 in the case where a Fixed Target Q-Network using two neural networks, a main QN 221 and a target QN 223, is adopted will be described with reference to FIG.

[0134] The process of step S1 is similar to the step of the first learning process described in Fig. 13. After that, the first learning unit 20 provides the position of the unit space where the mobile terminal 2 is currently located, acquired in step S1, as an input to the main QN 221, performs calculations of the neural network model, outputs an action value function Q, and calculates the next course a to be taken (step S120).

[0135] Next, the first learning unit 20 returns the behavior of the mobile terminal 2 on the path a obtained in step S120 to the environment 220, and obtains the position of the unit space to which the mobile terminal 2 has advanced, which is the next state s' of the mobile terminal 2, and the reward r (step S121).

[0136] The first learning unit 20 stores the experience (s, a, r, a') obtained in step S121 in the experience data 224 (step S122). Next, the first learning unit 20 calculates a loss function L in the DQN loss calculation 222, and updates the weight of the main QN 221 with the gradient of the loss function L (step S123). The first learning unit 20 repeats the processes from step S120 to step S123 a set number of times.

[0137] After that, the first learning unit 20 periodically copies the weights of the main QN 221 to the target QN 223 to synchronize them (step S124). The synchronization of the target QN 223 is performed less frequently than the update frequency of the weights of the main QN 221. Next, the first learning unit 20 extracts experience from the experience data 224, inputs past states to the target QN 223, and calculates the estimated value max a’ Q(s', a';θ - ) is output (step S126).

[0138] Next, the first learning unit 20 calculates the estimated value max a’Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ - ) (step S127). Next, the first learning unit 20 calculates a loss function L in the DQN loss calculation 222 using the target value calculated in step S127 (step S128). Next, the first learning unit 20 learns the weight of the main QN 221 so as to minimize the loss given by the loss function L (step S129). After that, the learned reinforcement learning model, i.e., the optimal movement route in the communication area 30A, is stored in the movement route storage unit 21 (step S25).

[0139] As described above, according to the route management system of the present embodiment, the optimal movement route in the communication area 30A learned by the preceding first mobile terminal 2a is transmitted to the route management device 1 at the timing when the location registration request signal is transmitted when the first mobile terminal 2a reaches the next communication area 30B. Furthermore, a group determination is performed in response to a location registration request signal transmitted when the following second mobile terminal 2b reaches the communication area 30A, and route management information of the communication area 30A obtained by learning based on the optimal movement route in the communication area 30A learned by the preceding first mobile terminal 2a is notified to the following second mobile terminal 2b. Therefore, the optimal action can be shared by multiple devices with a simpler configuration.

[0140] [Variations] Next, a modified example of the embodiment of the present invention will be described. In the above-mentioned embodiment, the case where the mobile terminal 2 is equipped with the first learning unit 20 has been described. In contrast, in the route management system according to this modified example, the route management device 1A is equipped with the first learning unit 18 and performs all learning processes, including reinforcement learning and supervised learning. Meanwhile, the first mobile terminal 2a, which moves ahead in the communication area 30A, transmits its own terminal's location information to the route management device 1A.

[0141] [Route management device functional block] Fig. 16 is a block diagram showing a configuration of a route management device 1A according to a modified example. As shown in Fig. 16, the route management device 1A includes a setting unit 10A, a first acquisition unit 11A, a second learning unit 12, a first storage unit 13, a second storage unit 14, a third storage unit 15, a fourth storage unit 16, a notification unit 17, and a first learning unit 18. The route management device 1A according to the modified example differs from the route management device 1 according to the embodiment in that the route management device 1A includes the first acquisition unit 11A and the first learning unit 18. The following mainly describes the configuration different from that of the route management device 1 according to the present embodiment.

[0142] The first acquisition unit 11A acquires the location information of the first mobile terminal 2a' that is added to a location registration request signal that is transmitted every hour in the communication area 30A by the first mobile terminal 2a' that moves in advance in the communication area 30A.

[0143] The first learning unit 18 performs a first learning process based on the location information of the first mobile terminal 2a' for each time acquired by the first acquisition unit 11A.

[0144] The setting unit 10A does not perform the grouping process performed by the setting unit 10 of the present embodiment, but instead sets a communication path for data communication.

[0145] [Mobile terminal functional block] 17 is a block diagram showing the configuration of a mobile terminal 2' according to a modified example. The mobile terminal 2' includes a sending unit 22, a second acquiring unit 23, a third acquiring unit 24, a calculation unit 25, a decision unit 26, a movement control unit 27, and a fifth storage unit 28. The mobile terminal 2' according to the modified example differs in configuration in that it does not include a first learning unit 20 and a movement route storage unit 21 that are included in the mobile terminal 2 according to the present embodiment.

[0146] The sending unit 22 adds the GPS position of the terminal acquired by the third acquiring unit 24 to the position registration request signal and transmits the signal to the route management device 1A at regular intervals. More specifically, when the sending unit 22 arrives at the destination communication area 30 and at regular intervals, the sending unit 22 transmits the position registration request signal to which the position information of the terminal has been added to the route management device 1A via the base station 3 and the core network 4.

[0147] [Operation of the route management system] Next, the operation of the route management system having the above-mentioned configuration will be described with reference to the sequence diagram of Fig. 18. The contents of the processes from step S1 to step S15 below are the same as the processes in the operation of the route management system according to the present embodiment described with reference to Fig. 13.

[0148] First, when the transmitter 22 of the first mobile terminal 2a', which moves ahead, arrives at the communication area 30A, it adds its own terminal's location information to a location registration request signal and transmits it to the route management device 1A (step S1A). In step S1A, while the first mobile terminal 2a' is present in the communication area 30A, it transmits a location registration request signal to which its own terminal's location information has been added at regular intervals. The base station ID of the present communication area 30A is added to the location registration request signal.

[0149] Next, the first acquisition unit 11A of the route management device 1A acquires the location information of the first mobile terminal 2a' added to the location registration request signal transmitted by the first mobile terminal 2a' in the communication area 30A at each time (step S1). Next, the first learning unit 18 learns, by reinforcement learning, the optimal movement route in the communication area 30A indicating the course that the first mobile terminal 2a' should take sequentially from the position at each time based on the location information at each time acquired in step S1 (step S2). Next, the second learning unit 12 learns the relationship between the current position of the first mobile terminal 2a' and the optimal movement route in the communication area 30A obtained in step S2 by supervised learning (step S5). The second storage unit 14 stores the learned supervised learning model (step S6).

[0150] Thereafter, when the subsequent second mobile terminal 2b' reaches the communication area 30A, it transmits a location registration request signal to the route management device 1A via the base station 3B and the core network 4 (step S7). In response to the location registration request signal, the notification unit 17 of the route management device 1A notifies the second mobile terminal 2b' of the learned supervised learning model, which is the route management information of the communication area 30A (step S9). More specifically, the location registration request signal is transmitted from the IMSI of the management target, and the base station ID is added to the location registration request signal in the core network 4 (FIG. 3). Then, a data communication path for data communication of the second mobile terminal 2b' is set (FIG. 3). In response to the establishment of the data communication path, the notification unit 17 notifies the second mobile terminal 2b' of the route management information of the communication area 30A via the UPF 44.

[0151] The second mobile terminal 2b' acquires route management information for the communication area 30A (step S10), and based on the current location information of the terminal, calculates the learned supervised learning model, determines the next route to take, and instructs the route, repeating the process until it reaches the final point of the communication area 30A (steps S11 to S15).

[0152] As described above, according to the route management system of the modified example, the route management device 1A performs reinforcement learning and supervised learning to generate route management information for each communication area 30, and notifies each mobile terminal 2 of the route management information in response to a location registration request signal transmitted when each mobile terminal 2 reaches the communication area 30. Therefore, even if the mobile terminal 2 does not have route information, it is possible to manage the route of the mobile terminal 2 from the network side with a simpler configuration. For example, in an environment where the mobile terminal 2 cannot receive a GPS signal, it is possible to appropriately guide the moving route of the mobile terminal 2.

[0153] In the embodiment described above, the reinforcement learning model used by the first learning unit 20 is exemplified as DQN related to Fixed Target Q-Network composed of a multi-layer neural network, but the reinforcement learning model can be CNN, multi-layer perceptron, etc. In addition to DQN exemplified as a reinforcement learning model, Double DQN, Dueling DQN, Actor-Critic (AC) method, Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Q-learning, etc. can be used.

[0154] In the embodiment described above, a multilayer neural network is used as the supervised learning model used by the second learning unit 12. However, the supervised learning model may be a multilayer perceptron, a decision tree-based model such as a random forest, a support vector machine, or the like.

[0155] In the embodiment described above, the first mobile terminal 2a and the second mobile terminal 2b are described as having the same configuration. However, when the first mobile terminal 2a that moves ahead is determined in advance, the first mobile terminal 2a and the second mobile terminal 2b can have different configurations. In this case, the first mobile terminal 2a is configured to include a first learning unit 20, a moving route storage unit 21, a sending unit 22, and a third acquisition unit 24. On the other hand, the following second mobile terminal 2b can be configured to include a sending unit 22, a second acquisition unit 23, a third acquisition unit 24, a calculation unit 25, a decision unit 26, a moving control unit 27, and a fifth storage unit 28.

[0156] The above describes embodiments of the route management system and route management method of the present invention. However, the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can make are possible within the scope of the invention described in the claims. [Explanation of symbols]

[0157] 1...route management device, 10...setting unit, 11...route acquisition unit, 12...second learning unit, 13...first memory unit, 14...second memory unit, 15...third memory unit, 16...fourth memory unit, 17...notification unit, 2...mobile terminal, 2a...first mobile terminal, 2b...second mobile terminal, 20...first learning unit, 21...movement route memory unit, 22...sending unit, 23...second acquisition unit, 24...third acquisition unit, 25...calculation unit, 26...determination unit, 27...movement control unit, 28... Fifth memory unit, 101, 201...bus, 102, 202...processor, 103, 203...main memory device, 104, 204...communication interface, 105, 205...auxiliary memory device, 106, 206...input / output I / O, 107...display device, 207...GPS receiver, 220...environment, 221...main QN, 222...DQN loss calculation, 223...target QN, 224...empirical data, NW...network.

Claims

1. a first learning unit configured to learn, using a first machine learning model, an action value function indicating a value of an action of the first mobile terminal sequentially proceeding from a position at each time in a first communication area based on position information of the first mobile terminal in the first communication area at each time, in order to determine an optimal movement route from an initial point to a final point in the first communication area; a second learning unit configured to learn, using a second machine learning model, a relationship between the position of the first mobile terminal in the first communication area at each time and the action value function obtained by learning by the first learning unit; A memory unit configured to store a trained second machine learning model constructed by training by the second training unit; a notification unit configured to notify the second mobile terminal of the trained second machine learning model as route management information of the first communication area in response to receiving a second location registration request signal transmitted by the second mobile terminal when the second mobile terminal arrives at the first communication area; A route management system comprising:

2. The route management system according to claim 1, The method further includes a route management device including the second learning unit and the notification unit, The first mobile terminal includes the first learning unit, The first mobile terminal transmits the action value function obtained by the learning by the first learning unit to the route management device, in response to a first location registration request signal transmitted when the first mobile terminal arrives at a second communication area adjacent to the first communication area. A route management system comprising:

3. In the route management system according to claim 2, a first group identifier, which is an identifier of a group to which the first mobile terminal belongs, is associated with the first location registration request signal transmitted by the first mobile terminal in the first communication area; The storage unit stores the trained second machine learning model in association with the first group identifier; The notification unit notifies the second mobile terminal of the learned second machine learning model as the route management information when an identifier of the group associated with the second location registration request signal transmitted when the second mobile terminal arrives at the first communication area is the first group identifier. A route management system comprising:

4. The route management system according to claim 1, The method further includes a route management device including the first learning unit, the second learning unit, and the notification unit, The route management device includes: a first acquisition unit configured to acquire the location information of the first mobile terminal added to a first location registration request signal that the first mobile terminal transmits in the first communication area at each time, The first learning unit performs learning based on the location information of the first mobile terminal acquired by the first acquisition unit. A route management system comprising:

5. In the route management system according to claim 2 or 4, The second mobile terminal, a second acquisition unit configured to acquire the route management information notified by the notification unit; A third acquisition unit configured to acquire a time-based location of the terminal within the first communication area; a calculation unit configured to provide the position of the terminal at each time acquired by the third acquisition unit as an unknown input to the trained second machine learning model, perform a calculation of the trained second machine learning model, and output the action value function indicating a value of an action proceeding sequentially from the position of the terminal at each time; a movement control unit configured to control movement of the terminal based on the optimal movement route within the first communication area determined based on the action value function output by the calculation unit; A route management system comprising:

6. The route management system according to claim 1, The first learning unit learns, by reinforcement learning, the action value function indicating a value of an action proceeding sequentially from the position of the first mobile terminal at each time from the initial point, which is a position where the first mobile terminal arrives in the first communication area, to the final point, which is a position where the first mobile terminal leaves the first communication area. A route management system comprising:

7. a first learning step of learning, using a first machine learning model, an action value function indicating a value of an action of the first mobile terminal proceeding sequentially from a position at each time in a first communication area based on position information of the first mobile terminal in the first communication area at each time, in order to determine an optimal movement route from an initial point to a final point in the first communication area; a second learning step of learning, using a second machine learning model, a relationship between the position of the first mobile terminal in the first communication area at each time and the action value function in the first communication area obtained by learning in the first learning step; a storage step of storing in a storage unit a trained second machine learning model constructed by the training in the second training step; a notification step of notifying the second mobile terminal of the learned second machine learning model as route management information of the first communication area in response to receiving a second location registration request signal transmitted by the second mobile terminal upon reaching the first communication area; A route management method comprising:

8. The route management method according to claim 7, A route management device executes the second learning step and the notification step; The first mobile terminal performs the first learning step; The first mobile terminal transmits the action value function of the first communication area obtained by learning in the first learning step to the route management device, in response to a first location registration request signal transmitted when the first mobile terminal arrives at a second communication area adjacent to the first communication area. A route management method comprising:

9. The route management method according to claim 8, a first group identifier, which is an identifier of a group to which the first mobile terminal belongs, is associated with the first location registration request signal transmitted by the first mobile terminal in the first communication area; The storing step includes storing the first group identifier in the storage unit in association with the trained second machine learning model; The notification step notifies the second mobile terminal of the learned second machine learning model as the route management information when an identifier of the group associated with the second location registration request signal transmitted when the second mobile terminal arrives at the first communication area is the first group identifier. A route management method comprising:

10. The route management method according to claim 7, a route management device executes the first learning step, the second learning step, and the notification step; The method further comprises a first acquisition step, executed by the route management device, of acquiring the location information of the first mobile terminal added to a first location registration request signal that the first mobile terminal transmits in the first communication area at each time, The first learning step performs learning based on the location information of the first mobile terminal acquired in the first acquisition step. A route management method comprising:

11. The route management method according to claim 8 or 10, Furthermore, the second mobile terminal executes: a second acquisition step of acquiring the route management information notified in the notification step; a third acquisition step of acquiring a time-based location of the terminal within the first communication area; a calculation step of providing the position of the terminal at each time acquired in the third acquisition step as an unknown input to the trained second machine learning model, performing a calculation of the trained second machine learning model, and outputting the action value function indicating the value of an action proceeding sequentially from the position of the terminal at each time; a movement control step of controlling the movement of the terminal based on the optimal movement route within the first communication area determined based on the action value function output in the calculation step; A route management method comprising:

12. The route management method according to claim 7, The first learning step learns, by reinforcement learning, the action value function indicating a value of an action of the first mobile terminal proceeding sequentially from the position of the first mobile terminal at each time from the initial point, which is a position where the first mobile terminal arrives at the first communication area, to the final point, which is a position where the first mobile terminal leaves the first communication area. A route management method comprising:

Images