Route management system and route management method

Abstract

The goal is to share optimal actions across multiple devices through a simpler configuration. [Solution] The route management system includes: a first learning unit 20 that learns the optimal travel route within the communication area 30A, which indicates the path that the first mobile terminal 2a should sequentially take from its current position at each moment, based on the time-based location information of the first mobile terminal within the communication area 30A, using a first machine learning model; a second learning unit 12 that learns the relationship between the current position of the first mobile terminal 2a and the optimal travel route within the communication area 30A obtained through learning by the first learning unit 20, using a second machine learning model; and a notification unit 17 that, upon receiving a location registration request signal transmitted by the second mobile terminal 2b when it reaches the communication area 30A, notifies the second mobile terminal 2b of the learned second machine learning model as route management information for the communication area 30A.

Description

[Technical Field]

[0001] The present invention relates to a route management system and a route management method. [Background technology]

[0002] Conventionally, there are known technologies that use machine learning to learn the optimal behavior of devices such as robots (see Patent Document 1). However, the technology described in Patent Document 1 does not consider sharing the optimal behavior of one device with other devices. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-173643 [Overview of the project] [Problems that the invention aims to solve]

[0004] With conventional technology, it was difficult to share optimal actions across multiple devices using a simpler configuration.

[0005] This invention was made to solve the above-mentioned problems and aims to enable multiple devices to share the optimal action with a simpler configuration. [Means for solving the problem]

[0006] To solve the above-mentioned problems, the route management system according to the present invention comprises: a first learning unit configured to learn, using a first machine learning model, the optimal travel route within the first communication area, which indicates the path that the first mobile terminal should sequentially take from its position at each given time, based on the time-based location information of the first mobile terminal within the first communication area; a second learning unit configured to learn, using a second machine learning model, the relationship between the current position of the first mobile terminal and the optimal travel route within the first communication area obtained through learning by the first learning unit; a storage unit configured to store the learned second machine learning model constructed through learning by the second learning unit; and a notification unit configured to notify the second mobile terminal of the learned second machine learning model as route management information for the first communication area in response to receiving a second location registration request signal transmitted when the second mobile terminal reaches the first communication area.

[0007] Furthermore, in the route management system according to the present invention, the route management device further comprises the second learning unit and the notification unit, the first mobile terminal comprises the first learning unit, and the first mobile terminal transmits the optimal travel route of the first communication area obtained through learning by the first learning unit to the route management device, triggered by a first location registration request signal transmitted when it reaches a second communication area adjacent to the first communication area.

[0008] Furthermore, in the route management system according to the present invention, the first location registration request signal transmitted by the first mobile terminal in the first communication area is associated with a first group identifier, which is an identifier of the group to which the first mobile terminal belongs, the storage unit stores the first group identifier associated with the trained second machine learning model, and the notification unit may notify the second mobile terminal of the trained second machine learning model as route management information if the group identifier associated with the second location registration request signal transmitted by the second mobile terminal when it reaches the first communication area is the first group identifier.

[0009] Furthermore, in the route management system according to the present invention, the route management device further comprises a first learning unit, a second learning unit, and a notification unit, wherein the route management device further comprises a first acquisition unit configured to acquire location information of the first mobile terminal attached to a first location registration request signal transmitted by the first mobile terminal at regular intervals in the first communication area, and the first learning unit may perform learning based on the location information of the first mobile terminal acquired by the first acquisition unit.

[0010] Furthermore, in the route management system according to the present invention, the second mobile terminal may include a second acquisition unit configured to acquire the route management information notified by the notification unit, a third acquisition unit configured to acquire the current location of the terminal, a calculation unit configured to provide the current location of the terminal acquired by the third acquisition unit as an unknown input to the trained second machine learning model, perform calculations on the trained second machine learning model to output the optimal movement path within the first communication area that indicates the path to be taken sequentially from the current location of the terminal, and a movement control unit configured to control the movement of the terminal based on the optimal movement path within the first communication area output by the calculation unit.

[0011] Furthermore, in the route management system according to the present invention, the first learning unit may learn, by reinforcement learning, the optimal travel route within the first communication area, which indicates the path that the first mobile terminal should sequentially take from the position of the first mobile terminal at each time step, from the position where it arrives in the first communication area to the position where it leaves the first communication area.

[0012] To solve the above-mentioned problems, the route management method according to the present invention comprises: a first learning step in which a first mobile terminal learns an optimal travel route within the first communication area, which indicates the path that the first mobile terminal should sequentially take from its position at each time, based on the time-based position information of the first mobile terminal within the first communication area, using a first machine learning model; a second learning step in which a second machine learning model learns the relationship between the current position of the first mobile terminal and the optimal travel route within the first communication area obtained through the learning in the first learning step; a storage step in which the learned second machine learning model constructed through the learning in the second learning step is stored in a storage unit; and a notification step in which, upon receiving a second position registration request signal transmitted when the second mobile terminal reaches the first communication area, the learned second machine learning model is notified to the second mobile terminal as route management information for the first communication area.

[0013] Furthermore, in the route management method according to the present invention, the route management device may perform the second learning step and the notification step, the first mobile terminal may perform the first learning step, and the first mobile terminal may transmit the optimal travel path of the first communication area obtained through learning in the first learning step to the route management device, triggered by a first location registration request signal transmitted when it reaches a second communication area adjacent to the first communication area.

[0014] Furthermore, in the route management method according to the present invention, the first location registration request signal transmitted by the first mobile terminal in the first communication area is associated with a first group identifier, which is an identifier of the group to which the first mobile terminal belongs; the storage step stores the first group identifier associated with the trained second machine learning model in the storage unit; and the notification step may notify the second mobile terminal of the trained second machine learning model as route management information if the group identifier associated with the second location registration request signal transmitted by the second mobile terminal when it reaches the first communication area is the first group identifier.

[0015] Furthermore, in the route management method according to the present invention, the route management device performs the first learning step, the second learning step, and the notification step, and further includes a first acquisition step performed by the route management device to acquire location information of the first mobile terminal attached to the first location registration request signal transmitted by the first mobile terminal at regular intervals in the first communication area, wherein the first learning step may perform learning based on the location information of the first mobile terminal acquired in the first acquisition step.

[0016] Furthermore, the route management method according to the present invention may further include: a second acquisition step performed by the second mobile terminal to acquire the route management information notified in the notification step; a third acquisition step to acquire the current location of the terminal itself; a calculation step in which the current location of the terminal itself acquired in the third acquisition step is given as an unknown input to the trained second machine learning model, and the trained second machine learning model performs calculations to output the optimal movement path within the first communication area that indicates the path to be taken sequentially from the current location of the terminal itself; and a movement control step in which the movement of the terminal itself is controlled based on the optimal movement path within the first communication area output in the calculation step.

[0017] Furthermore, in the route management method according to the present invention, the first learning step may learn, by reinforcement learning, the optimal travel route within the first communication area, which indicates the route that the first mobile terminal should sequentially take from the position of the first mobile terminal at each time step, from the position where it arrives in the first communication area to the position where it leaves the first communication area. [Effects of the Invention]

[0018] According to the present invention, upon receiving a second location registration request signal transmitted by a second mobile terminal upon reaching a first communication area, the second mobile terminal is notified of a trained second machine learning model as route management information within the first communication area. Therefore, optimal actions can be shared among multiple devices with a simpler configuration. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 is a block diagram showing the configuration of a route management system according to an embodiment of the present invention. [Figure 2] Figure 2 is a diagram illustrating the overview of the route management system according to this embodiment. [Figure 3] Figure 3 is a diagram illustrating the structure of the configuration information table provided by the route management system according to this embodiment. [Figure 4] Figure 4 is a block diagram showing the configuration of a mobile terminal included in the route management system according to this embodiment. [Figure 5] Figure 5 is a diagram illustrating the optimal travel route stored by the route management system according to this embodiment. [Figure 6] Figure 6 is a schematic diagram showing the configuration of the first learning unit according to this embodiment. [Figure 7] Figure 7 is a block diagram showing the configuration of the first learning unit according to this embodiment. [Figure 8] Figure 8 is a schematic diagram illustrating the configuration of the second learning process according to this embodiment. [Figure 9] Figure 9 is a block diagram showing the hardware configuration of the route management device according to this embodiment. [Figure 10] Figure 10 is a block diagram showing the hardware configuration of the mobile terminal according to this embodiment. [Figure 11] Figure 11 is a sequence diagram showing the operation of the route management system according to this embodiment. [Figure 12] Figure 12 is a sequence diagram showing the operation of the route management system according to this embodiment. [Figure 13] Figure 13 is a sequence diagram showing the operation of the route management system according to this embodiment. [Figure 14] Figure 14 is a flowchart showing the first learning process of the route management system according to this embodiment. [Figure 15]Figure 15 is a flowchart showing the first learning process of the route management system according to this embodiment. [Figure 16] Figure 16 is a block diagram showing the configuration of a route management device included in a modified version of this embodiment. [Figure 17] Figure 17 is a block diagram showing the configuration of a mobile terminal included in a modified route management system according to this embodiment. [Figure 18] Figure 18 is a sequence diagram showing the operation of a route management system according to a modified example of this embodiment. [Modes for carrying out the invention]

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to Figures 1 to 18.

[0021] [Configuration of the routing management system] First, with reference to Figure 1, an overview of a route management system comprising a route management device 1 and a mobile terminal 2 according to an embodiment of the present invention will be described.

[0022] The route management system according to this embodiment comprises a route management device 1 and a plurality of mobile terminals 2, and the route management device 1 and the mobile terminals 2 are connected to each other so as to be able to communicate via a wireless communication network NW that conforms to a predetermined communication standard such as LTE / 4G, 5G, or 6G. When a mobile terminal 2 reaches a communication area 30A, the route management system notifies the mobile terminal 2 of route management information that indicates the optimal travel route within the communication area 30A, which has been learned in advance, in response to a location registration request signal transmitted by the mobile terminal 2.

[0023] Mobile terminal 2 includes aircraft capable of autonomous flight such as drones and unmanned aerial vehicles, autonomous vehicles, mobile robots, excavators, and ships. The mobile space in which mobile terminal 2 moves is each communication area 30. In this embodiment, there is a first mobile terminal 2a and a plurality of second mobile terminals 2b. The first mobile terminal 2a moves in the communication area 30A ahead of the other mobile terminals 2. The second mobile terminals 2b arrive in the communication area 30A after the first mobile terminal 2a. Hereinafter, when the first mobile terminal 2a and the plurality of second mobile terminals 2b are not distinguished, they are collectively referred to as "mobile terminal 2".

[0024] A group ID (group identifier) ​​is assigned to a group of mobile terminals 2, including a first mobile terminal 2a and a plurality of second mobile terminals 2b, based on predetermined attributes. The group ID can be assigned to terminals that share common attributes related to communication and services. The mobile management processing in this embodiment is performed for each of the multiple mobile terminals 2 with the same group ID.

[0025] In the following explanation, we will use the case where mobile terminal 2 is a mobile robot as an example. Mobile terminal 2 controls autonomous movement using a controller that processes information from sensors 208 and other devices (described later) to control the rotation speed of the motor 209 and the drive mechanism 210. Mobile terminal 2 also acquires its own GPS position using a GPS receiver 207. Mobile terminal 2 is configured as an IoT terminal with an IP address, and each IP address can uniquely identify mobile terminal 2.

[0026] Furthermore, the mobile terminal 2 according to this embodiment is equipped with a SIM card and has an IMSI (International Mobile Subscriber Identity) stored in the SIM card. The mobile terminal 2 is uniquely identified by its IMSI. When the mobile terminal 2 crosses the communication area 30 of its destination, and at regular intervals, it transmits a location registration request signal to the base station 3 and the core network 4. The location registration request signal includes the IMSI of the mobile terminal 2. The mobile terminal 2 also transmits its own location information attached to the location registration request signal. Each mobile terminal 2 has the same configuration. Details of the functional blocks and hardware configuration of the mobile terminal 2 will be described later.

[0027] As shown in Figure 1, the mobile terminal 2 moves sequentially across L communication areas 30A (first communication area) and 30B (second communication area) covered by L base stations 3A and 3B (where L is an integer greater than or equal to 2). Communication area 30A is the communication area 30 immediately preceding communication area 30B, and these are any adjacent communication areas 30 out of the L communication areas 30.

[0028] Figure 2 shows the mobile space corresponding to communication area 30A. The mobile space is a three-dimensional matrix composed of unit spaces divided into multiple spaces. Each unit space constituting the mobile space has the same volume. Furthermore, each unit space has a node ID, and each unit space is represented by a single position (x, y, z). Position information can be obtained using three-dimensional GPS position coordinates consisting of latitude, longitude, and altitude. For example, a representative value such as the center position of the unit space can be used as the position of that unit space.

[0029] As shown in Figure 2, the mobile terminal 2 moves from a unit space position corresponding to the initial point S upon reaching communication area 30A, using each unit space as a waypoint, to the unit space position of the final point G, which is the position upon leaving communication area 30A. For example, the final point G for each communication area 30A and 30B is assumed to be predetermined. Furthermore, upon leaving communication area 30A and reaching the adjacent communication area 30B, the mobile terminal 2 similarly moves to a position away from the position upon reaching communication area 30B. In this way, in this embodiment, the optimal travel route for each communication area 30 is learned in advance and shared with multiple mobile terminals 2 that reach each communication area 30. The optimal travel route is learned by the first mobile terminal 2a, which moves through the communication area 30 first, and transmitted to the route management device 1 via the network NW.

[0030] Base station 3 is a wireless base station compatible with the 5G communication standard and relays communication between mobile terminals 2 located in the communication area 30 and the core network 4. In this embodiment, base station 3 has L base stations 3A and 3B (where L is an integer of 2 or more), each covering communication areas 30A and 30B. Base station 3 is connected to the core network 4 via a backhaul link. In this embodiment, one base station 3 covers one communication area 30, and base station 3 and communication area 30 are identified by the base station ID. Hereinafter, when base stations 3A, 3B and communication areas 30A and 30B are not distinguished, they may be collectively referred to as "base station 3" and "communication area 30," respectively.

[0031] Core network 4 is connected to routing device 1 via a network NW such as LAN, WAN, or the Internet. Core network 4 includes nodes within the C-plane: AMF (Access and Mobility Management Function) 40, UDM (Unified Data Management) / UDR (Unified Data Repository) 41, SMF (Session Management Function) 42, and PCF (Policy Control Function) 43. Core network 4 also includes multiple UPF (User Plane Function) 44 within the U-plane. Functional nodes within the U-plane and C-plane that are included in core network 4 are not shown in the diagram.

[0032] The AMF40 is a node that provides mobility control functions and performs movement control such as location registration, paging, and handover. The location information of the mobile terminal 2, which is attached to the location registration request signal received by the AMF40, is transmitted to the UDM / UDR41.

[0033] The UDM / UDR41 manages subscriber profiles, performs authentication, and manages mobility. In this embodiment, the UDM / UDR41 adds a location information field to the subscriber profile. The UDM / UDR41 according to this embodiment includes a communication interface 41a for communicating with the route management device 1. In the UDM / UDR41, location information is stored using the IMSI included in the location registration request signal as the key. The transmission timestamp of the location registration request signal identifies when and at what GPS location the mobile terminal 2, identified by the IMSI, is located.

[0034] Furthermore, in response to instructions from the routing device 1, the UDM / UDR41 creates and stores a configuration information table T1, as shown in Figure 3, which associates the IMSI, group ID, base station ID of multiple mobile terminals 2, and the communication path of the UPF44 set in the data communication of the mobile terminals 2. The instructions from the routing device 1 include grouping information that associates multiple IMSIs and group IDs to be managed, which have been specified in advance. The IP address value of the UPF44 in the configuration information table T1 is registered after the communication path is set by the SMF42 and PCF43 described later. The UDM / UDR41 can configure the configuration information table T1 by adding a group ID field to the subscriber profile. The configuration information table T1 managed by the UDM / UDR41 is also synchronized with the routing device 1, and the same contents are stored there (fourth storage unit 16).

[0035] In addition, the UDM / UDR41 may be configured as a single device with the UDM and UDR, or it may be a device in which the UDM and UDR are arranged separately.

[0036] SMF42 is a session management function that establishes, modifies, and releases PDU (Packet Data Unit) sessions between mobile terminal 2 and data networks such as the Internet. Based on the PCC (Policy and Charging Control) policy from PCF43, SMF42 sets the appropriate communication path for data communication between mobile terminal 2 and UPF44.

[0037] PCF43 determines QoS and policies and provides them to SMF42. PCF43 applies PCC rules according to the 3GPP (registered trademark) specification and creates PCC policies for configuring the communication path of UPF44 that the mobile terminal 2 communicates with, in response to instructions from the routing device 1.

[0038] UPF44 is a user plane function that processes data between the base station 3 and a data network (DN) 5 such as the internet. In this embodiment, UPF44 includes a communication interface 44a for communicating with the routing device 1.

[0039] Data network 5 refers to external networks such as the internet and cloud networks.

[0040] In this embodiment of the route management system, the first mobile terminal 2a, which moves first within the communication area 30A, learns the optimal movement path within the communication area 30A using reinforcement learning. Furthermore, the optimal movement path within the communication area 30A obtained through reinforcement learning is transmitted to the route management device 1 when the first mobile terminal 2a reaches the next communication area 30B, triggered by a location registration request signal. The route management device 1 learns the relationship between the current location of the mobile terminal 2 and the optimal movement path within the communication area 30A learned by the first mobile terminal 2a using a supervised learning model. Furthermore, it notifies the second mobile terminal 2b of the learned supervised learning model as route management information for managing the route of the subsequent second mobile terminal 2b.

[0041] The second mobile terminal 2b, based on the notified route management information, takes its current position in a unit space as an unknown input, performs calculations on a pre-trained supervised learning model, and outputs the sequential path it should take. Then, based on the outputted sequential path, it determines the path from the initial point to the final point of the communication area 30A and controls its movement within the communication area 30A.

[0042] Upon receiving notification of route management information, the second mobile terminal 2b changes its course according to the determined course, as shown by the arrows in Figure 2, and moves in the direction it should proceed in each unit space. The course can include various paths, i.e., directions of movement. In Figure 2, the movement space is explained in a two-dimensional plane, but the course of the mobile terminal 2 can be a three-dimensional course. Here, the course refers to the movement from one position in each unit space to the position in each adjacent unit space. The route also includes the entire route from the initial position to the final position.

[0043] [Functional blocks of the route management device] As shown in Figure 1, the route management device 1 comprises a setting unit 10, a route acquisition unit 11, a second learning unit 12, a first storage unit 13, a second storage unit (storage unit) 14, a third storage unit 15, a fourth storage unit 16, and a notification unit 17. The route management device 1 manages the routes of the mobile terminals 2 for each communication area 30, which is a mobile space.

[0044] The configuration unit 10 specifies the IMSI and group ID based on the location registration request signal from each mobile terminal 2 and instructs the UPF44 settings for each mobile terminal 2. Specifically, the configuration unit 10 transmits grouping information to the UDM / UDR41, which associates the previously registered managed IMSI and group ID. Based on the grouping information from the configuration unit 10, the UDM / UDR41 creates a configuration information table T1 (Figure 3). Furthermore, the configuration unit 10 instructs the user plane function settings for data communication of each mobile terminal 2 based on the group ID associated with the IMSI included in the location registration request signal from each mobile terminal 2.

[0045] The configuration unit 10 requests the creation of a PCC policy from the PCF43 of the core network 4, specifying the group ID, IMSI, and the IP address of the routing device 1. In response to the creation request, the PCF43, SMF42, and UDM / UDR41 of the core network 4 cooperate to set the appropriate communication path for UPF44 for the data communication of the mobile terminal 2. Once the UPF44 communication path is set for the data communication of the mobile terminal 2, the configuration information table T1 updated in the UDM / UDR41 is synchronized with, for example, the fourth storage unit 16, and the updated configuration information table T1 is stored in the fourth storage unit 16.

[0046] The route acquisition unit 11 acquires the optimal travel route in communication area 30A learned by the first mobile terminal 2a. As mentioned above, the first mobile terminal 2a is a terminal that moves in communication areas 30A and 30B ahead of other mobile terminals 2. The first mobile terminal 2a learns the optimal travel route within communication area 30A while moving ahead in communication area 30A. The route acquisition unit 11 acquires the optimal travel route in the previous communication area 30A from the first mobile terminal 2a at the timing when the first mobile terminal 2a transmits a location registration request signal (first location registration request signal) upon reaching the next communication area 30B. A group ID is associated with the location registration request signal transmitted by the first mobile terminal 2a.

[0047] The second learning unit 12 learns the relationship between the current location of the first mobile terminal 2a and the optimal travel route within the communication area 30A, which is obtained by the route acquisition unit 11, using a supervised learning model (second machine learning model). Details of the second learning unit 12 will be described later.

[0048] The first storage unit 13 stores the optimal travel route for the communication area 30A that the preceding first mobile terminal 2a has learned and constructed, which has been acquired by the route acquisition unit 11. The first storage unit 13 stores the optimal travel route in association with a group ID. The first storage unit 13 may also store the optimal travel route in association with a base station ID.

[0049] The second memory unit 14 (memory unit) stores the trained supervised learning model constructed by the second learning unit 12 through supervised learning. The second memory unit 14 stores the trained supervised learning model in association with a group ID. The second memory unit 14 may also store the trained supervised learning model in association with a base station ID.

[0050] The third memory unit 15 stores the final destination of each communication area 30A and 30B. The third memory unit 15 also stores location information of the unit spaces that constitute the mobile space of communication areas 30A and 30B, and identification information such as the IMSI of the mobile terminal 2.

[0051] The fourth storage unit 16 stores grouping information that associates IMSI with group ID, and the configuration information table T1 stored in the UDM / UDR 41.

[0052] The notification unit 17, upon receiving a location registration request signal (second location registration request signal) transmitted by the second mobile terminal 2b when it reaches the communication area 30A, notifies the second mobile terminal 2b of the trained supervised learning model as route management information for the communication area 30A. More specifically, the notification unit 17 notifies the second mobile terminal 2b of the trained supervised learning model as route management information if the group ID associated with the location registration request signal transmitted by the second mobile terminal 2b when it reaches the communication area 30A is "1". The notification unit 17 can transmit the route management information to the second mobile terminal 2b via the network NW.

[0053] [Mobile device function blocks] Next, the functional blocks of mobile terminal 2 will be described. Figure 3 is a block diagram showing the configuration of mobile terminal 2. The first mobile terminal 2a and the second mobile terminal 2b can have the same configuration. In this case, mobile terminal 2 functions as the first mobile terminal 2a when it is preceding in the communication area 30, and as the second mobile terminal 2b when it is following.

[0054] The mobile terminal 2 comprises a first learning unit 20, a movement path storage unit 21, a transmission unit 22, a second acquisition unit 23, a third acquisition unit 24, a calculation unit 25, a determination unit 26, a movement control unit 27, and a fifth storage unit 28. When the mobile terminal 2 functions as the preceding first mobile terminal 2a, it learns the optimal movement path in communication areas 30A and 30B and transmits it to the route management device 1. When the mobile terminal 2 functions as the following second mobile terminal 2b, it determines the next path it should take based on the route management information notified by the route management device 1 and controls the movement of its terminal until it reaches the final location within the communication area 30.

[0055] The first learning unit 20 learns, using the first machine learning model, an optimal movement route within the communication area 30A indicating the route that the own terminal should sequentially follow from the position at each time in the communication area 30A, based on the position information of the own terminal at each time in the communication area 30A. As an example, the first learning unit 20 learns, by reinforcement learning, an optimal movement route within the communication area 30A indicating the route that the own terminal should sequentially follow from the position when the own terminal arrives at the communication area 30A to the position where the own terminal leaves the communication area 30A. The position of the own terminal at each time is acquired by a third acquisition unit 24 described later.

[0056] The first learning unit 20 applies a reward function to the estimation result of calculating the route that should be sequentially followed until the own terminal reaches the position of the final point from the position of the initial point of the communication area 30, and updates it so that the reward for the own terminal to reach the position of the final point is maximized, and learns the policy of the route that the own terminal should sequentially follow from the current position, using a reinforcement learning model.

[0057] In the present embodiment, as a policy of the route that the mobile terminal 2 should sequentially follow from the position of each unit space, an action a related to movement in a predetermined n (n is an integer of 2 or more) directions with respect to the traveling direction n is exemplified. Also, the traveling direction is the direction based on the position of the unit space where the mobile terminal 2 was immediately before.

[0058] The first learning unit 20 uses, as a reinforcement learning model, a neural network model including an input layer s, a hidden layer h, and an output layer q as shown in FIG. 6. Also, as a neural network model, a state s that is the position of the mobile terminal 2 t is received, and all action value functions Q(s t , a1), Q(s t , a2), Q(s t , a3), ···, Q(s t , a n-1 ), Q(s t , a n ) is output, and Deep Q-Network (DQN), a neural network, is adopted.

[0059] More specifically, the first learning unit 20 receives the current position in a unit space, which indicates the current position of the mobile terminal, as input to the neural network model, performs calculations on the neural network model, and determines the next path that the mobile terminal 2 should take from its current position in the unit space, and determines the actions a for each of the n directions of movement. n This outputs the first estimate Q1 of the action-value function, which represents the expected value of the cumulative future rewards obtained when taking this action.

[0060] The reward is a state s indicating the current location of mobile terminal 2, and an action a in which mobile terminal 2 moves in a predetermined direction. n The reward function r=r(s,a,s') is given by the next position of the mobile terminal 2, i.e., the next state s'. In this embodiment, the reward function includes as a variable the degree to which the mobile terminal 2 has reached a unit space position related to its final destination. In addition, it can also include as a variable the degree to which it has reached a unit space position corresponding to a space with obstacles such as skyscrapers or transmission towers. For example, if the mobile terminal 2 moves closer to the final destination or reaches the final destination by the shortest distance through an action related to movement in a predetermined direction, the reward, which is a scalar quantity, is set to a larger value.

[0061] On the other hand, if the mobile terminal 2 moves away from the final destination or reaches a unit space containing obstacles, it can be designed to give a negative reward value (for example, r = -1). By setting the reward for a unit space containing obstacles as a negative value in this way, the mobile terminal 2 can avoid these points and reach the destination.

[0062] Furthermore, the first learning unit 20 receives the next unit space location reached by its own terminal as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate Q2 of the action-value function. The first learning unit 20 learns the weight parameters of the neural network model so that the first estimate Q1 becomes the target value calculated from the second estimate Q2.

[0063] If we denote the weight parameters of the neural network model as θ and the action-value function as Q(s,a;θ), the learning minimization loss function is given by the following equation (1). L(θ) = 1 / 2{r + γmax} a’ Q(s',a';θ)-Q(s,a;θ)} 2 ...(1)

[0064] In equation (1) above, r is the reward (immediate reward) and γ is the discount rate. Q(s,a;θ) corresponds to the first estimate Q1, and Q(s',a';θ) corresponds to the value of the action in state s' after one step, i.e., the second estimate Q2. The target value is r + γmax a’ It can be represented by Q(s',a';θ).

[0065] The first learning unit 20 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by equation (1) above.

[0066] More specifically, the first learning unit 20 can employ a Fixed Target Q-Network using two neural networks, main QN221 and target QN223, as shown in Figure 7. Main QN221 selects the optimal action and updates the action-value function Q. Meanwhile, target QN223 estimates and evaluates the value of the action a' to be taken in the next state s' resulting from the action. Main QN221 and target QN223 have neural networks with the same layer structure, but the parameter of main QN221 is "θ" and the parameter of target QN223 is "θ". - It is given by ".

[0067] The main QN221 receives the current position of the mobile terminal 2 as state s from the environment 220. The environment 220 is the system of the mobile space in which the mobile terminal 2 is located, i.e., the communication area 30. Under this environment 220, the mobile terminal 2 moves to another unit space by taking action a related to movement in a predetermined direction, transitions to the next state s', and simultaneously receives a reward r from the environment 220.

[0068] The first learning unit 20 inputs the state s related to the current position of the mobile terminal 2 to the main QN221 and calculates the action-value function Q(s,a;θ). The first learning unit 20 calculates action a using, for example, the ε-greedy method, or the optimal action argmax at the present time. a We find Q(s,a;θ). In environment 220, mobile terminal 2 takes action related to the optimal path at the present time. a Perform Q(s,a;θ). Environment 220 is where mobile terminal 2 takes action argmax. a As a result of performing Q(s,a;θ), the position in the unit space at the destination is observed as the next state s', and the reward r is output. Experience data 224 stores the experience (s,a,r,s') output from environment 220.

[0069] The first learning unit 20 calculates the loss function L in the DQN loss calculation 222 and updates the weights of the main QN 221 using the gradient of the loss function L.

[0070] The first learning unit 20 periodically copies the weights of the main QN221 to the target QN223 and synchronizes them. The synchronization of the target QN223 is performed at a lower frequency than the update frequency of the weights of the main QN221. The first learning unit 20 extracts experience from the experience data 224, inputs the past state into the target QN223, and estimates the max value. a’ Q(s',a';θ - The first learning unit 20 outputs the estimated value max output by the target QN223. a’ Q(s',a';θ - ) Target value r+γmax a’ Q(s',a';θ - Using this method, the weights of the main QN221 are trained with a DQN loss calculation of 222.

[0071] The movement path memory unit 21 stores the optimal movement path, i.e., the learned reinforcement learning model, which shows the sequential path to take from the initial position of the communication area 30A to the final position, as learned by the first learning unit 20. Figure 5 shows the optimal movement path stored in the movement path memory unit 21. The timestamp [t] in Figure 5 indicates the timing when the third acquisition unit 24 acquired the GPS position of the terminal. Thus, the optimal movement path is information that associates the GPS position of the terminal at each time point with the action value function Q at that position.

[0072] The transmission unit 22 transmits the optimal travel path in communication area 30A obtained by the first learning unit 20 to the route management device 1 at the timing when it transmits a location registration request signal when its own terminal reaches communication area 30B adjacent to communication area 30. Specifically, the transmission unit 22 transmits the optimal travel path after the data communication path of its own terminal has been set in response to the location registration request signal.

[0073] The first learning unit 20 and the movement path storage unit 21 described above are necessary when the mobile terminal 2 functions as a preceding first mobile terminal 2a. On the other hand, the second acquisition unit 23, calculation unit 25, determination unit 26, movement control unit 27, and fifth storage unit 28 described below are necessary when the mobile terminal 2 functions as a subsequent second mobile terminal 2b.

[0074] The second acquisition unit 23 acquires route management information notified by the notification unit 17 of the route management device 1.

[0075] The third acquisition unit 24 acquires the current location of the terminal. More specifically, the third acquisition unit 24 acquires the location of the unit space where the terminal is located at each time t, based on the GPS location of the terminal. The third acquisition unit 24 refers to the fifth storage unit 28 and acquires the location in the unit space corresponding to the current GPS location received by the GPS receiver 207 as the current location of the terminal.

[0076] The calculation unit 25 provides the current position of the terminal acquired by the third acquisition unit 24 as an unknown input to a pre-trained supervised learning model, performs calculations on the pre-trained supervised learning model, and outputs the optimal movement path within the communication area 30A that indicates the path the terminal should sequentially take from its current position.

[0077] The decision unit 26 determines the next path to take from the current position of its terminal, based on the optimal movement path within the communication area 30A output by the calculation unit 25. More specifically, based on the path strategy output by the calculation unit 25, it determines the current position in a unit space and states s t For each state s t The system then determines the path to take by selecting the path that maximizes the value of the action-value function Q, which is action a.

[0078] The movement control unit 27 controls the movement of its own terminal based on the optimal movement path within the communication area 30A output by the calculation unit 25. Specifically, the movement control unit 27 controls the movement of its own terminal based on the next path to be taken, which is determined by the determination unit 26. The movement control unit 27 can calculate a control command for the next path to be taken from the current position and transmit the control command value to the motor 209.

[0079] The fifth memory unit 28 stores map data including the position coordinates of each communication area 30, which is a mobile space, and information that associates the position coordinates of the unit spaces constituting the mobile space with the node IDs of the unit spaces.

[0080] Here, the second learning unit 12 of the route management device 1 described above will be explained in more detail with reference to Figure 8.

[0081] Figure 8 shows the structure of a neural network model adopted as an example of a supervised learning model used by the second learning unit 12. The neural network model comprises an input layer x, a hidden layer h, and an output layer y. The second learning unit 12 provides the current unit space position of the mobile terminal 2, i.e., the unit space position corresponding to the GPS position of the mobile terminal 2 at each time t, to the input layer of the neural network model, applies an activation function to the weighted sum of the inputs, and passes the output determined by thresholding to the output layer. Each output node of the output layer outputs the model's predicted output corresponding to n action-value functions Q at each time t in one communication area 30.

[0082] The second learning unit 12 introduces the objective function E shown in equation (2) below, and learns the parameters of the neural network model so that the predicted path that the mobile terminal 2 should take sequentially from its current position within the communication area 30, which is a value from the neural network model for the mobile terminal 2's current position, becomes the optimized value of the path that the mobile terminal 2 should take sequentially from its current position, which has been reinforced and learned by the first learning unit 20 of the mobile terminal 2.

[0083]

number

[0084] In equation (2) above, y1, y2, ..., y n The predicted output values ​​for each output node are shown. Also, Y1, Y2, ..., Y n Here, the labels are the training data, and here they are n optimized action-value functions Q for the current position within the communication area 30, obtained by reinforcement learning by the first learning unit 20 of the mobile terminal 2. Furthermore, as mentioned above, the first learning unit 20 of the mobile terminal 2, as shown in Figure 5, calculates the action-value function Q1(s1,a1)~Q1(s1,a n ), from time t m Position (X m ,Y m ,Z m ) Action value function Q m (s m ,a1)~Qm (s m ,a n The data up to ) will be used as training data.

[0085] The objective function E in equation (2) above is further the optimized action-value function Q1~Q for the travel path within the communication area 30. m It can be configured to comprehensively learn. In this case, when Y1 is the label at the first time t1 position (X1, Y1, Z1), the n optimized action-value functions Q1(s1, a1), Q1(s1, a2), Q1(s1, a3), ..., Q1(s1, a n-1 ), Q1(s1,a n ), when Y2 is the label at the second time t2 position (X2,Y2,Z2), the n optimized action-value functions Q2(s2,a1), Q2(s2,a2), Q2(s2,a3), ..., Q2(s2,a n-1 ), Q2(s2,a n ), ..., the mth time t m Position (X m ,Y m ,Z m ) Label Y m The optimized n action-value functions Q that make up the m (s m a1), Q m (s m a2), Q m (s m a3), ..., Q m (s m ,a n-1 ), Q m (s m ,a n ) is used as training data.

[0086] In this case, equation (2) above can be further expressed by the following equation (3).

number

[0087] In the objective function E of equation (3) above, y 1_1 ,y1_2 , ···, y 1-n , ···, y 2_1 , y 2_2 , ···, y 2-n , ···, y m_1 , y m_2 , ···, y m-n is the predicted output value of the output nodes of n (n action value functions Q) × m (m positions from time t1 to t m . Y 1_1 ~Y 1_n corresponds to the optimized action value function Q1 at the position at time t1, and Y 2_1 ~Y 2_n corresponds to the optimized action value function Q2 at the position at time t2, and Y m_1 ~Y m_n is the teacher data corresponding to the optimized action value function Q m at the position at time t m .

[0088] In this way, all of the optimized action value functions Q at the positions at each time t obtained from the learned reinforcement learning model are used as teacher data. In the configuration example of the neural network model in FIG. 8, teacher data is shown on the right side of each output node that outputs the predicted output value of the model. From the upper output node in order, Y 1_1 ~Y 1_n shows the n action value functions Q1 at the position at time t1, but for Y2···Y m of the teacher data, each of the n values corresponding to the respective action value functions Q2, ···, Q m is shown with omission.

[0089] The value of the objective function E in the above formula (3) is the output value y with respect to the position in the unit space corresponding to the GPS position of the mobile terminal 2 at time t, which is the above input value x of the supervised learning model 1_1 , y 1_2 , ···, y 1-n , ···, y 2_1 , y 2_2 , ···, y<\ 2-n , ···, y m_1 , y m_2 , ···, y m-nis the target output Y of the teacher data 1_1 ~Y 1_n ,Y 2_1 ~Y 2_n ,···,Y m_1 ~Y m_n becomes 0 when they match. The second learning unit 12 adjusts the weight parameters of the neural network related to the supervised learning model so that the objective function E is minimized, that is, becomes 0. The second learning unit 12 can optimize the objective function E using the error backpropagation method or the like.

[0090] [Hardware Configuration of the Route Management Device] Next, an example of the hardware configuration for realizing the route management device 1 having the above-described functions will be described using FIG. 9.

[0091] As shown in FIG. 9, the route management device 1 can be realized by, for example, a computer including a processor 102, a main storage device 103, a communication interface 104, an auxiliary storage device 105, and an input / output I / O 106 connected via a bus 101, and a program for controlling these hardware resources. Further, the route management device 1 can include a display device 107 connected via the bus 101.

[0092] The processor 102 is realized by a CPU, GPU, FPGA, ASIC, or the like.

[0093] In the main storage device 103, programs for the processor 102 to perform various controls and operations are stored in advance. The functions of the route management device 1 such as the setting unit 10, the route acquisition unit 11, the second learning unit 12, and the notification unit 17 shown in FIG. 1 are realized by the processor 102 and the main storage device 103.

[0094] The communication interface 104 is an interface circuit for network-connecting the route management device 1 and various external electronic devices.

[0095] The auxiliary storage device 105 consists of a read / write storage medium and a drive device for reading and writing various information such as programs and data to the storage medium. The auxiliary storage device 105 can use semiconductor memory such as a hard disk or flash memory as the storage medium.

[0096] The auxiliary storage device 105 has a program storage area for storing the route management program executed by the route management device 1. Furthermore, the auxiliary storage device 105 has an area for storing a supervised learning program. The auxiliary storage device 105 realizes the first storage unit 13, second storage unit 14, third storage unit 15, and fourth storage unit 16 described in Figure 1. Specifically, the auxiliary storage device 105 has an area for storing the final destination of each communication area 30 of the mobile terminal 2. The auxiliary storage device 105 also has an area for storing the position coordinates of the mobile space and the position coordinates of the unit space. Furthermore, the auxiliary storage device 105 has an area for storing identification information such as the IP address of the mobile terminal 2. Furthermore, it may have, for example, a backup area for backing up the above-mentioned data and programs.

[0097] The I / O106 is an input / output device that accepts signals from external devices and outputs signals to external devices.

[0098] The display device 107 is composed of an organic EL display or a liquid crystal display. The display device 107 can display a map of the moving space, the current location and progress of multiple mobile terminals 2 for each group, and the location information of the final destination for each communication area 30.

[0099] [Mobile device hardware configuration] Next, an example of a hardware configuration for realizing the mobile terminal 2 having the functions described above will be explained using Figure 10.

[0100] As shown in Figure 10, the mobile terminal 2 can be realized by, for example, a microcomputer equipped with a processor 202 connected via a bus 201, main memory 203, communication interface 204, auxiliary storage 205, and input / output I / O 206, a program to control these hardware resources, a GPS receiver 207, a sensor 208, a motor 209, a drive mechanism 210, and a battery 211. A controller that controls the autonomous movement of the mobile terminal 2 is realized by a computer such as a microcomputer and a program.

[0101] The main memory 203 contains pre-stored programs for the processor 202 to perform movement control and calculations. The processor 202 and the main memory 203 work together to realize the various functions of the mobile terminal 2, such as the first learning unit 20, the second acquisition unit 23, the calculation unit 25, the determination unit 26, and the movement control unit 27, as shown in Figure 4.

[0102] The communication interface 204 is an interface circuit for network connection between the mobile terminal 2 and the route management device 1. The transmission unit 22, as described in Figure 4, is realized through the communication interface 204.

[0103] The auxiliary storage device 205 consists of a read / write storage medium and a drive device for reading and writing various information such as programs and data to the storage medium. The auxiliary storage device 205 can use semiconductor memory such as a hard disk or flash memory as the storage medium.

[0104] The auxiliary storage device 205 has a program storage area for storing the mobile control program executed by the mobile terminal 2. It also has a program storage area for storing the reinforcement learning program executed by the mobile terminal 2. Furthermore, the auxiliary storage device 205 has an area for storing calculation programs for performing calculations on the trained supervised learning model. The auxiliary storage device 205 enables the realization of the mobile path storage unit 21 and the fifth storage unit 28 described in Figure 4. The auxiliary storage device 205 also has an area for storing identification information such as the IP address of the mobile terminal 2. Furthermore, the auxiliary storage device 205 may have an area for storing an application that notifies the route management device 1 of the GPS location. In addition, it may have, for example, a backup area for backing up the above-mentioned data and programs.

[0105] The I / O206 is an input / output device that accepts signals from external devices and outputs signals to external devices.

[0106] The GPS receiver 207 has a built-in antenna for receiving GPS signals. The GPS receiver 207 enables the third acquisition unit 24 shown in Figure 4.

[0107] Sensor 208 consists of various sensors such as an altitude sensor, attitude sensor, camera, LiDAR, and RADAR. In addition to the GPS receiver 207, the altitude sensor enables the third acquisition unit 24 shown in Figure 4. Furthermore, the controller controls the movement of the mobile terminal 2 based on the various sensor data measured by sensor 208.

[0108] The motor 209 rotates due to a rotational drive, driving the drive mechanism 210 attached to the rotation shaft of the motor 209.

[0109] Battery 211 is a lithium-ion battery or the like, and supplies power to the mobile terminal 2.

[0110] [Route management system operation] Next, the operation of a route management system comprising the route management device 1 and mobile terminal 2 having the above-described configuration will be explained with reference to the sequence shown in Figures 11 to 12.

[0111] Figure 11 shows a sequence illustrating the grouping of mobile terminals 2 by the route management system and the process of setting communication paths for data communication. Below, we will explain the grouping and communication path setting process that occurs when the first mobile terminal 2a, which has been moving first in communication area 30A, has reinforced learning the optimal movement path in communication area 30A and has reached the communication area 30B of the next destination base station 3B.

[0112] First, the route management device 1 transmits grouping information, which includes the group ID and IMSI, to the UDM / UDR41 (step S100). The grouping information is stored in the route management device 1 in advance. Next, the UDM / UDR41 creates a configuration information table T1 (Figure 3) based on the received grouping information (step S101). In step S101, the values ​​for "group ID" and "IMSI" in the configuration information table T1 are registered, but the values ​​for the other items are not yet entered.

[0113] Subsequently, the first mobile terminal 2a, having moved from communication area 30A to communication area 30B, transmits a location registration request signal from base station 3B to core network 4 (step S102). The location registration request signal includes the IMSI of the mobile terminal 2 and the base station ID of base station 3B in the communication area 30B where it is located. When the UDM / UDR 41 receives the location registration request signal via AMF 40, the UDM / UDR 41 associates the group ID with the base station ID and IMSI included in the received location registration request signal using the information in the configuration information table T1 (Figure 3) created in step S101, and then forwards the location registration request signal to the route management device 1 (step S103). Note that in step S103, the value of the base station ID in the configuration information table T1 is set.

[0114] Next, the configuration unit 10 of the route management device 1 requests the PCF43 to create a PCC policy by specifying the IMSI, base station ID, group ID, and its own IP address included in the location registration request signal received in step S103 (step S104). Subsequently, the PCF43 creates a PCC policy based on the specified requirements (step S105). The PCF43 creates a PCC policy that includes information on which UPF44 the data communication of the mobile terminal 2 will pass through. The PCC policy specifies the optimal UPF44 as the communication path for data communication from the mobile terminal 2 to the data network 5.

[0115] Next, PCF43 sends the created PCC policy to SMF42 (step S106). Then, SMF42 sends data communication path configuration information related to user plane functions such as setting communication paths defined in the PCC policy to UPF44, which is specified by the PCC policy (step S107). Next, UPF44 registers the received data communication path configuration information in memory (step S108).

[0116] Next, UPF44 sends an ACK to SMF42, notifying it of its IP address (step S109). Furthermore, SMF42 sends an ACK to the routing device 1, notifying it of UPF44's IP address (step S110). Subsequently, the routing device 1 notifies UDM / UDR41 of UPF44's IP address and sends an ACK to the first mobile terminal 2a (step S111). At this time, the IP address of UPF44 communicating with the first mobile terminal 2a is set in the configuration information table T1. The configuration information table T1 updated by UDM / UDR41 is also stored in the fourth storage unit 16 of the routing device 1. After that, a data communication path is established between the first mobile terminal 2a and UPF44 (step S112).

[0117] Subsequently, the transmission unit 22 of the first mobile terminal 2a transmits the optimal travel route within the communication area 30A of the source of travel, which was reinforced and learned by the first learning unit 20, to the route management device 1 via the UPF 44 (step S113) (step S114). The transmission unit 22 transmits the optimal travel route to the route management device 1 along with the base station ID of the communication area 30A. After that, the route management device 1 performs accounting management processing, including the second learning process and notification processing (step S115).

[0118] Figure 12 shows the sequence of events when the subsequent second mobile terminal 2b reaches the communication area 30A after the processing in Figure 11, including the grouping of mobile terminals 2 by the route management system and the setting of a communication path for data communication. The processing from step S100 to step S111 in Figure 12 is the same as that described in Figure 11. The difference from Figure 11 is that in step S112', when the route management device 1 receives a signal indicating that a data communication path has been established, the route management device 1 notifies the second mobile terminal 2b of the route management information for the communication area 30A (step S213). The route management device 1 specifies the group ID and IMSI and notifies the second mobile terminal 2b of the route management information for the communication area 30A via the UPF 44 (step S214).

[0119] Next, the operation of the first mobile terminal 2a, the second mobile terminal 2b, and the route management device 1 will be explained with reference to the sequence in Figure 13. First, the first mobile terminal 2a, with group ID "1", moves first through the communication area 30A. The third acquisition unit 24 of the first mobile terminal 2a acquires its current position in a unit space from its own GPS position (step S1). Subsequently, the first learning unit 20 of the first mobile terminal 2a performs the first learning process (step S2). In the first learning process, the first learning unit 20 applies a reward function to the estimated path that the terminal should sequentially take from the initial position to the final position in the communication area 30A, and updates it to maximize the reward for reaching the final position, learning a strategy for the path that the terminal should sequentially take from its current position using a reinforcement learning model. Details of the first learning process will be described later.

[0120] Subsequently, the first mobile terminal 2a moves from the source communication area 30A to the next destination communication area 30B (step S3). The transmission unit 22 of the first mobile terminal 2a transmits the optimal travel route for communication area 30A learned in step S2 to the route management device 1 at the same time as sending a location registration request signal to the route management device 1 via the core network 4 (step S4). Specifically, step S4 is executed after the grouping and establishment of the data communication path as described in Figure 11 (steps S100 to S112 in Figure 11). The route acquisition unit 11 of the route management device 1 acquires the optimal travel route for communication area 30A from the first mobile terminal 2a.

[0121] Next, the second learning unit 12 of the route management device 1 uses the optimal travel route in the communication area 30A (Figure 3) acquired from the first mobile terminal 2a in step S4 as training data to learn the relationship between the current position of the first mobile terminal 2a and the optimal travel route within the communication area 30A using a supervised learning model (step S5).

[0122] Specifically, the second learning unit 12 repeatedly adjusts and updates parameters such as weights and thresholds to determine the values ​​of these parameters, such that the error between the predicted output value of the sequential path to be taken (when the position in a unit space corresponding to the current GPS position of the first mobile terminal 2a, i.e., the current state) is given to the supervised learning model as input, and the training data, minimizes the objective function E in equation (3) above. In step S5, the second learning unit 12 can determine the parameters that minimize the objective function E by backpropagation or the like.

[0123] Next, the second memory unit 14 stores the trained supervised learning model constructed in step S5 (step S6). In step S6, the trained supervised learning model and the group ID are stored in association. Furthermore, the second memory unit 14 can store the base station ID in association with the trained supervised learning model.

[0124] Subsequently, when the second mobile terminal 2b reaches the communication area 30A, it transmits a location registration request signal to the route management device 1 via the core network 4 (step S7). The location registration request signal transmitted by the second mobile terminal 2b includes the IMSI and is associated with a group ID. Next, the notification unit 17 of the route management device 1 determines, based on the group ID attached to the location registration request signal received in step S7, whether the subsequent mobile terminal 2 belongs to group ID "1" (step S8). Furthermore, it can determine from the base station ID attached to the location registration request signal which communication area 30's route management information to notify. After it is determined that the second mobile terminal 2b is the subsequent mobile terminal 2b of group ID "1", the notification unit 17 notifies the second mobile terminal 2b of the route management information for communication area 30A (step S9). The second acquisition unit 23 of the second mobile terminal 2b acquires the route management information.

[0125] Next, the third acquisition unit 24 of the second mobile terminal 2b acquires the current position in a unit space based on the GPS position of the terminal (step S11). Specifically, the position in a unit space corresponding to the GPS position received by the GPS receiver 207 can be acquired as the current position of the terminal. Next, the calculation unit 25 uses the route management information acquired in step S9, provides the current position in a unit space of the terminal acquired in step S11 as an unknown input, performs calculations on the trained supervised learning model, and outputs a strategy for the path to be taken sequentially from the current position in a unit space (step S12).

[0126] For example, in the mobile space corresponding to the communication area 30A shown in Figure 2, if the initial position of the second mobile terminal 2b within the communication area 30A is input to a trained supervised learning model as its current position at time t=1, then n action-value functions Q for the next steps to take from the initial position at time t=1 will be output. More specifically, if the current position of the second mobile terminal 2b is given as an unknown input, then the values ​​of specific n route policies corresponding to the current position of mobile terminal 2b, selected from the optimized route policies for m positions corresponding to different times, will be output as the calculation result of the trained supervised learning model. The values ​​of the other n route policies that do not correspond to the current position of mobile terminal 2b will be 0.

[0127] Next, the decision unit 26 of the second mobile terminal 2b determines the path to take sequentially by selecting the path that takes the action a with the maximum value of the n action value functions Q output in step S12 (step S13). Next, the movement control unit 27 controls the movement of its own terminal based on the path to take next determined in step S13 (step S14). More specifically, the movement control unit 27 can calculate a control command for the next path to take from the current position and transmit the control command value to the motor 209.

[0128] Mobile terminal 2 repeats the processes from steps S10 to S14 until it reaches the unit space of the final destination from the initial point in communication area 30A (step S15: NO). When it reaches the unit space of the final destination (step S15: YES), the process ends. In this way, the route management system executes the processes from steps S1 to S15 for each communication area 30, so that multiple mobile terminals 2 with a common group ID can move through L communication areas 30 using the optimal route.

[0129] Next, the first learning process of the first mobile terminal 2a will be described in detail with reference to the flowcharts in Figures 14 and 15. First, step S1, described in Figure 13, is executed. First, the first learning unit 20 provides the current state of the terminal, which is the position in the unit space where the terminal is currently located, obtained in step S1, as input to the neural network model, performs calculations on the neural network model, and outputs a first estimate Q1 of the action-value function, which represents the expected value of the cumulative value of future rewards obtained when each action related to movement in a predetermined direction relative to the direction of movement is taken as the next path the terminal should take from its current position in the unit space (step S20).

[0130] Next, the third acquisition unit 24 acquires the position of the terminal in unit space at the next time t as the next state s' (step S21). The next position in unit space reached by the terminal is determined based on the GPS position of the terminal acquired by the third acquisition unit 24 at each time step. Furthermore, the first learning unit 20 provides the position in unit space reached by the terminal, acquired in step S21, as input to the neural network model, performs calculations on the neural network model, and outputs the second estimated value Q2 of the action-value function (step S22).

[0131] Next, the first learning unit 20 calculates a target value from the second estimated value Q2 (step S23). Subsequently, the first learning unit 20 learns the weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S24). Specifically, the first learning unit 20 updates the weight parameters of the neural network model to minimize the loss function in equation (1) above. The first learning unit 20 repeats the process from step S1 to step S24 a set number of times.

[0132] Subsequently, the movement path memory unit 21 stores the trained reinforcement learning model obtained in step S24 (step S25).

[0133] Next, referring to Figure 15, we will explain the first learning process performed by the first learning unit 20 when a Fixed Target Q-Network is adopted, which uses two neural networks: the main QN221 and the target QN223.

[0134] The process in step S1 is the same as the steps of the first learning process described in Figure 13. Subsequently, the first learning unit 20 provides the main QN221 with the position in the unit space where the mobile terminal 2 is currently located, obtained in step S1, as input, performs calculations on the neural network model, outputs the action-value function Q, and calculates the next path a to take (step S120).

[0135] Next, the first learning unit 20 returns the action of the mobile terminal 2 along the path a determined in step S120 to the environment 220, and obtains the next state s' of the mobile terminal 2, which is the position in the unit space where the mobile terminal 2 has moved and the reward r (step S121).

[0136] The first learning unit 20 saves the experience (s, a, r, a') obtained in step S121 to the experience data 224 (step S122). Next, in the DQN loss calculation 222, the first learning unit 20 calculates the loss function L and updates the weights of the main QN 221 using the gradient of the loss function L (step S123). The first learning unit 20 repeats the process from step S120 to step S123 a set number of times.

[0137] Subsequently, the first learning unit 20 periodically copies the weights of the main QN221 to the target QN223 and synchronizes them (step S124). The synchronization of the target QN223 is performed at a lower frequency than the update frequency of the weights of the main QN221. Next, the first learning unit 20 extracts experience from the experience data 224, inputs the past state into the target QN223, and estimates the max a’ Q(s',a';θ - Output (step S126).

[0138] Next, the first learning unit 20 processes the estimated value max output by the target QN223. a’Q(s',a';θ - ) Target value r+γmax a’ Q(s',a';θ - The first learning unit 20 calculates the target value (step S127). Next, the first learning unit 20 calculates the loss function L using the DQN loss calculation 222 with the target value calculated in step S127 (step S128). Next, the first learning unit 20 learns the weights of the main QN 221 to minimize the loss given by the loss function L (step S129). After that, the learned reinforcement learning model, i.e., the optimal movement path in the communication area 30A, is stored in the movement path storage unit 21 (step S25).

[0139] As described above, according to the route management system of this embodiment, the optimal travel route in communication area 30A learned by the preceding first mobile terminal 2a is transmitted to the route management device 1 at the timing when a location registration request signal is transmitted when the first mobile terminal 2a reaches the next communication area 30B. Furthermore, a group determination is performed in response to the location registration request signal transmitted when the subsequent second mobile terminal 2b reaches communication area 30A, and route management information for communication area 30A obtained through learning based on the optimal travel route in communication area 30A learned by the preceding first mobile terminal 2a is notified to the subsequent second mobile terminal 2b. Therefore, the optimal action can be shared among multiple devices with a simpler configuration.

[0140] [Differentiation] Next, a modified version of the embodiment of the present invention will be described. In the embodiment described above, the case in which the mobile terminal 2 is equipped with a first learning unit 20 was described. In contrast, in the route management system according to this modified version, the route management device 1A is equipped with a first learning unit 18 and performs all learning processing for reinforcement learning and supervised learning. Meanwhile, the first mobile terminal 2a, which is moving ahead in the communication area 30A, transmits its own terminal's location information to the route management device 1A.

[0141] [Functional blocks of the route management device] Figure 16 is a block diagram showing the configuration of a modified route management device 1A. As shown in Figure 16, the route management device 1A includes a setting unit 10A, a first acquisition unit 11A, a second learning unit 12, a first storage unit 13, a second storage unit 14, a third storage unit 15, a fourth storage unit 16, a notification unit 17, and a first learning unit 18. The modified route management device 1A differs from the configuration of the route management device 1 in the embodiment in that it includes a first acquisition unit 11A and a first learning unit 18. The following description will focus on the configuration that differs from the route management device 1 in this embodiment.

[0142] The first acquisition unit 11A acquires the location information of the first mobile terminal 2a' which is attached to the location registration request signal transmitted by the first mobile terminal 2a', which is moving ahead in the communication area 30A, at regular intervals within the communication area 30A.

[0143] The first learning unit 18 performs a first learning process based on the location information of the first mobile terminal 2a' at each time interval, which is acquired by the first acquisition unit 11A.

[0144] Furthermore, the setting unit 10A does not perform the grouping process that the setting unit 10 in this embodiment performed, but instead sets the communication path for data communication.

[0145] [Mobile device function blocks] Figure 17 is a block diagram showing the configuration of a modified mobile terminal 2'. The mobile terminal 2' comprises a transmission unit 22, a second acquisition unit 23, a third acquisition unit 24, a calculation unit 25, a determination unit 26, a movement control unit 27, and a fifth storage unit 28. The modified mobile terminal 2' differs in configuration from the mobile terminal 2 of this embodiment in that it does not have a first learning unit 20 and a movement path storage unit 21.

[0146] The transmission unit 22 adds the GPS location of its own terminal, acquired by the third acquisition unit 24, to the location registration request signal and transmits it to the route management device 1A at regular intervals. More specifically, when the transmission unit 22 reaches the communication area 30 of its destination, and at regular intervals, it transmits a location registration request signal with the location information of its own terminal added to it to the route management device 1A via the base station 3 and the core network 4.

[0147] [Route management system operation] Next, the operation of the route management system having the above-described configuration will be explained with reference to the sequence diagram in Figure 18. The content of the following steps S1 to S15 is the same as the processing in the operation of the route management system according to this embodiment described in Figure 13.

[0148] First, when the transmitting unit 22 of the first mobile terminal 2a', which is moving ahead, reaches the communication area 30A, it adds its own terminal's location information to the location registration request signal and transmits it to the route management device 1A (step S1A). In step S1A, while the first mobile terminal 2a' is in the communication area 30A, it transmits a location registration request signal with its own terminal's location information added at regular intervals. The location registration request signal includes the base station ID of the communication area 30A in which it is located.

[0149] Next, the first acquisition unit 11A of the route management device 1A acquires the location information of the first mobile terminal 2a' that is attached to the location registration request signal transmitted by the first mobile terminal 2a' at each time in the communication area 30A (step S1). Subsequently, the first learning unit 18 learns the optimal travel path within the communication area 30A that the first mobile terminal 2a' should sequentially follow from its current position at each time, based on the time-based location information acquired in step S1, using reinforcement learning (step S2). Next, the second learning unit 12 learns the relationship between the current position of the first mobile terminal 2a' and the optimal travel path in the communication area 30A obtained in step S2, using supervised learning (step S5). The second storage unit 14 stores the learned supervised learning model (step S6).

[0150] Subsequently, when the second mobile terminal 2b' reaches the communication area 30A, it transmits a location registration request signal to the route management device 1A via the base station 3B and the core network 4 (step S7). The notification unit 17 of the route management device 1A notifies the second mobile terminal 2b' of the learned supervised learning model, which is the route management information for the communication area 30A, in response to the location registration request signal (step S9). More specifically, the location registration request signal is transmitted from the managed IMSI, and the base station ID is added to the location registration request signal in the core network 4 (Figure 3). Subsequently, a data communication path is set up for the data communication of the second mobile terminal 2b' (Figure 3). In response to the establishment of the data communication path, the notification unit 17 notifies the second mobile terminal 2b' of the route management information for the communication area 30A via the UPF 44.

[0151] The second mobile terminal 2b' acquires route management information for the communication area 30A (step S10), and based on its current location information, performs calculations on a pre-trained supervised learning model, determines the next path to take, and gives directions, repeating the process until it reaches the final point of the communication area 30A (steps S11 to S15).

[0152] As explained above, according to the modified route management system, the route management device 1A performs reinforcement learning and supervised learning to generate route management information for each communication area 30, and notifies each mobile terminal 2 of the route management information in response to a location registration request signal transmitted when each mobile terminal 2 reaches a communication area 30. Therefore, even if the mobile terminal 2 does not have route information, the route of the mobile terminal 2 can be managed from the network side with a simpler configuration. For example, in an environment where the mobile terminal 2 cannot receive GPS signals, it is possible to appropriately guide the mobile route of the mobile terminal 2.

[0153] In the embodiment described, the reinforcement learning model used by the first learning unit 20 is exemplified as a DQN related to a Fixed Target Q-Network composed of a multilayer neural network. However, other reinforcement learning models such as CNNs and multilayer perceptrons can be used. In addition to the DQN exemplified as a reinforcement learning model, other methods such as Double DQN, Dueling DQN, Actor-Critic (AC) method, Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), and Q-learning can also be used.

[0154] Furthermore, in the embodiment described, the supervised learning model used by the second learning unit 12 was exemplified as a multilayer neural network. However, the supervised learning model can also be a multilayer perceptron, a decision tree-based model such as a random forest, or a support vector machine.

[0155] Furthermore, in the embodiments described, the first mobile terminal 2a and the second mobile terminal 2b were described as having the same configuration. However, if the first mobile terminal 2a that moves first is predetermined, the first mobile terminal 2a and the second mobile terminal 2b can have different configurations. In this case, the first mobile terminal 2a is configured to include a first learning unit 20, a movement path storage unit 21, a transmission unit 22, and a third acquisition unit 24. On the other hand, the subsequent second mobile terminal 2b can be configured to include a transmission unit 22, a second acquisition unit 23, a third acquisition unit 24, a calculation unit 25, a determination unit 26, a movement control unit 27, and a fifth storage unit 28.

[0156] Although embodiments of the route management system and route management method of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications that a person skilled in the art can envision are possible within the scope of the invention described in the claims. [Explanation of Symbols]

[0157] 1…Route management device, 10…Setting unit, 11…Route acquisition unit, 12…Second learning unit, 13…First storage unit, 14…Second storage unit, 15…Third storage unit, 16…Fourth storage unit, 17…Notification unit, 2…Mobile terminal, 2a…First mobile terminal, 2b…Second mobile terminal, 20…First learning unit, 21…Movement route storage unit, 22…Transmission unit, 23…Second acquisition unit, 24…Third acquisition unit, 25…Calculation unit, 26…Decision unit, 27…Movement control unit, 28… Fifth memory unit, 101, 201...bus, 102, 202...processor, 103, 203...main memory, 104, 204...communication interface, 105, 205...secondary memory, 106, 206...input / output I / O, 107...display device, 207...GPS receiver, 220...environment, 221...main QN, 222...DQN loss calculation, 223...target QN, 224...experience data, NW...network.

Claims

1. A first learning unit is configured to learn, using a first machine learning model, the optimal travel path within the first communication area, which indicates the path the first mobile terminal should sequentially take from its current position within the first communication area, based on the time-based location information of the first mobile terminal within the first communication area. A second learning unit is configured to learn the relationship between the current location of the first mobile terminal and the optimal travel path within the first communication area, obtained through learning by the first learning unit, using a second machine learning model. A storage unit configured to store the trained second machine learning model constructed by the second learning unit, A notification unit is configured to receive a second location registration request signal transmitted when the second mobile terminal reaches the first communication area, and to notify the second mobile terminal of the trained second machine learning model as route management information for the first communication area. A route management system equipped with the following features.

2. In the route management system according to claim 1, Furthermore, the route management device comprises the second learning unit and the notification unit, The first mobile terminal comprises the first learning unit, The first mobile terminal, upon reaching a second communication area adjacent to the first communication area, transmits a first location registration request signal to the route management device, triggered by the first learning unit's learning process, the optimal travel route within the first communication area. A route management system characterized by the following:

3. In the route management system described in claim 2, The first location registration request signal transmitted by the first mobile terminal in the first communication area is associated with a first group identifier, which is an identifier for the group to which the first mobile terminal belongs. The memory unit stores the first group identifier associated with the trained second machine learning model. The notification unit notifies the second mobile terminal of the trained second machine learning model as route management information if the group identifier associated with the second location registration request signal transmitted when the second mobile terminal reaches the first communication area is the first group identifier. A route management system characterized by the following:

4. In the route management system according to claim 1, Furthermore, the route management device comprises the first learning unit, the second learning unit, and the notification unit, The aforementioned route management device is Furthermore, the system includes a first acquisition unit configured to acquire the location information of the first mobile terminal that is attached to the first location registration request signal transmitted by the first mobile terminal at each time interval in the first communication area. The first learning unit performs learning based on the location information of the first mobile terminal acquired by the first acquisition unit. A route management system characterized by the following:

5. In the route management system according to claim 2 or 4, The aforementioned second mobile terminal is A second acquisition unit configured to acquire the route management information notified by the notification unit, A third acquisition unit configured to acquire the current location of the terminal, A calculation unit is configured to provide the current position of the terminal acquired by the third acquisition unit as an unknown input to the trained second machine learning model, perform calculations on the trained second machine learning model, and output the optimal movement path within the first communication area that indicates the path to be taken sequentially from the current position of the terminal. A movement control unit is configured to control the movement of the terminal based on the optimal movement path within the first communication area output by the calculation unit. A route management system equipped with the following features.

6. In the route management system according to claim 1, The first learning unit learns, through reinforcement learning, the optimal travel path within the first communication area, which indicates the path that the first mobile terminal should sequentially take from its position at each time point, from the position where it arrives at the first communication area to the position where it leaves the first communication area. A route management system characterized by the following:

7. A first learning step in which, based on the time-based location information of the first mobile terminal within the first communication area, the first mobile terminal learns the optimal travel path within the first communication area, which indicates the path it should sequentially take from its time-based location, using a first machine learning model; A second learning step in which the relationship between the current location of the first mobile terminal and the optimal travel path within the first communication area, obtained through learning in the first learning step, is learned using a second machine learning model, A storage step in which the trained second machine learning model constructed in the second learning step is stored in the memory unit, A notification step in which, upon receiving a second location registration request signal transmitted when the second mobile terminal reaches the first communication area, the trained second machine learning model is notified to the second mobile terminal as route management information for the first communication area. A route management method comprising the following features.

8. In the route management method described in claim 7, The route management device performs the second learning step and the notification step, The first mobile terminal performs the first learning step, The first mobile terminal transmits the optimal travel path within the first communication area, obtained through learning in the first learning step, to the route management device, triggered by a first location registration request signal transmitted when it reaches a second communication area adjacent to the first communication area. A route management method characterized by the following.

9. In the route management method described in claim 8, The first location registration request signal transmitted by the first mobile terminal in the first communication area is associated with a first group identifier, which is an identifier for the group to which the first mobile terminal belongs. The memory step involves associating the first group identifier with the trained second machine learning model and storing it in the memory unit. The notification step involves notifying the second mobile terminal of the trained second machine learning model as route management information if the group identifier associated with the second location registration request signal transmitted by the second mobile terminal when it reaches the first communication area is the first group identifier. A route management method characterized by the following.

10. In the route management method described in claim 7, The route management device performs the first learning step, the second learning step, and the notification step, Furthermore, the system includes a first acquisition step in which the route management device acquires the location information of the first mobile terminal that is attached to the first location registration request signal transmitted by the first mobile terminal in the first communication area at each time interval, The first learning step performs learning based on the location information of the first mobile terminal acquired in the first acquisition step. A route management method characterized by the following.

11. In the route management method according to claim 8 or 10, Furthermore, the second mobile terminal performs the following: A second acquisition step for acquiring the route management information notified in the notification step, A third acquisition step involves obtaining the current location of the device, A calculation step in which the current position of the self-terminal acquired in the third acquisition step is given as an unknown input to the trained second machine learning model, the trained second machine learning model performs calculations, and outputs the optimal movement path within the first communication area that indicates the path to be taken sequentially from the current position of the self-terminal, A movement control step that controls the movement of the terminal based on the optimal movement path within the first communication area output in the calculation step, and A route management method comprising the following features.

12. In the route management method described in claim 7, The first learning step involves learning, through reinforcement learning, the optimal travel path within the first communication area, which indicates the path that the first mobile terminal should sequentially take from its position at each time point, from the position where it arrives at the first communication area to the position where it leaves the first communication area. A route management method characterized by the following.

Images