Route management device, route management method, and route management system

Abstract

The objective is to manage the route of a moving object traveling to a destination point via designated waypoints using a simpler configuration. [Solution] The path management device 1 includes a first learning unit 12 that applies a reward function to the estimated path that the mobile body 2 should sequentially take from the initial point to the final point of each of the multiple subpaths, and updates the result so as to maximize the reward for the mobile body 2 to reach the final point, and learns a strategy for the path that the mobile body 2 should sequentially take from its current position using a reinforcement learning model, and a second learning unit 13 that learns the relationship between the current position of the mobile body 2 and the strategy for the path that the mobile body 2 should sequentially take from its current position for each of the multiple subpaths, obtained through learning by the first learning unit 12, using a supervised learning model.

Description

[Technical Field]

[0001] The present invention relates to a route management device, a route management method, and a route management system. [Background technology]

[0002] Conventionally, techniques for learning the optimal movement path of a moving object from a starting point to a destination point using reinforcement learning have been known (see Patent Document 1).

[0003] However, the mobile path management technology described in Patent Document 1 has a problem in that the reward design becomes complicated when the mobile learns a path from a starting point to a destination point via specified waypoints. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-173643 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Conventional technologies have made it difficult to manage the route of a moving object traveling to a destination point via designated waypoints using a simpler configuration.

[0006] The present invention was made to solve the above-mentioned problems and aims to manage the route of a moving object that travels to a destination point via designated waypoints with a simpler configuration. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the path management device according to the present invention comprises: a division unit configured to divide the path of a moving object from a starting point to a destination point via intermediate points into a plurality of sub-paths; a first learning unit configured to apply a reward function to the estimation result of calculating the path that the moving object should sequentially take from the position of the initial point constituting the sub-path to the position of the final point, for each of the plurality of sub-paths, and update it so as to maximize the reward for the moving object to reach the position of the final point, and to learn a strategy for the path that the moving object should sequentially take from its current position using a reinforcement learning model; a second learning unit configured to learn the relationship between the current position of the moving object and the strategy for the path that the moving object should sequentially take from its current position for each of the plurality of sub-paths, obtained by learning by the first learning unit, using a supervised learning model; and a storage unit configured to store the learned supervised learning model constructed by the second learning unit.

[0008] Furthermore, in the route management device according to the present invention, the waypoints include a plurality of waypoints whose order of passage is predetermined, and the plurality of partial routes may include the route from the starting point to the first waypoint the moving object passes through, the routes between the waypoints passed through sequentially, and the route from the last waypoint to the destination point.

[0009] Furthermore, the route management device according to the present invention may also include a setting unit configured to set the learned supervised learning model on the mobile body as control information that controls the route the mobile body takes from the starting point to the destination point via the intermediate points.

[0010] Furthermore, in the route management device according to the present invention, the moving space on which the moving body moves is defined as a unit space divided into a plurality of spaces, and the strategy for the path that the moving body should sequentially move from the current position may be a strategy for the path that the moving body should sequentially move from the position in the unit space corresponding to the current position to the unit space.

[0011] Furthermore, in the path management device according to the present invention, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer, the first learning unit is given the position in the unit space corresponding to the current position as input to the neural network model, performs calculations on the neural network model and outputs a first estimate of an action-value function that represents the expected value of the cumulative value of future rewards obtained when the moving body takes each action related to movement in a predetermined direction relative to the direction of movement as the next path to take from the position in the unit space corresponding to the current position, the first learning unit is further given the position in the unit space that the moving body next reaches as input to the neural network model, performs calculations on the neural network model and outputs a second estimate of the action-value function, and the first learning unit may learn the weight parameters of the neural network model such that the first estimate becomes the target value calculated from the second estimate.

[0012] To solve the above-mentioned problems, the path management method according to the present invention comprises: a division step of dividing the path of a moving object from a starting point to a destination point via intermediate points into a plurality of sub-paths; a first learning step of applying a reward function to the estimation result of calculating the path that the moving object should sequentially take from the position of the initial point constituting the sub-path to the position of the final point, for each of the plurality of sub-paths, and updating it so as to maximize the reward for the moving object to reach the position of the final point, and learning a strategy for the path that the moving object should sequentially take from its current position using a reinforcement learning model; a second learning step of learning the relationship between the current position of the moving object and the strategy for the path that the moving object should sequentially take from its current position for each of the plurality of sub-paths obtained by learning in the first learning step, using a supervised learning model; and a storage step of storing the learned supervised learning model constructed in the second learning step in a storage unit.

[0013] Furthermore, in the route management method according to the present invention, the waypoints include a plurality of waypoints whose order of passage is predetermined, and the plurality of sub-routes may include the route from the starting point to the first waypoint the moving object passes through, the routes between the waypoints passed through sequentially, and the route from the last waypoint to the destination point.

[0014] Furthermore, the route management method according to the present invention may further include a setting step in which the learned supervised learning model is set on the moving body as control information that controls the path the moving body takes from the starting point to the destination point via the intermediate points.

[0015] Furthermore, in the route management method according to the present invention, the moving space on which the moving body moves is defined as a unit space divided into a plurality of spaces, and the strategy for the path that the moving body should sequentially move from the current position may be a strategy for the path that the moving body should sequentially move from the position in the unit space corresponding to the current position in the unit space.

[0016] Furthermore, in the path management method according to the present invention, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the first learning step provides the position in a unit space corresponding to the current position as input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action-value function that represents the expected value of the cumulative value of future rewards obtained when the moving body takes each action related to movement in a predetermined direction relative to the direction of movement as the next path to take from the position in the unit space corresponding to the current position, and the first learning step further provides the position in the unit space that the moving body next reaches as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action-value function, and the first learning step may learn the weight parameters of the neural network model such that the first estimate becomes the target value calculated from the second estimate.

[0017] To solve the above-mentioned problems, the route management system according to the present invention is a route management system comprising the above-described route management device and the mobile body, wherein the mobile body comprises a second acquisition unit configured to acquire the learned supervised learning model constructed by the route management device, a third acquisition unit configured to acquire the current position of the mobile body, a calculation unit configured to provide the current position of the mobile body acquired by the third acquisition unit as an unknown input to the learned supervised learning model, perform calculations on the learned supervised learning model and output a strategy for the route to be taken sequentially from the current position of the mobile body, and a movement control unit configured to control the movement of the mobile body from the starting point to the destination point via the waypoints based on the strategy for the route to be taken sequentially from the current position of the mobile body output by the calculation unit. [Effects of the Invention]

[0018] According to the present invention, the relationship between the current position of the moving object and the strategy for the path the moving object should sequentially take from its current position for each of the multiple subpaths obtained through learning by the first learning unit is learned using a supervised learning model. Therefore, the path of a moving object moving to a destination point via specified waypoints can be managed with a simpler configuration. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 is a block diagram showing the configuration of a route management system equipped with a route management device according to an embodiment of the present invention. [Figure 2] Figure 2 is a diagram illustrating the learning process performed by the first learning unit of the route management device according to this embodiment. [Figure 3] Figure 3 is a block diagram showing the configuration of the first learning unit included in the route management device according to this embodiment. [Figure 4] Figure 4 is a diagram illustrating the learning process performed by the second learning unit of the route management device according to this embodiment. [Figure 5]Figure 5 is a block diagram showing the hardware configuration of the route management device according to this embodiment. [Figure 6] Figure 6 is a block diagram showing the configuration of the mobile unit included in the route management system according to this embodiment. [Figure 7] Figure 7 is a block diagram showing the hardware configuration of the mobile device included in the route management system according to this embodiment. [Figure 8] Figure 8 is a sequence diagram showing the operation of the route management system according to this embodiment. [Figure 9] Figure 9 is a flowchart showing the first learning process of the route management device according to this embodiment. [Figure 10] Figure 10 is a flowchart showing the first learning process of the route management device according to this embodiment. [Modes for carrying out the invention]

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to Figures 1 to 10.

[0021] [Configuration of the routing management system] First, with reference to Figure 1, an overview of a route management system comprising a route management device 1 and a mobile unit 2 according to an embodiment of the present invention will be described.

[0022] The route management system according to this embodiment comprises a route management device 1 and a mobile body 2, and the route management device 1 and the mobile body 2 are connected to each other so as to be able to communicate via a wireless communication network NW that conforms to a predetermined communication standard such as LTE / 4G, 5G, or 6G. The route management system manages the route for the mobile body 2 to move to a destination point via waypoints set in the mobile space A. As shown in Figure 1, the mobile space A on which the mobile body 2 moves is, for example, capable of communication using a 5G wireless communication method.

[0023] Mobile Entity 2 includes aircraft capable of autonomous flight such as drones and unmanned aerial vehicles, autonomous vehicles, mobile robots, and ships. Mobile Entity 2 starts from any current position in mobile space A, or a pre-set position, and moves from the starting point to the destination point via pre-set waypoints, and performs a predetermined task. Furthermore, Mobile Entity 2 can perform pre-set tasks at designated waypoints.

[0024] In the following explanation, we will use the case where the mobile body 2 is a drone as an example. The mobile body 2 controls autonomous flight using a flight controller that processes information from sensors 208 and the like (described later) to control the rotation speed of the motor 209 and the angle of the propeller 210. The mobile body 2 also acquires its own GPS position using a GPS receiver 207. The mobile body 2 is configured as an IoT terminal having an IP address, and each IP address can uniquely identify the mobile body 2. Furthermore, in this embodiment, the mobile body 2 controls autonomous flight to a destination point via designated waypoints based on control information stored in the auxiliary storage device 205. Details of the functional blocks and hardware configuration of the mobile body 2 will be described later. Note that there may be multiple mobile bodies 2, in which case each mobile body 2 has the same configuration.

[0025] As shown in Figure 1, the moving space A on which the moving object 2 moves is a three-dimensional matrix-like space composed of unit spaces divided into multiple spaces. Each unit space constituting the moving space A has the same volume. Furthermore, each unit space has a node ID, and each unit space is represented by a single position (x, y, z). The position information can be three-dimensional GPS position coordinates consisting of latitude, longitude, and altitude. For example, a representative value such as the center position of the unit space can be used as the position of that unit space.

[0026] Furthermore, as shown in Figure 1, the moving object 2 moves from a unit space position corresponding to the starting point S to the unit space position of the destination point G, using each unit space corresponding to the positions of the intermediate points WP1, WP2, ..., WPn (where n is a positive integer of 1 or more) as waypoints. In this embodiment, the object moves from the starting point S to the destination point G, passing through a series of intermediate points WP1, WP2, ..., WPn, in a predetermined order.

[0027] The path management system according to this embodiment divides the points between the starting point S, waypoints WP1, WP2, ..., WPn, and destination point G into sub-paths, and learns the optimal path for each sub-path using reinforcement learning. Furthermore, using the strategies for the mobile body 2's path for multiple sub-paths obtained through reinforcement learning as training data, the relationship between the mobile body 2's current position in unit space and the strategies for the path the mobile body 2 should sequentially take is learned using a supervised learning model. Furthermore, the learned supervised learning model is set on the mobile body 2 as control information to control the mobile body 2's path. Based on the set control information, the mobile body 2 takes its current position in unit space as an unknown input, performs calculations on the learned supervised learning model, and outputs strategies for the path it should sequentially take. Then, based on the output strategies for the path to sequentially take, it determines the path from the starting point to the destination point via the specified waypoints, and controls its movement to the destination point.

[0028] The mobile object 2, with control information set, changes its path according to the path determined based on the path policy obtained from calculations of a trained supervised learning model, as shown by the arrows at the starting point S in Figure 1, and moves in the direction it should move in each unit space. The path can include various paths, i.e., directions of movement. In Figure 1, the movement space A is explained in a two-dimensional plane, but the path of the mobile object 2 can be a three-dimensional path. With the control information set by the path management device 1 for the mobile object 2, the mobile object 2 can reach the unit space of the destination point G from the unit space of the starting point via specified waypoints.

[0029] Here, a path refers to movement from one position in a unit space to an adjacent position in another unit space. A route includes the entire route from the starting point to the destination point, passing through a specified number of intermediate unit spaces. Furthermore, the paths between points are specifically called sub-paths, and the entire route is constructed by combining these sub-paths.

[0030] [Functional blocks of the route management device] As shown in Figure 1, the route management device 1 comprises a first acquisition unit 10, a division unit 11, a first learning unit 12, a second learning unit 13, a first storage unit (storage unit) 14, a second storage unit 15, a third storage unit 16, and a setting unit 17. The route management device 1 manages the route of the moving object 2 to the destination point via intermediate points set in the moving space A.

[0031] The first acquisition unit 10 acquires the starting point, intermediate points, and destination point set in the moving space A. The first acquisition unit 10 also acquires the current position of the moving object 2. Specifically, the first acquisition unit 10 acquires the node ID of the unit space corresponding to the starting point, the node ID of the unit space corresponding to the intermediate point, and the node ID of the unit space corresponding to the destination point, which are stored in the third storage unit 16 described later. The first acquisition unit 10 also acquires the unit space in which the moving object 2 is currently located as the current position. The current position is the position of the unit space in which the moving object 2 exists at each set time t.

[0032] The first acquisition unit 10 acquires the position in a unit space corresponding to the GPS position of the mobile body 2 at each time t as the current position of the mobile body 2. In some cases, the unit space may contain multiple position coordinates. In such cases, the first acquisition unit 10 can acquire the position in the unit space that is closest in distance to the GPS position received by the GPS receiver 207 of the mobile body 2 as the current position of the mobile body 2.

[0033] The division unit 11 divides the route of the mobile body 2 from its starting point to its destination point via intermediate points into multiple sub-routes. More specifically, the division unit 11 divides the route of the mobile body 2 from its starting point to the first intermediate point it passes through, the routes between the intermediate points it passes through sequentially, and the route from the last intermediate point it passes through to its destination point. In the example in Figure 1, the division unit 11 divides the route into a sub-route from the starting point S to the first intermediate point WP1, a sub-route from intermediate point WP1 to the next intermediate point WP2, a sub-route from intermediate point WP2 to the next intermediate point WP3, and so on, and further divides it into a sub-route from the last intermediate point WPn to the destination point G. Each sub-route is assigned an ID, which allows the initial and final points of each sub-route to be identified. For example, the initial point of the sub-route identified by ID 1 is the starting point S, and the final point is intermediate point WP1.

[0034] The first learning unit 12 calculates the estimated path that the mobile object 2 should sequentially take from the initial position to the final position of each of the multiple subpaths, applies a reward function to the estimated path, and updates it so that the reward for the mobile object 2 to reach the final position is maximized. The first learning unit 12 learns a strategy for the path that the mobile object 2 should sequentially take from its current position using a reinforcement learning model.

[0035] In this embodiment, the strategy for the path that the moving body 2 should sequentially take from each position in each unit space is an action a related to movement in a predetermined n (where n is an integer of 2 or more) directions relative to the direction of travel. n An example of adopting this approach is given. Furthermore, the direction of movement is based on the position in the unit space where the moving object 2 was immediately before.

[0036] The first learning unit 12 uses a neural network model as a reinforcement learning model, which includes an input layer s, a hidden layer h, and an output layer q as shown in Figure 2. Furthermore, the neural network model is defined as the state s, which is the position of the moving object 2. t It receives all action value functions Q(s t a1), Q(s t a2), Q(s t, a3), ···, Q(s t , a n-1 ), Q(s t , a n ) outputs the Deep Q-Network (DQN), which is a neural network.

[0037] More specifically, the first learning unit 12 gives, as an input to the neural network model, the position of the current unit space indicating the position of the current moving body 2, performs the operation of the neural network model, and as the route that the moving body 2 should proceed to next from the current position of the unit space, the action a related to each movement in n directions n outputs the first estimated value Q1 of the action value function representing the expected value of the cumulative value of future rewards obtained when taking.

[0038] The reward is given by the reward function r = r(s, a, s') of the state s indicating the current position of the moving body 2, the action a of the moving body 2 moving in a predetermined direction n , and the next position of the moving body 2, that is, the next state s'. In the present embodiment, the reward function includes, as a variable, the degree of reach to the position of the unit space related to the final point of the moving body 2. In addition, it can include, as a variable, the degree of reach to the position of the unit space corresponding to a space with obstacles such as high-rise buildings and towers. For example, when the moving body 2 approaches the final point or reaches the final point at the shortest distance by the action related to the movement in a predetermined direction of the moving body 2, the reward, which is a scalar quantity, is set as a larger value.

[0039] On the other hand, when the moving body 2 moves away from the final point or reaches a unit space where an obstacle exists, it can be designed to give a negative reward value (for example, r = -1). By setting the reward of the unit space where an obstacle exists as a negative value in this way, the moving body 2 can avoid these points and reach the destination.

[0040] Furthermore, the first learning unit 12 receives the next spatial position reached by the moving object 2 as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate Q2 of the action-value function. The first learning unit 12 learns the weight parameters of the neural network model so that the first estimate Q1 becomes the target value calculated from the second estimate Q2.

[0041] If we denote the weight parameters of the neural network model as θ and the action-value function as Q(s,a;θ), the learning minimization loss function is given by the following equation (1). L(θ) = 1 / 2{r + γmax} a’ Q(s',a';θ)-Q(s,a;θ)} 2 ...(1)

[0042] In equation (1) above, r is the reward (immediate reward) and γ is the discount rate. Q(s,a;θ) corresponds to the first estimate Q1, and Q(s',a';θ) corresponds to the value of the action in state s' after one step, i.e., the second estimate Q2. The target value is r + γmax a’ It can be represented by Q(s',a';θ).

[0043] The first learning unit 12 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by equation (1) above.

[0044] More specifically, the first learning unit 12 can employ a Fixed Target Q-Network using two neural networks, main QN121 and target QN123, as shown in Figure 3. Main QN121 selects the optimal action and updates the action-value function Q. Meanwhile, target QN123 estimates and evaluates the value of the action a' to be taken in the next state s' resulting from the action. Main QN121 and target QN123 have neural networks with the same layer structure, but the parameter of main QN121 is "θ" and the parameter of target QN123 is "θ". - It is given by ".

[0045] The main QN121 receives the current position of the mobile object 2 as state s from environment 120. Environment 120 is a system of movement space in which the mobile object 2 is located. Under this environment 120, the mobile object 2 moves to another unit space by taking action a related to movement in a predetermined direction, transitions to the next state s', and simultaneously receives a reward r from environment 120.

[0046] The first learning unit 12 inputs the state s related to the current position of the moving object 2 to the main QN121 and calculates the action-value function Q(s,a;θ). The first learning unit 12 calculates action a using, for example, the ε-greedy method, or the optimal action argmax at the present time. a We find Q(s,a;θ). In environment 120, the action of moving object 2 regarding the optimal path at the present time is argmax a Perform Q(s,a;θ). Environment 120 is where mobile object 2 takes action argmax. a The result of performing Q(s,a;θ) is observed as the next state s' in the unit space at the destination, and the reward r is output. Experience data 124 stores the experience (s,a,r,s') output from environment 120.

[0047] The first learning unit 12 calculates the loss function L in the DQN loss calculation 122 and updates the weights of the main QN 121 using the gradient of the loss function L.

[0048] The first learning unit 12 periodically copies the weights of the main QN121 to the target QN123 and synchronizes them. The synchronization of the target QN123 is performed at a lower frequency than the update frequency of the weights of the main QN121. The first learning unit 12 extracts experience from the experience data 124, inputs the past state into the target QN123, and estimates the max value. a’ Q(s',a';θ - The first learning unit 12 outputs the estimated value max output by target QN123. a’ Q(s',a';θ - ) Target value r+γmax a’ Q(s',a';θ -Using this method, the weights of the main QN121 are trained using DQN loss calculation 122.

[0049] The learned reinforcement learning model, which represents the sequential path to take from the initial position to the final position of each of the multiple subpaths, obtained through learning by the first learning unit 12, is stored in the first memory unit 14. Furthermore, the learned reinforcement learning model constructed for each of the multiple subpaths is used as training data in learning by the second learning unit 13.

[0050] The second learning unit 13 learns, using a supervised learning model, the relationship between the current position of the mobile object 2 and the strategies for the paths that the mobile object 2 should sequentially take from its current position for each of the multiple subpaths obtained through learning by the first learning unit 12.

[0051] Figure 4 shows the structure of a neural network model adopted as an example of a supervised learning model used by the second learning unit 13. The neural network model comprises an input layer x, a hidden layer h, and an output layer y. The second learning unit 13 provides the current unit space position of the mobile object 2, i.e., the unit space position corresponding to the GPS position of the mobile object 2 at each time t, to the input layer of the neural network model, applies an activation function to the weighted sum of the inputs, and passes the output determined by thresholding to the output layer. Each output node of the output layer outputs the model's predicted output corresponding to the n action-value functions Q of each subpath.

[0052] The second learning unit 13 learns the parameters of the neural network model so that the predicted path that the moving object 2 should take sequentially from its current position, which is the predicted value from the neural network model for the current position of the moving object 2, becomes the value of the optimized path that the moving object 2 should take sequentially from its current position, which has been reinforced by the first learning unit 12 for each subpath.

[0053]

number

[0054] In equation (2) above, y1, y2, ..., y n The predicted output values ​​for each output node are shown. Also, Y1, Y2, ..., Y n Here, the training data is the n optimized action-value functions Q of a given subpath for the current position, obtained by reinforcement learning by the first learning unit 12. Furthermore, as mentioned above, the first learning unit 12 performs reinforcement learning on each of the N subpaths, and the n optimized action-value functions Q(s) are obtained for each subpath. t a1), Q(s t a2), Q(s t ,a3),...,Q(s t ,a n-1 ), Q(s t ,a n ) has.

[0055] The objective function E in equation (2) above is further the optimized action-value functions Q1~Q of the N subpaths. N This can be configured to comprehensively learn the following. In this case, if Y1 is the training data for the first subpath, then the optimized n action-value functions Q1(s t ,a1), Q1(s t ,a2), Q1(s t ,a3),...,Q1(s t ,a n-1 ), Q1(s t ,a n ), when Y2 is the training data for the second subpath, the optimized n action-value functions Q2(s t ,a1), Q2(s t ,a2), Q2(s t ,a3),...,Q2(s t ,a n-1 ), Q2(s t ,a n ), ..., training data Y for the Nth subpath N The optimized n action-value functions Q that make up the N (s t a1), Q N(s t a2), Q N (s t a3), ..., Q N (s t ,a n-1 ), Q N (s t ,a n ) is used as training data.

[0056] In this case, equation (2) above can be further expressed by the following equation (3).

number

[0057] In the objective function E of equation (3) above, y 1_1 ,y 1_2 ,···,y 1_n ,···,y 2_1 ,y 2_2 ,···,y 2_n ,···,y N_1 ,y N_2 ,···,y N_n This is the predicted output value of n (n action-value functions Q) × N (number of subpaths N) output nodes. 1_1 ~Y 1_n This corresponds to the optimized action-value function Q1 of the first subpath, and Y 2_1 ~Y 2_n This corresponds to the optimized action-value function Q2 of the second subpath, and Y N_1 ~Y N_n Q is the optimized action-value function of the Nth subpath. N This is the corresponding training data.

[0058] Thus, all of the optimized action-value functions Q obtained from the trained reinforcement learning model, from the initial point to the final point for each of the multiple subpaths, are used as training data. In the example neural network model configuration in Figure 4, the training data is shown to the right of each output node that outputs the model's predicted output value. Starting from the top output node, the training data Y 1_1 ~Y 1_nn action-value functions Q1 related to the first partial path are shown, but for the teacher data Y2 ··· Y N for each, the n values corresponding to the respective action-value functions Q2, ···, Q N are illustrated with each n values omitted.

[0059] The value of the objective function E in the above formula (3) is the output value y for the position in the unit space corresponding to the GPS position of the moving body 2 at time t, which is the above input value x of the supervised learning model 1_1 , y 1_2 , ···, y 1_n , ···, y 2_1 , y 2_2 , ···, y 2_n , ···, y N_1 , y N_2 , ···, y N_n is the target output Y of the teacher data 1_1 ~Y 1_n , Y 2_1 ~Y 2_n , ···, Y N_1 ~Y N_n becomes 0 when they match. The second learning unit 13 adjusts the weight parameters of the neural network related to the supervised learning model so that the objective function E is minimized, that is, becomes 0. The second learning unit 13 can optimize the objective function E using the error backpropagation method or the like.

[0060] The first storage unit 14 stores the learned reinforcement learning model constructed by the reinforcement learning by the first learning unit 12. Since the learned reinforcement learning model is constructed for each of the plurality of partial paths, N learned reinforcement learning models corresponding to the N partial paths are stored.

[0061] The second storage unit 15 stores the learned supervised learning model constructed by the supervised learning by the second learning unit 13.

[0062] The third storage unit 16 stores the starting point, the preset via points, and the destination point of the moving body 2. Further, the third storage unit 16 stores the position information of the unit space constituting the moving space A and the identification information of the moving body 2.

[0063] The setting unit 17 sets the trained supervised learning model as control information for the mobile body 2 to travel from the starting point to the destination point via intermediate points. For example, the setting unit 17 can transmit the control information to the mobile body 2 via the network NW.

[0064] [Hardware configuration of the routing device] Next, an example of a hardware configuration for realizing the route management device 1 having the functions described above will be explained using Figure 5.

[0065] As shown in Figure 5, the route management device 1 can be implemented, for example, by a computer equipped with a processor 102, main memory 103, communication interface 104, auxiliary storage 105, and input / output I / O 106 connected via a bus 101, and a program that controls these hardware resources. Furthermore, the route management device 1 may include a display device 107 connected via the bus 101.

[0066] Processor 102 is implemented using CPUs, GPUs, FPGAs, ASICs, etc.

[0067] The main memory 103 contains pre-stored programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory 103 work together to realize the various functions of the route management device 1, such as the first acquisition unit 10, the division unit 11, the first learning unit 12, the second learning unit 13, and the setting unit 17 shown in Figure 1.

[0068] The communication interface 104 is an interface circuit for networking the route management device 1 with various external electronic devices.

[0069] The auxiliary storage device 105 consists of a read / write storage medium and a drive device for reading and writing various information such as programs and data to the storage medium. The auxiliary storage device 105 can use semiconductor memory such as a hard disk or flash memory as the storage medium.

[0070] The auxiliary storage device 105 has a program storage area for storing the route management program executed by the route management device 1. It also has a program storage area for storing the reinforcement learning program executed by the route management device 1. Furthermore, the auxiliary storage device 105 has an area for storing the supervised learning program. The auxiliary storage device 105 realizes the first storage unit 14, the second storage unit 15, and the third storage unit 16 described in Figure 1. Specifically, the auxiliary storage device 105 has an area for storing the waypoints and destination points of the mobile body 2. Furthermore, the auxiliary storage device 105 has an area for storing the position coordinates of the moving space and the position coordinates of the unit space. Furthermore, the auxiliary storage device 105 has an area for storing identification information such as the IP address of the mobile body 2. Furthermore, it may have, for example, a backup area for backing up the above-mentioned data and programs.

[0071] The I / O106 is an input / output device that accepts signals from external devices and outputs signals to external devices.

[0072] The display device 107 is composed of an organic EL display or a liquid crystal display. The display device 107 can display a map of the moving space, the current position of the moving object 2, its progress, partial route and overall route, and location information of the destination point.

[0073] [Functional blocks for mobile objects] Next, the functional blocks of the mobile unit 2 will be explained with reference to Figure 6. Figure 6 is a block diagram showing the configuration of the mobile unit 2.

[0074] The mobile unit 2 comprises a fourth storage unit 20, a second acquisition unit 21, a fifth storage unit 22, a third acquisition unit 23, a calculation unit 24, a determination unit 25, and a movement control unit 26. Based on the control information set by the route management device 1, the mobile unit 2 determines the next course it should take and controls its flight to the destination point via designated waypoints.

[0075] The fourth memory unit 20 stores control information set by the setting unit 17 of the route management device 1. The control information is a pre-trained supervised learning model that the second learning unit 13 of the route management device 1 has learned through supervised learning, which determines the strategy for the route to be taken sequentially from the current position, from the starting point to the destination point, via designated waypoints.

[0076] The second acquisition unit 21 acquires control information set by the route management device 1. Specifically, the second acquisition unit 21 loads the control information stored in the fourth storage unit 20.

[0077] The fifth storage unit 22 stores map data including the position coordinates of the travel space, and information associating the position coordinates of the unit spaces constituting the travel space with the node IDs of the unit spaces. The fifth storage unit 22 also stores the position information of designated waypoints and destination points. Furthermore, the fifth storage unit 22 stores the IDs of multiple sub-routes.

[0078] The third acquisition unit 23 acquires the current position of the device. More specifically, the third acquisition unit 23 acquires the position of the device in a unit space at each time t, based on the GPS position of the device. The third acquisition unit 23 refers to the fifth storage unit 22 and acquires the position in the unit space corresponding to the current GPS position received by the GPS receiver 207 as the current position of the device.

[0079] The calculation unit 24 provides the current position of the device acquired by the third acquisition unit 23 as an unknown input to a pre-trained supervised learning model, performs calculations on the pre-trained supervised learning model, and outputs a strategy for the path the device should sequentially take from its current position.

[0080] The decision-making unit 25 determines the next path to be traveled from the current position of the own device based on the path strategy of the path that the own device should sequentially travel from the current position, which is output by the calculation unit 24. More specifically, based on the path strategy output by the calculation unit 24, the position of the current unit space is set as the state s t and, for each state s t the path with the action a that maximizes the value of the action-value function Q is selected to determine the path to be sequentially traveled.

[0081] The movement control unit 26 controls the movement of the own device based on the path strategy of the path that the own device should sequentially travel from the current position, which is output by the calculation unit 24. Specifically, the movement control unit 26 controls the movement of the own device based on the path to be traveled next, which is output by the calculation unit 24 and determined by the decision-making unit 25. The movement control unit 26 can calculate a control command for the path to be traveled next from the current position and transmit a control command value to the motor 209. In this way, the mobile body 2 can move in a group from different starting points to a common destination point by selecting the action a that maximizes the value of the action-value function Q for each state s t

[0082] [Hardware Configuration of Mobile Body] Next, an example of the hardware configuration for realizing the mobile body 2 having the above-described functions will be described with reference to FIG. 7.

[0083] As shown in FIG. 7, the mobile body 2 can be realized by, for example, a microcomputer including a processor 202, a main storage device 203, a communication interface 204, an auxiliary storage device 205, and an input / output I / O 206 connected via a bus 201, and a program for controlling these hardware resources, a GPS receiver 207, a sensor 208, a motor 209, a propeller 210, and a battery 211. A flight controller for controlling the autonomous flight of the mobile body 2 is realized by a computer such as a microcomputer and a program. The mobile body 2 can mount these components on a frame having a predetermined shape.

[0084] The main memory 203 contains pre-stored programs for the processor 202 to perform movement control and calculations. The processor 202 and the main memory 203 work together to realize the various functions of the mobile body 2, such as the second acquisition unit 21, calculation unit 24, determination unit 25, and movement control unit 26, as shown in Figure 6.

[0085] The communication interface 204 is an interface circuit for network connection between the mobile device 2 and the route management device 1.

[0086] The auxiliary storage device 205 consists of a read / write storage medium and a drive device for reading and writing various information such as programs and data to the storage medium. The auxiliary storage device 205 can use semiconductor memory such as a hard disk or flash memory as the storage medium.

[0087] The auxiliary storage device 205 has a program storage area for storing the movement control program executed by the mobile device 2. The auxiliary storage device 205 also has an area for storing calculation programs for performing calculations on a trained supervised learning model. The auxiliary storage device 205 enables the realization of the fourth storage unit 20 and the fifth storage unit 22 described in Figure 6. Furthermore, the auxiliary storage device 205 has an area for storing identification information such as the IP address of the mobile device 2. The auxiliary storage device 205 also has an area for storing an application that notifies the route management device 1 of the GPS location. In addition, it may have, for example, a backup area for backing up the aforementioned data and programs.

[0088] The I / O206 is an input / output device that accepts signals from external devices and outputs signals to external devices.

[0089] The GPS receiver 207 has a built-in antenna for receiving GPS signals. The GPS receiver 207 enables the third acquisition unit 23 shown in Figure 6.

[0090] Sensor 208 consists of various sensors such as an altitude sensor, attitude sensor, camera, LiDAR, and RADAR. In addition to the GPS receiver 207, the altitude sensor enables the third acquisition unit 23 shown in Figure 6. Furthermore, the flight controller controls the flight of the mobile body 2 based on the various sensor data measured by sensor 208.

[0091] The motor 209 rotates due to a rotational drive, causing the propeller 210, which is attached to the rotation axis of the motor 209, to rotate.

[0092] Battery 211 is a lithium-ion battery or the like, and supplies power to the mobile unit 2.

[0093] Furthermore, mobile device 2 may be equipped with a SIM card and possess the SIM card's IMSI (International Mobile Subscriber Identity).

[0094] [Route management system operation] Next, the operation of the route management system, which includes the route management device 1 and mobile body 2 having the above-described configuration, will be explained with reference to the sequence in Figure 8. The mobile body 2 is positioned at a predetermined unit space location within the travel space, and this location is the starting point.

[0095] The first acquisition unit 10 of the route management device 1 acquires information on the starting point set in the travel space A, a plurality of waypoints whose order of passage is determined, and the destination point (step S1). Next, the division unit 11 divides the route from the starting point to the destination point via the waypoints acquired in step S1 into a plurality of sub-routes (step S2). More specifically, in step S2, as explained in Figure 1, the division unit 11 divides the route into a sub-route from the starting point S to the first waypoint WP1, a plurality of sub-routes between waypoints WP1 to WPn, and a sub-route from the last waypoint WPn to the destination point G. Each of the sub-routes has a different starting point and ending point. The entire route is formed by combining the plurality of sub-routes.

[0096] Next, the first acquisition unit 10 acquires the current location of the mobile object 2 as the location of the unit space in which the mobile object 2 is currently located (step S3). For example, the first acquisition unit 10 can acquire the current location of the mobile object 2 by having the mobile object 2 notify the current GPS location via push notification to the IP address of the mobile object 2.

[0097] The first acquisition unit 10 acquires the position of the mobile body 2 in a unit space at each time t. Here, since the starting point is predetermined, the position of the mobile body 2 in a unit space at the first time t is the position in the unit space corresponding to the starting point. The first acquisition unit 10 acquires the position in the unit space corresponding to the current GPS position received by the mobile body 2's GPS receiver 207 as the mobile body 2's current position.

[0098] Next, the first learning unit 12 performs the first learning process (step S4). In the first learning process, for each of the multiple subpaths, the first learning unit 12 calculates the estimated path that the mobile body 2 should sequentially take from the initial position of the subpath to the final position of the subpath, applies a reward function to the result, and updates it so that the reward for the mobile body 2 to reach the final position is maximized. The first learning unit 12 learns a strategy for the path that the mobile body 2 should sequentially take from its current position using a reinforcement learning model. In step S3, the learning process is performed for each of the multiple subpaths. Details of the first learning process will be described later.

[0099] Subsequently, the first memory unit 14 stores the reinforcement learning model obtained in step S4 (step S5). Next, the second learning unit 13 learns the relationship between the current position of the moving object 2 and the strategies for the paths that the moving object 2 should sequentially take from its current position, obtained in the first learning process for each of the multiple subpaths in step S4, using a supervised learning model (second learning process) (step S6).

[0100] Specifically, the second learning unit 13 repeatedly adjusts and updates parameters such as weights and thresholds to determine the values ​​of these parameters, such that the error between the predicted output value of the sequential path to be taken (when the position in a unit space corresponding to the current GPS position of the moving object 2, i.e., the current state) is given to the supervised learning model as input, and the training data, minimizes the objective function E in equation (3) above. In step S6, the second learning unit 13 can determine the parameters that minimize the objective function E by backpropagation or the like.

[0101] The training data used in step S6 consists of strategies for the paths to be taken sequentially from the current position in the unit space, obtained by the trained reinforcement learning model constructed for each of the multiple subpaths in the first learning process in step S4.

[0102] Next, the second memory unit 15 stores the trained supervised learning model constructed in step S6 (step S7). Then, the setting unit 17 sets the trained supervised learning model as control information for the mobile device 2 (step S8). In step S8, the setting unit 17 can transmit the trained supervised learning model to the mobile device 2 via the network NW.

[0103] Subsequently, the second acquisition unit 21 of the mobile unit 2 acquires the trained supervised learning model constructed by the route management device 1 (step S9). The second acquisition unit 21 reads out the control information transmitted from the route management device 1 and stored in the fourth storage unit 20, i.e., the trained supervised learning model.

[0104] Next, the third acquisition unit 23 acquires the current position of the device in unit space as the current position (step S10). Specifically, it can acquire the position in unit space corresponding to the GPS position received by the GPS receiver 207 as the current position of the device. Next, the calculation unit 24 uses the control information acquired in step S9, provides the current position in unit space of the device acquired in step S10 as an unknown input, performs calculations on the trained supervised learning model, and outputs a strategy for the path to be taken sequentially from the current position in unit space (step S11).

[0105] For example, in the movement space A of Figure 1, if the position of the starting point S of the moving object 2 is input as the current position at time t=1 to a trained supervised learning model, the n action-value functions Q for the next steps to take from the starting point S at time t=1 will be output. More specifically, if the current position of the moving object 2 is given as an unknown input, the values ​​of the n path policies for a specific subpath corresponding to the current position of the moving object 2 will be output as the calculation result of the trained supervised learning model, from among the optimized path policies of the N subpaths. The values ​​of the n path policies for the other subpaths that do not correspond to the current position of the moving object 2 will be 0.

[0106] Next, the decision unit 25 determines the path to be taken sequentially by selecting the path that takes the action a with the maximum value of the n action-value functions Q output in step S11 (step S12). Next, the movement control unit 26 controls the movement of the device based on the path to be taken next, which was determined in step S12 (step S13). More specifically, the movement control unit 26 can calculate a control command for the next path to be taken from the current position and transmit the control command value to the motor 209.

[0107] The mobile object 2 repeats the processes from steps S10 to S13 until it reaches the unit space of a designated waypoint from the starting point (step S14: NO). If it reaches the unit space of a designated waypoint (step S14: YES), it repeats the processes from steps S10 to S14 until it reaches the destination point (step S15: NO). After that, when the mobile object 2 reaches the position of the unit space of the destination point (step S15: YES), the process ends. In this way, the mobile object 2 can move from the starting point to the destination point via a designated waypoint and perform a predetermined task by executing the processes from steps S9 to S15 using a trained supervised learning model which is the control information.

[0108] Next, the first learning process by the path management device 1 (step S4 in Figure 8) will be explained using the flowcharts in Figures 9 and 10. First, step S3, as explained in Figure 8, is executed. Then, learning processing is performed for each of the multiple subpaths. First, the first learning unit 12 provides the current state of the moving object 2, which is the position in the unit space where the moving object 2 is currently located, obtained in step S3, as input to the neural network model. The neural network model performs calculations and outputs a first estimated value Q1 of the action-value function, which represents the expected value of the cumulative value of future rewards obtained when the moving object 2 takes each action related to movement in a predetermined direction relative to the direction of travel, as the next path the moving object 2 should take from its current position in the unit space (step S20).

[0109] Next, the first acquisition unit 10 acquires the position of the moving object 2 in unit space at the next time t as the next state s' (step S21). The next position in unit space reached by the moving object 2 is determined based on the GPS position of the moving object 2 acquired by the first acquisition unit 10 at each time step. Furthermore, the first learning unit 12 provides the position in unit space reached by the moving object 2, acquired in step S21, as input to the neural network model, performs calculations on the neural network model, and outputs the second estimated value Q2 of the action-value function (step S22).

[0110] Next, the first learning unit 12 calculates the target value from the second estimated value Q2 (step S23). Subsequently, the first learning unit 12 learns the weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S24). Specifically, the first learning unit 12 updates the weight parameters of the neural network model to minimize the loss function in equation (1) above.

[0111] Subsequently, the process from step S3 to step S24 is repeated until a reinforcement learning model has been trained for all of the multiple subpaths (step S25: NO). After that, if training has been performed for all of the multiple subpaths (step S25: YES), the first memory unit 14 stores the trained reinforcement learning model obtained in step S24 (step S4).

[0112] Next, referring to Figure 10, we will explain the first learning process performed by the first learning unit 12 when a Fixed Target Q-Network is adopted, which uses two neural networks: main QN121 and target QN123.

[0113] The process in step S3 is the same as the steps of the first learning process described in Figure 9. Subsequently, the first learning unit 12 provides the main QN 121 with the position in the unit space where the moving object 2 is currently located, obtained in step S3, as input, performs calculations on the neural network model, outputs the action-value function Q, and calculates the next path a to take (step S120).

[0114] Next, the first learning unit 12 returns the action of the moving body 2 along the path a determined in step S120 to the environment 120, and obtains the next state s' of the moving body 2, which is the position in the unit space where the moving body 2 has moved and the reward r (step S121).

[0115] The first learning unit 12 saves the experience (s, a, r, a') obtained in step S121 to the experience data 124 (step S122). Next, in the DQN loss calculation 122, the first learning unit 12 calculates the loss function L and updates the weights of the main QN 121 using the gradient of the loss function L (step S123). The first learning unit 12 repeats the process from step S120 to step S123 a set number of times.

[0116] Subsequently, the first learning unit 12 periodically copies the weights of the main QN121 to the target QN123 and synchronizes them (step S124). The synchronization of the target QN123 is performed at a lower frequency than the update frequency of the weights of the main QN121. Next, the first learning unit 12 extracts experience from the experience data 124, inputs the past state into the target QN123, and estimates the max a’ Q(s',a';θ - Output ) (step S126).

[0117] Next, the first learning unit 12 processes the estimated value max output by the target QN123. a’ Q(s',a';θ - ) Target value r+γmax a’ Q(s',a';θ - The first learning unit 12 calculates the target value (step S127). Next, the first learning unit 12 calculates the loss function L in the DQN loss calculation 122 using the target value calculated in step S127 (step S128). Next, the first learning unit 12 learns the weights of the main QN 121 to minimize the loss given by the loss function L (step S129). After that, the first memory unit 14 stores each of the learned reinforcement learning models for all of the multiple subpaths (step S4).

[0118] As described above, the route management device 1 according to this embodiment divides the route from the starting point to the destination point via a designated waypoint into multiple sub-routes, learns the optimal route strategy for the mobile body 2 for each sub-routes using reinforcement learning, and uses the route strategies for the multiple sub-routes obtained through reinforcement learning as training data to learn the relationship between the current position of the mobile body 2 and the route strategies to be followed sequentially using a supervised learning model. Furthermore, the learned supervised learning model is set for the mobile body 2 as control information to control the route. Therefore, the route of a mobile body moving to a destination point via a designated waypoint can be managed with a simpler configuration.

[0119] Furthermore, according to the path management device of this embodiment, the optimal course strategy for each of the multiple sub-paths obtained by dividing the path from the starting point to the destination point via designated waypoints is learned through reinforcement learning. Therefore, even when multiple waypoints are specified, the complexity of the reward design can be suppressed.

[0120] Furthermore, according to the route management system of this embodiment, by setting control information on the mobile body 2, it is possible to realize a mobile body 2 that can move along the optimal route to the destination point via specified waypoints, thus simplifying the configuration of the mobile body 2.

[0121] In the embodiment described, the reinforcement learning model used by the first learning unit 12 is exemplified as a DQN related to a Fixed Target Q-Network composed of a multilayer neural network. However, other reinforcement learning models such as CNNs and multilayer perceptrons can be used. In addition to the DQN exemplified as a reinforcement learning model, Double DQN, Dueling DQN, Actor-Critic (AC) method, Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Q-learning, etc., can also be used.

[0122] Furthermore, in the embodiment described, the supervised learning model used by the second learning unit 13 was exemplified as a multilayer neural network. However, the supervised learning model can also be a multilayer perceptron, a decision tree-based model such as a random forest, or a support vector machine.

[0123] The embodiments of the route management device, route management method, and route management system of the present invention have been described above. However, the present invention is not limited to the embodiments described above, and various modifications that a person skilled in the art can envision are possible within the scope of the invention described in the claims. [Explanation of Symbols]

[0124] 1...Route management device, 10...First acquisition unit, 11...Division unit, 12...First learning unit, 13...Second learning unit, 14...First storage unit, 15...Second storage unit, 16...Third storage unit, 17...Setting unit, 2...Mobile unit, 20...Fourth storage unit, 21...Second acquisition unit, 22...Fifth storage unit, 23...Third acquisition unit, 24...Calculation unit, 25...Decision unit, 26...Movement control unit, 101, 201...Bus, 102, 202...Processor, 103, 203...Main memory, 104, 204...Communication interface, 105, 205...Auxiliary storage, 106, 206...Input / output I / O, 107...Display device, 207...GPS receiver, 120...Environment, 121...Main QN, 122...DQN loss calculation, 123...Target QN, 124...Experience data, NW...Network.

Claims

1. A division unit configured to divide the path of a moving object from a starting point to a destination point via intermediate points into multiple sub-paths, A first learning unit is configured to apply a reward function to the estimated path that the moving object should sequentially take from the initial position to the final position of each of the aforementioned subpaths, and to update the result so as to maximize the reward for the moving object to reach the final position, and to learn a strategy for the path that the moving object should sequentially take from its current position using a reinforcement learning model. A second learning unit is configured to learn, using a supervised learning model, the relationship between the current position of the moving object and the strategies for the paths the moving object should sequentially take from its current position for the plurality of partial paths, which are obtained through learning by the first learning unit. A storage unit configured to store the trained supervised learning model constructed by the second learning unit, A route management device equipped with the following features.

2. In the route management device according to claim 1, The aforementioned waypoints include multiple waypoints whose order of passage is predetermined. The aforementioned multiple sub-routes include a route from the starting point to the first intermediate point the moving object passes through, a route between sequential intermediate points, and a route from the last intermediate point to the destination point. A route management device characterized by the following features.

3. In the route management device according to claim 1, Furthermore, the system includes a setting unit configured to set the previously learned supervised learning model to the mobile body as control information that controls the path the mobile body takes to reach the destination point via the intermediate points from the starting point. A route management device characterized by the following features.

4. In the route management device according to claim 1, The moving space in which the moving object moves is defined by a unit space that is divided into multiple spaces, The strategy for the path the moving body should sequentially take from its current position is the strategy for the path the moving body should sequentially take in the unit space, starting from the position in the unit space corresponding to its current position. A route management device characterized by the following features.

5. In the route management device according to claim 4, The aforementioned reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer. The first learning unit receives the position in the unit space corresponding to the current position as input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action-value function that represents the expected value of the cumulative value of future rewards obtained when the moving object takes each action related to movement in a predetermined direction relative to the direction of movement, as the next path to take from the position in the unit space corresponding to the current position. The first learning unit further provides the position in the unit space that the moving object next reaches as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action-value function. The first learning unit learns the weight parameters of the neural network model such that the first estimate becomes the target value calculated from the second estimate. A route management device characterized by the following features.

6. A division step that divides the path of a moving object from the starting point to the destination point via intermediate points into multiple sub-paths, For each of the aforementioned subpaths, a reward function is applied to the estimated path that the moving body should sequentially take from the initial position of the subpath to the final position, and the reward function is updated to maximize the reward for the moving body to reach the final position. A first learning step involves learning a strategy for the path that the moving body should sequentially take from its current position using a reinforcement learning model. A second learning step involves learning, using a supervised learning model, the relationship between the current position of the moving object and the strategies for the paths the moving object should sequentially take from its current position for the plurality of subpaths, which were obtained through learning in the first learning step. A storage step in which the trained supervised learning model constructed in the second learning step is stored in the memory unit. A route management method comprising the following features.

7. In the route management method described in claim 6, The aforementioned waypoints include multiple waypoints whose order of passage is predetermined. The aforementioned multiple sub-routes include a route from the starting point to the first intermediate point the moving object passes through, a route between sequential intermediate points, and a route from the last intermediate point to the destination point. A route management method characterized by the following.

8. In the route management method described in claim 6, Furthermore, the system includes a setting step in which the previously learned supervised learning model is set on the mobile body as control information that controls the path the mobile body takes to reach the destination point via the intermediate points from the starting point. A route management method characterized by the following.

9. In the route management method described in claim 6, The moving space in which the moving object moves is defined by a unit space that is divided into multiple spaces, The strategy for the path the moving body should sequentially take from its current position is the strategy for the path the moving body should sequentially take in the unit space, starting from the position in the unit space corresponding to its current position. A route management method characterized by the following.

10. In the route management method described in claim 9, The aforementioned reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer. The first learning step provides the unit space position corresponding to the current position as input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action-value function that represents the expected value of the cumulative value of future rewards obtained when the moving object takes each action related to movement in a predetermined direction relative to the direction of movement, as the next path to take from the unit space position corresponding to the current position. The first learning step further provides the next spatial position reached by the moving object as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action-value function. The first learning step involves learning the weight parameters of the neural network model such that the first estimate becomes the target value calculated from the second estimate. A route management method characterized by the following.

11. A route management device according to any one of claims 1 to 5, The moving body and A route management system comprising, The aforementioned moving body is A second acquisition unit configured to acquire the trained supervised learning model constructed by the route management device, A third acquisition unit configured to acquire the current position of the device, A calculation unit is configured to provide the current position of the device acquired by the third acquisition unit as an unknown input to the trained supervised learning model, perform calculations on the trained supervised learning model, and output a strategy for the path the device should sequentially take from its current position. A movement control unit is configured to control the movement of the device from the starting point to the destination point via the intermediate points, based on a strategy for the path the device should sequentially take from its current position, which is output by the calculation unit. A route management system equipped with the following features.

Images