Route management device, route management method, and route management system

Abstract

To manage, with a simpler configuration, the route of a mobile object moving to a destination point via a specified waypoint. [Solution] The route management device 1 includes a first learning unit 12 that applies a reward function to an estimated result of calculating a course that the moving body 2 should take sequentially from an initial point position constituting each of the multiple partial routes until the moving body 2 reaches a final point position, and updates the course so as to maximize a reward for the moving body 2 to reach the final point position, using a reinforcement learning model, and a second learning unit 13 that learns the relationship between the current position of the moving body 2 and the course measures that the moving body 2 should take sequentially from its current position for the multiple partial routes obtained by learning by the first learning unit 12, using a supervised learning model.

Description

[Technical field]

[0001] The present invention relates to a route management device, a route management method, and a route management system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been known a technique for learning an optimal movement route for a moving object from a starting point to a destination point by using reinforcement learning (see Patent Document 1).

[0003] However, the route management technology for a moving object described in Patent Document 1 has a problem in that when a moving object learns a route from a starting point to a destination point via specified intermediate points, the reward design becomes complicated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-173643 A Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional technology, it has been difficult to manage, with a simpler configuration, the route of a mobile object moving to a destination point via specified waypoints.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to manage the route of a moving object moving to a destination point via specified intermediate points with a simpler configuration. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the route management device of the present invention includes a division unit configured to divide a route of a moving body from a starting point to a destination point via intermediate points into a plurality of partial routes; a first learning unit configured to apply a reward function to an estimated result of calculating a course that the moving body should take sequentially from the position of an initial point that constitutes a partial route until the moving body reaches the position of a final point, for each of the plurality of partial routes, to update the course so as to maximize the reward for the moving body to reach the position of the final point, and to learn a course plan for the moving body to take sequentially from its current position using a reinforcement learning model; a second learning unit configured to learn, using a supervised learning model, the relationship between the current position of the moving body and the course plan for the moving body to take sequentially from its current position for each of the plurality of partial routes obtained by learning by the first learning unit; and a memory unit configured to store the learned supervised learning model constructed by the second learning unit.

[0008] In addition, in the route management device of the present invention, the waypoint may include a plurality of waypoints in a predetermined order, and the plurality of partial routes may include a route from the starting point to the first waypoint that the moving body passes through, a route between the waypoints that are passed through sequentially, and a route from the last waypoint that is passed through to the destination point.

[0009] In addition, the route management device of the present invention may further include a setting unit configured to set the learned supervised learning model to the moving body as control information for controlling the route of the moving body from the starting point via the intermediate point to the destination point.

[0010] In addition, in the route management device of the present invention, the movement space in which the moving body moves may be defined as a unit space divided into a plurality of spaces, and a course plan for the moving body to proceed sequentially from the current position may be a course plan for the unit space along which the moving body should proceed sequentially from a position in the unit space corresponding to the current position.

[0011] In addition, in the route management device of the present invention, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the first learning unit provides a position in the unit space corresponding to the current position as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action value function that represents an expected value of the future cumulative value of the reward to be obtained when the moving body takes each action related to moving in a predetermined direction relative to the traveling direction as a course to be followed next from the position in the unit space corresponding to the current position, the first learning unit further provides a position in the unit space that the moving body next reaches as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action value function, and the first learning unit may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate.

[0012] In order to solve the above-mentioned problems, the route management method of the present invention includes a division step of dividing a route of a moving body from a starting point to a destination point via intermediate points into a plurality of partial routes; a first learning step of applying a reward function to an estimated result of calculating a course that the moving body should take sequentially from an initial point position constituting a partial route until the moving body reaches a final point position, for each of the plurality of partial routes, updating the course so as to maximize a reward for the moving body to reach the final point position, and learning a course plan for the moving body to take sequentially from its current position using a reinforcement learning model; a second learning step of learning, using a supervised learning model, the relationship between the current position of the moving body and the course plan for the moving body to take sequentially from its current position for each of the plurality of partial routes obtained by learning in the first learning step; and a storage step of storing the learned supervised learning model constructed in the second learning step in a storage unit.

[0013] In addition, in the route management method of the present invention, the waypoint may include a plurality of waypoints in a predetermined order, and the plurality of partial routes may include a route from the starting point to the first waypoint that the moving body passes through, a route between the waypoints that are passed through sequentially, and a route from the last waypoint that is passed through to the destination point.

[0014] In addition, the route management method of the present invention may further include a setting step of setting the learned supervised learning model to the moving body as control information for controlling the route of the moving body from the starting point via the intermediate point to the destination point.

[0015] In addition, in the route management method of the present invention, the movement space in which the moving body moves may be defined as a unit space divided into a plurality of spaces, and a course plan for the moving body to proceed sequentially from the current position may be a course plan for the unit space along which the moving body should proceed sequentially from a position in the unit space corresponding to the current position.

[0016] In addition, in the route management method of the present invention, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the first learning step may include providing a position in the unit space corresponding to the current position as an input to the neural network model, performing calculations on the neural network model, and outputting a first estimate of an action value function representing an expected value of the future cumulative value of the reward to be obtained when the moving body takes each action related to moving in a predetermined direction relative to the traveling direction as a course to be followed next from the position in the unit space corresponding to the current position, the first learning step may further provide a position in the unit space next reached by the moving body as an input to the neural network model, performing calculations on the neural network model, and outputting a second estimate of the action value function, and the first learning step may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate.

[0017] In order to solve the above-mentioned problems, the route management system of the present invention is a route management system comprising the above-mentioned route management device and the mobile body, wherein the mobile body comprises a second acquisition unit configured to acquire the learned supervised learning model constructed by the route management device, a third acquisition unit configured to acquire the current position of the device, a calculation unit configured to provide the current position of the device acquired by the third acquisition unit as an unknown input to the learned supervised learning model, perform calculations of the learned supervised learning model, and output a course plan for the device to proceed sequentially from the current position of the device, and a mobility control unit configured to control the movement of the device from the starting point to the destination point via the intermediate point based on the course plan for the device to proceed sequentially from the current position of the device output by the calculation unit. Effect of the Invention

[0018] According to the present invention, the relationship between the current position of the moving object and the measures of the paths that the moving object should take from the current position for each of the partial routes obtained by the learning by the first learning unit is learned using a supervised learning model. Therefore, the route of the moving object moving to the destination point via the specified waypoints can be managed with a simpler configuration. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram showing a configuration of a route management system including a route management device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining the learning process by the first learning unit included in the route management device according to the present embodiment. [Diagram 3] FIG. 3 is a block diagram showing a configuration of a first learning unit included in the route management device according to the present embodiment. [Figure 4] FIG. 4 is a diagram for explaining the learning process by the second learning unit included in the route management device according to the present embodiment. [Diagram 5]FIG. 5 is a block diagram showing a hardware configuration of the path management device according to the present embodiment. [Figure 6] FIG. 6 is a block diagram showing the configuration of a moving object provided in the route management system according to the present embodiment. [Figure 7] FIG. 7 is a block diagram showing a hardware configuration of a moving object provided in the route management system according to the present embodiment. [Figure 8] FIG. 8 is a sequence diagram showing the operation of the route management system according to the present embodiment. [Figure 9] FIG. 9 is a flowchart showing the first learning process of the route management device according to the present embodiment. [Figure 10] FIG. 10 is a flowchart showing the first learning process of the route management device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to FIGS.

[0021] [Route management system configuration] First, with reference to FIG. 1, an overview of a route management system including a route management device 1 and a mobile object 2 according to an embodiment of the present invention will be described.

[0022] The route management system according to the present embodiment includes a route management device 1 and a mobile body 2, and the route management device 1 and the mobile body 2 are communicably connected to each other via a wireless communication network NW that complies with a predetermined communication standard such as LTE / 4G, 5G, or 6G. The route management system manages a route for the mobile body 2 to move to a destination point via waypoints set in a moving space A. As shown in FIG. 1, the moving space A in which the mobile body 2 moves is capable of communication by, for example, a 5G wireless communication method.

[0023] The moving body 2 includes an air vehicle capable of autonomous flight such as a drone or an unmanned aerial vehicle, an autonomous vehicle, a mobile robot, a ship, etc. The moving body 2 starts from an arbitrary current position or a preset position in the mobile space A, moves from the starting position to a destination position via a preset waypoint, and executes a predetermined task. Furthermore, the moving body 2 can execute the set task at the specified waypoint.

[0024] In the following, an example will be described in which the moving body 2 is a drone. The moving body 2 controls its autonomous flight by a flight controller that processes information from a sensor 208 (to be described later) and controls the rotation speed of a motor 209 and the angle of a propeller 210. The moving body 2 also acquires its own GPS position by a GPS receiver 207. The moving body 2 is configured as an IoT terminal having an IP address, and each IP address can uniquely identify the moving body 2. Furthermore, in this embodiment, the moving body 2 controls the autonomous flight to a destination point via a specified waypoint based on control information stored in the auxiliary storage device 205. The functional blocks and hardware configuration of the moving body 2 will be described in detail later. Note that a configuration in which a plurality of moving bodies 2 exist may be used, and in that case, each of the moving bodies 2 has the same configuration.

[0025] As shown in FIG. 1, the mobile space A in which the mobile object 2 moves is a three-dimensional matrix-like space composed of unit spaces divided into a plurality of spaces. Each unit space constituting the mobile space A has the same volume. Furthermore, each unit space has a node ID, and each unit space is represented by one position (x, y, z). The position information can be three-dimensional GPS position coordinates consisting of latitude, longitude, and altitude. For example, a representative value such as the center position of the unit space can be used as the position of the unit space.

[0026] 1, the moving body 2 moves from a position in the unit space corresponding to the position of the starting point S to a position in the unit space of the destination point G, using each of the unit spaces corresponding to the positions of the waypoints WP1, WP2, ..., WPn (n is a positive integer equal to or greater than 1) as waypoints. In this embodiment, the moving body 2 moves from the starting point S to the destination point G via a series of waypoints WP1, WP2, ..., WPn, the order of which is determined.

[0027] The route management system according to the present embodiment divides the route between the starting point S, the waypoints WP1, WP2, . . . , WPn, and the destination point G into partial routes, and learns an optimal route for each partial route by reinforcement learning. Furthermore, using the route measures of the moving body 2 for the multiple partial routes obtained by reinforcement learning as teacher data, the relationship between the current unit space position of the moving body 2 and the route measures of the moving body 2 to proceed sequentially is learned using a supervised learning model. Furthermore, the learned supervised learning model is set in the moving body 2 as control information for controlling the route of the moving body 2. Based on the set control information, the moving body 2 performs calculations of the learned supervised learning model using the current unit space position of the device as an unknown input, and outputs the route measures to proceed sequentially. Then, based on the output route measures to proceed sequentially, the route from the starting point to the destination point via the specified waypoints is determined, and movement to the destination point is controlled.

[0028] The moving body 2 for which the control information has been set moves in a direction to move in each unit space by changing the course as shown by each arrow of the starting point S in FIG. 1 according to the course determined based on the course policy obtained by the calculation of the learned supervised learning model. The course can include various courses, i.e., moving directions. In FIG. 1, the moving space A is described as a two-dimensional plane, but the course of the moving body 2 can be a three-dimensional course. The control information set by the route management device 1 to the moving body 2 allows the moving body 2 to reach the unit space of the destination point G from the unit space of the starting point via the designated intermediate point.

[0029] Here, a course refers to a movement from a position in each unit space to a position in an adjacent unit space. Also, a route includes the entire route from a starting point to a destination point via a number of unit spaces of designated intermediate points. Furthermore, a route between points is specifically called a partial route, and the entire route is formed by combining partial routes.

[0030] [Route management device functional block] 1, the route management device 1 includes a first acquisition unit 10, a division unit 11, a first learning unit 12, a second learning unit 13, a first storage unit (storage unit) 14, a second storage unit 15, a third storage unit 16, and a setting unit 17. The route management device 1 manages a route of a moving object 2 to a destination position via waypoints set in a movement space A.

[0031] The first acquisition unit 10 acquires the starting point, the waypoint, and the destination point set in the moving space A. The first acquisition unit 10 also acquires the current position of the moving body 2. Specifically, the first acquisition unit 10 acquires the node ID of the unit space corresponding to the starting point, the node ID of the unit space corresponding to the waypoint, and the node ID of the unit space corresponding to the destination point, which are stored in the third storage unit 16 described later. The first acquisition unit 10 also acquires the unit space in which the moving body 2 is currently located as the current position. The current position is the position of the unit space in which the moving body 2 exists for each set time t.

[0032] The first acquisition unit 10 acquires a position in the unit space corresponding to the GPS position of the moving object 2 for each time t as the current position of the moving object 2. Note that there are cases where a plurality of position coordinates are included in the unit space. In such a case, the first acquisition unit 10 can acquire the position in the unit space that is closest to the GPS position received by the GPS receiver 207 of the moving object 2 as the current position of the moving object 2.

[0033] The dividing unit 11 divides the route from the departure point of the moving object 2 to the destination point via waypoints into a plurality of partial routes. More specifically, the dividing unit 11 divides the route into a plurality of partial routes including a route from the departure point to the waypoint that the moving object 2 passes through first, a route between waypoints that the moving object 2 passes through in sequence, and a route from the waypoint that the moving object 2 passes through last to the destination point. In the example of FIG. 1, the dividing unit 11 divides the route into a partial route from the departure point S to the waypoint WP1 that the moving object 2 passes through first, a partial route from the waypoint WP1 to the next waypoint WP2, a partial route from the waypoint WP2 to the next waypoint WP3, and so on, and further divides the route into a partial route from the waypoint WPn that the moving object 2 passes through last to the destination point G. An ID is given to each partial route, and the initial point and the final point of each partial route can be specified. For example, the initial point of the partial route specified by ID1 is the departure point S, and the final point is the waypoint WP1.

[0034] The first learning unit 12 applies a reward function to the estimated results of calculating the sequential path that the moving body 2 should take from the initial point position of the partial path to the final point position for each of the multiple partial paths, and updates the result so as to maximize the reward for the moving body 2 to reach the final point position, and learns the strategy for the sequential path that the moving body 2 should take from its current position using a reinforcement learning model.

[0035] In this embodiment, as a course of action for the moving body 2 to move sequentially from the position of each unit space, the moving body 2 may take actions a, b, c, and cs related to movement in a predetermined number of directions (n ​​is an integer of 2 or more) relative to the moving direction. n The moving direction is based on the position of the moving object 2 in the unit space immediately before.

[0036] The first learning unit 12 uses a neural network model including an input layer s, a hidden layer h, and an output layer q as a reinforcement learning model as shown in FIG. t , and all the action value functions Q(s t ,a1), Q(s t ,a2), Q(s t,a3), ,Q(s t ,a n-1 ), Q(s t ,a n We adopt Deep Q-Network (DQN), a neural network that outputs

[0037] More specifically, the first learning unit 12 inputs the current position of the unit space, which indicates the current position of the moving object 2, as an input to the neural network model, performs calculations on the neural network model, and learns actions a related to each of the movements in n directions as a course to be taken by the moving object 2 from the current position in the unit space. n The first estimate Q1 of the action value function that represents the expected cumulative value of future rewards obtained if

[0038] The reward is a state s indicating the current position of the moving object 2, and an action a for the moving object 2 to move in a given direction. n , and a reward function r=r(s, a, s') for the next position of the moving object 2, i.e., the next state s'. In this embodiment, the reward function includes as a variable the degree of arrival at a position in a unit space related to the final point of the moving object 2. In addition, it can include as a variable the degree of arrival at a position in a unit space corresponding to a space containing obstacles such as high-rise buildings and steel towers. For example, when an action related to the movement of the moving object 2 in a specified direction brings the moving object 2 closer to the final point or when the moving object 2 reaches the final point in the shortest distance, the reward, which is a scalar quantity, is set to a larger value.

[0039] On the other hand, when the moving object 2 moves away from the final point or reaches a unit space where an obstacle exists, a negative reward value (e.g., r=-1) can be given. In this way, by setting the reward for a unit space where an obstacle exists as a negative value, the moving object 2 can reach the destination point while avoiding these points.

[0040] Furthermore, the first learning unit 12 provides the next position in the unit space reached by the moving object 2 as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function. The first learning unit 12 learns weight parameters of the neural network model so that the first estimated value Q1 becomes a target value calculated from the second estimated value Q2.

[0041] If the weight parameter of the neural network model is θ and the action value function is Q(s, a; θ), the minimization loss function for learning is given by the following equation (1). L(θ)=1 / 2{r+γmax a’ Q(s',a';θ)-Q(s,a;θ)} 2 (1)

[0042] In the above formula (1), r is the reward (immediate reward) and γ is the discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value at the state s' one step forward, that is, the second estimated value Q2. The target value is r+γmax a’ It is expressed as Q(s',a';θ).

[0043] The first learning unit 12 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above formula (1).

[0044] More specifically, the first learning unit 12 may employ a Fixed Target Q-Network that uses two neural networks, a main QN121 and a target QN123, as shown in FIG. 3. The main QN121 selects an optimal action and updates an action-value function Q. Meanwhile, the target QN123 estimates and evaluates the value of an action a' to be taken in the next state s' of the result of the action. The main QN121 and the target QN123 have neural networks with the same layer structure, but the parameter of the main QN121 is "θ" and the parameter of the target QN123 is "θ". - " is given by

[0045] The main QN 121 receives the current position of the moving object 2 as a state s from the environment 120. The environment 120 is a system of a moving space in which the moving object 2 is placed, and in this environment 120, the moving object 2 moves to another unit space by taking an action a related to movement in a predetermined direction, and at the same time, transitions to the next state s' and acquires a reward r from the environment 120.

[0046] The first learning unit 12 inputs the state s relating to the current position of the moving object 2 to the main QN 121 and obtains the action value function Q(s, a; θ). The first learning unit 12 calculates the action a using, for example, the ε-greedy method, or calculates the optimal action argmax a Q(s, a; θ) is calculated. In the environment 120, the moving object 2 takes the action argmax a Q(s, a; θ). The environment 120 is a space where the moving object 2 takes the action argmax a As a result of performing Q(s, a; θ), the position in the unit space to which the robot has moved is observed as the next state s', and a reward r is output. The experience data 124 stores the experience (s, a, r, s') output from the environment 120.

[0047] The first learning unit 12 calculates a loss function L in a DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L.

[0048] The first learning unit 12 periodically copies the weights of the main QN 121 to the target QN 123 to perform synchronization. The synchronization of the target QN 123 is performed less frequently than the update frequency of the weights of the main QN 121. The first learning unit 12 extracts experience from the experience data 124, inputs past states to the target QN 123, and calculates the estimated value max a’ Q(s', a';θ - The first learning unit 12 outputs the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ -) is used to learn the weights of the main QN 121 in the DQN loss calculation 122.

[0049] A strategy for a path to be taken sequentially from an initial point position to a final point position of each of the multiple partial paths, obtained by learning by the first learning unit 12, i.e., a learned reinforcement learning model, is stored in a first storage unit 14. In addition, the learned reinforcement learning model constructed for each of the multiple partial paths is used as teacher data in learning by the second learning unit 13.

[0050] The second learning unit 13 uses a supervised learning model to learn the relationship between the current position of the moving body 2 and the course plan that the moving body 2 should take sequentially from its current position for each of the multiple partial routes obtained by learning by the first learning unit 12.

[0051] FIG. 4 shows the structure of a neural network model employed as an example of a supervised learning model used by the second learning unit 13. The neural network model includes an input layer x, a hidden layer h, and an output layer y. The second learning unit 13 provides the current unit space position of the mobile object 2, i.e., the unit space position corresponding to the GPS position of the mobile object 2 at each time t, to the input layer of the neural network model, applies an activation function to the weighted sum of the input, and passes the output determined by threshold processing to the output layer. Each output node of the output layer outputs a predicted output of the model corresponding to n action value functions Q of each partial path.

[0052] The second learning unit 13 introduces an objective function E shown in the following equation (2) to learn parameters of the neural network model so that the plan for the path to be taken sequentially from the current position of the moving body 2, which is a predicted value from the neural network model for the current position of the moving body 2, becomes the value of an optimized plan for the path to be taken sequentially from the current position of the moving body 2, which is reinforcement learned for each partial route by the first learning unit 12.

[0053]

number

[0054] In the above formula (2), y1, y2, , y n indicates the predicted output value of each output node. Also, Y1, Y2, . . . , Y n is the teacher data, and in this case, is n optimized action value functions Q(s) of a certain partial path for the current position, obtained by the reinforcement learning by the first learning unit 12. Furthermore, as described above, the first learning unit 12 performs reinforcement learning for each of the N partial paths, and obtains n optimized action value functions Q(s) of each partial path. t ,a1), Q(s t ,a2), Q(s t ,a3), ,Q(s t ,a n-1 ), Q(s t ,a n ).

[0055] The objective function E in the above formula (2) is further divided into the optimized action value functions Q1 to Q N In this case, when the training data for the first partial path is Y1, the optimized n action value functions Q1(s t ,a1), Q1(s t ,a2), Q1(s t ,a3),···,Q1(s t ,a n-1 ), Q1(s t ,a n ), the n optimized action value functions Q2(s t ,a1), Q2(s t ,a2), Q2(s t ,a3),···,Q2(s t ,a n-1 ), Q2(s t ,a n ), ..., training data Y for the Nth partial path N The optimized n action value functions Q N (s t ,a1), Q N(s t ,a2), Q N (s t ,a3), ,Q N (s t ,a n-1 ), Q N (s t ,a n ) is used as training data.

[0056] In this case, the above formula (2) is further expressed as the following formula (3).

number

[0057] In the objective function E in the above equation (3), y 1_1 ,y 1_2 ,···,y 1_n ,···,y 2_1 ,y 2_2 ,···,y 2_n ,···,y N_1 ,y N_2 ,···,y N_n is the predicted output value of n (n action value functions Q) × N (number of partial paths N) output nodes. Y 1_1 ~Y 1_n corresponds to the optimized action value function Q1 of the first partial path, and Y 2_1 ~Y 2_n corresponds to the optimized action value function Q2 of the second partial path, and Y N_1 ~Y N_n is the optimized action value function Q of the Nth partial path. N This is the training data corresponding to

[0058] In this way, all of the optimized action value functions Q from the initial point to the final point of each of the multiple partial paths obtained by the trained reinforcement learning model are used as training data. In the example of the neural network model configuration in Figure 4, training data is shown to the right of each output node that outputs the predicted output value of the model. Starting from the top output node, the training data Y 1_1 ~Y 1_nshows n action value functions Q1 related to the first partial path, while the training data Y2...Y N For each of the action value functions Q2, ,Q N In the figure, the n values ​​corresponding to the above are omitted.

[0059] The value of the objective function E in the above formula (3) is the above input value x of the supervised learning model, and the output value y for the position in the unit space corresponding to the GPS position of the mobile unit 2 at time t. 1_1 ,y 1_2 ,···,y 1_n ,···,y 2_1 ,y 2_2 ,···,y 2_n ,···,y N_1 ,y N_2 ,···,y N_n is the target output of the training data Y 1_1 ~Y 1_n ,Y 2_1 ~Y 2_n ,···,Y N_1 ~Y N_n The second learning unit 13 adjusts the weight parameters of the neural network related to the supervised learning model so that the objective function E is minimized, that is, becomes 0. The second learning unit 13 can optimize the objective function E by using the backpropagation method or the like.

[0060] The first storage unit 14 stores a trained reinforcement learning model constructed by reinforcement learning by the first learning unit 12. Since the trained reinforcement learning model is constructed for each of a plurality of partial paths, N trained reinforcement learning models corresponding to the N partial paths are stored.

[0061] The second storage unit 15 stores the trained supervised learning model constructed by the second learning unit 13 through supervised learning.

[0062] The third storage unit 16 stores a departure point, a waypoint, and a destination point set in advance of the moving object 2. The third storage unit 16 also stores position information of the unit spaces constituting the moving space A, and identification information of the moving object 2.

[0063] The setting unit 17 sets the trained supervised learning model in the moving object 2 as control information for controlling the course of the moving object 2 from the starting point to the destination point via the intermediate points. For example, the setting unit 17 can transmit the control information to the moving object 2 via the network NW.

[0064] [Hardware configuration of route management device] Next, an example of a hardware configuration for realizing the route management device 1 having the above-mentioned functions will be described with reference to FIG.

[0065] 5, the path management device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output I / O 106, which are connected via a bus 101, and a program that controls these hardware resources. Furthermore, the path management device 1 can include a display device 107 connected via the bus 101.

[0066] The processor 102 is realized by a CPU, a GPU, an FPGA, an ASIC, or the like.

[0067] The main memory device 103 prestores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 implement the functions of the route management device 1, such as the first acquisition unit 10, the division unit 11, the first learning unit 12, the second learning unit 13, and the setting unit 17 shown in FIG.

[0068] The communication interface 104 is an interface circuit for connecting the route management device 1 to various external electronic devices via a network.

[0069] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or a flash memory as the storage medium.

[0070] The auxiliary storage device 105 has a program storage area for storing the route management program executed by the route management device 1. The auxiliary storage device 105 also has a program storage area for storing the reinforcement learning program executed by the route management device 1. The auxiliary storage device 105 also has an area for storing the supervised learning program. The first storage unit 14, the second storage unit 15, and the third storage unit 16 described in FIG. 1 are realized by the auxiliary storage device 105. Specifically, the auxiliary storage device 105 has an area for storing the waypoints and the destination point of the moving object 2. The auxiliary storage device 105 also has an area for storing the position coordinates of the moving space and the position coordinates of the unit space. The auxiliary storage device 105 also has an area for storing identification information such as the IP address of the moving object 2. Furthermore, the auxiliary storage device 105 may have, for example, a backup area for backing up the above-mentioned data and programs.

[0071] The input / output I / O 106 is an input / output device that inputs signals from an external device and outputs signals to an external device.

[0072] The display device 107 is configured with an organic EL display, a liquid crystal display, etc. The display device 107 can display a map display of the moving space, the current position of the moving object 2, a progress status, a partial route and the entire route, and position information of the destination point.

[0073] [Functional blocks of moving objects] Next, functional blocks of the moving object 2 will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the moving object 2.

[0074] The moving body 2 includes a fourth storage unit 20, a second acquisition unit 21, a fifth storage unit 22, a third acquisition unit 23, a calculation unit 24, a determination unit 25, and a movement control unit 26. The moving body 2 determines the next route that the moving body 2 should take based on the control information set by the route management device 1, and controls the flight of the moving body 2 to the destination position via a specified waypoint.

[0075] The fourth storage unit 20 stores the control information set by the setting unit 17 of the route management device 1. The control information is a learned supervised learning model in which the second learning unit 13 of the route management device 1 has learned, by supervised learning, a course plan to be taken from the current position from the starting point to the destination point via specified intermediate points.

[0076] The second acquisition unit 21 acquires the control information set by the route management device 1. Specifically, the second acquisition unit 21 loads the control information stored in the fourth storage unit 20.

[0077] The fifth storage unit 22 stores map data including position coordinates of the moving space, and information associating position coordinates of unit spaces constituting the moving space with node IDs of the unit spaces. The fifth storage unit 22 also stores position information of designated waypoints and destination points. The fifth storage unit 22 also stores IDs of multiple partial routes.

[0078] The third acquisition unit 23 acquires the current position of the own device. More specifically, the third acquisition unit 23 acquires the position of the unit space in which the own device is located for each time t based on the GPS position of the own device. The third acquisition unit 23 refers to the fifth storage unit 22 and acquires the position of the unit space corresponding to the current GPS position received by the GPS receiver 207 as the current position of the own device.

[0079] The calculation unit 24 provides the current position of the device acquired by the third acquisition unit 23 as an unknown input to the trained supervised learning model, performs calculations on the trained supervised learning model, and outputs a course plan for the device to follow sequentially from its current position.

[0080] The determination unit 25 determines the next route to be taken from the current position of the device itself based on the route plan that the device itself should take from the current position, which is output by the calculation unit 24. More specifically, the determination unit 25 determines the current position in the unit space as a state s t For each state s t The path to be taken is determined by selecting the path that is action a with the maximum value of the action value function Q.

[0081] The movement control unit 26 controls the movement of the own device based on the plan of the route that the own device should take from the current position, which is output by the calculation unit 24. Specifically, the movement control unit 26 controls the movement of the own device based on the next route that is output by the calculation unit 24 and determined by the determination unit 25. The movement control unit 26 can calculate a control command for the next route that the own device should take from the current position, and transmit the control command value to the motor 209. In this way, the moving object 2 can control the movement of the own device based on the plan of the route that the own device should take from the current position, which is output by the calculation unit 24 and determined by the determination unit 25. t By selecting the action a with the maximum value of the action value function Q, a group can move from different starting points to a common destination.

[0082] [Hardware configuration of mobile object] Next, an example of a hardware configuration for realizing the moving object 2 having the above-mentioned functions will be described with reference to FIG.

[0083] 7, the moving body 2 can be realized by, for example, a microcomputer including a processor 202, a main memory device 203, a communication interface 204, an auxiliary memory device 205, and an input / output I / O 206, which are connected via a bus 201, a program for controlling these hardware resources, a GPS receiver 207, a sensor 208, a motor 209, a propeller 210, and a battery 211. A flight controller that controls the autonomous flight of the moving body 2 is realized by a computer such as a microcomputer and a program. The moving body 2 can mount these components on a frame of a predetermined shape.

[0084] A program for the processor 202 to control movement and perform calculations is stored in advance in the main memory device 203. The processor 202 and the main memory device 203 realize each function of the moving object 2, such as the second acquisition unit 21, the calculation unit 24, the determination unit 25, and the movement control unit 26 shown in FIG.

[0085] The communication interface 204 is an interface circuit for connecting the mobile unit 2 and the route management device 1 to a network.

[0086] The auxiliary storage device 205 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from the storage medium. The auxiliary storage device 205 can use a semiconductor memory such as a hard disk or a flash memory as the storage medium.

[0087] The auxiliary storage device 205 has a program storage area for storing a movement control program executed by the moving object 2. The auxiliary storage device 205 also has an area for storing a calculation program for performing calculations of a learned supervised learning model. The auxiliary storage device 205 realizes the fourth storage unit 20 and the fifth storage unit 22 described in FIG. 6. The auxiliary storage device 205 also has an area for storing identification information such as an IP address of the moving object 2. The auxiliary storage device 205 also has an area for storing an application that notifies the route management device 1 of a GPS position. Furthermore, for example, the auxiliary storage device 205 may have a backup area for backing up the above-mentioned data and programs.

[0088] The input / output I / O 206 is an input / output device that inputs signals from an external device and outputs signals to an external device.

[0089] An antenna for receiving GPS signals is built into the GPS receiver 207. The GPS receiver 207 realizes the third acquisition unit 23 in FIG.

[0090] The sensor 208 is composed of various sensors such as an altitude sensor, an attitude sensor, a camera, LiDAR, and RADAR. In addition to the GPS receiver 207, the altitude sensor realizes the third acquisition unit 23 in Fig. 6. Furthermore, the flight controller performs flight control of the moving object 2 based on the various sensor data measured by the sensor 208.

[0091] The motor 209 rotates by the rotary drive, and rotates a propeller 210 attached to the rotary shaft of the motor 209 .

[0092] The battery 211 is a lithium ion battery or the like, and supplies power to the components of the moving object 2.

[0093] The mobile unit 2 may include a SIM and may have an International Mobile Subscriber Identity (IMSI) of the SIM.

[0094] [Operation of the route management system] Next, the operation of the route management system including the route management device 1 and the moving object 2 having the above-mentioned configuration will be described with reference to the sequence of Fig. 8. The moving object 2 is placed at a position of a predetermined unit space in the moving space, and the position is set as a starting point.

[0095] The first acquisition unit 10 of the route management device 1 acquires information on a starting point set in the moving space A, a plurality of waypoints whose order of passing through is determined, and a destination point (step S1). Next, the division unit 11 divides the route from the starting point to the destination point via the waypoints acquired in step S1 into a plurality of partial routes (step S2). More specifically, in step S2, as described in FIG. 1, the division unit 11 divides the route into a partial route from the starting point S to the first waypoint WP1, a plurality of partial routes between the waypoints WP1 to WPn, and a partial route from the last waypoint WPn to the destination point G. Each of the partial routes has a different initial point and final point. The entire route is formed by combining the plurality of partial routes.

[0096] Next, the first acquisition unit 10 acquires the position of the unit space in which the mobile object 2 is currently located as the current position of the mobile object 2 (step S3). For example, the first acquisition unit 10 can acquire the current position of the mobile object 2 by having the mobile object 2 notify the current GPS position of the mobile object 2 by push notification to the IP address of the mobile object 2.

[0097] The first acquisition unit 10 acquires the position of the unit space where the moving object 2 currently resides for each time t. Since the starting point is determined in advance, the position of the unit space where the moving object 2 currently resides at the first time t is the position of the unit space corresponding to the starting point. The first acquisition unit 10 acquires the position of the unit space corresponding to the current GPS position received by the GPS receiver 207 of the moving object 2 as the current position of the moving object 2.

[0098] Next, the first learning unit 12 performs a first learning process (step S4). In the first learning process, the first learning unit 12 applies a reward function to an estimation result obtained by calculating a course that the moving body 2 should take from the initial point position of the partial path to the final point position of the partial path for each of the multiple partial paths, updates the course so as to maximize the reward for the moving body 2 to reach the final point position, and learns a policy for the course that the moving body 2 should take from the current position using a reinforcement learning model. In step S3, the learning process is performed for each of the multiple partial paths. The first learning process will be described in detail later.

[0099] Thereafter, the first storage unit 14 stores the trained reinforcement learning model obtained in step S4 (step S5). Next, the second learning unit 13 learns the relationship between the current position of the moving object 2 and the measures of the paths that the moving object 2 should take from the current position, which are obtained in the first learning process for each of the multiple partial routes in step S4, using a supervised learning model (second learning process) (step S6).

[0100] Specifically, the second learning unit 13 repeatedly adjusts and updates parameters such as weights and thresholds so that the error between the predicted output value of the policy for the sequential course to be taken when the position in the unit space corresponding to the current GPS position of the mobile object 2, i.e., the current state, is given as an input value to the supervised learning model and the teacher data minimizes the objective function E in the above formula (3), and determines the values ​​of these parameters. In step S6, the second learning unit 13 can determine the parameters that minimize the objective function E by using an error backpropagation method or the like.

[0101] The training data used in step S6 is a course plan for a course to be taken sequentially from the current position in the unit space, which is obtained by the trained reinforcement learning model constructed for each of the plurality of partial paths in the first learning process in step S4.

[0102] Next, the second storage unit 15 stores the trained supervised learning model constructed in step S6 (step S7). After that, the setting unit 17 sets the trained supervised learning model as control information in the moving object 2 (step S8). In step S8, the setting unit 17 can transmit the trained supervised learning model to the moving object 2 via the network NW.

[0103] Thereafter, the second acquisition unit 21 of the moving object 2 acquires the trained supervised learning model constructed by the route management device 1 (step S9). The second acquisition unit 21 reads out the control information transmitted from the route management device 1 and stored in the fourth storage unit 20, i.e., the trained supervised learning model.

[0104] Next, the third acquisition unit 23 acquires the current unit space position of the device as the current position (step S10). Specifically, the unit space position corresponding to the GPS position received by the GPS receiver 207 can be acquired as the current position of the device. Next, the calculation unit 24 uses the control information acquired in step S9 to provide the current unit space position of the device acquired in step S10 as an unknown input, performs calculations of the learned supervised learning model, and outputs a course plan to be taken from the current unit space position (step S11).

[0105] For example, in the moving space A of FIG. 1, when the position of the starting point S of the moving object 2 is input as the current position at time t=1 to the trained supervised learning model, n action value functions Q to be followed next from the starting point S at time t=1 are output. More specifically, when the current position of the moving object 2 is given as an unknown input, the value of the policy of n routes of a specific partial path corresponding to the current position of the moving object 2 from among the optimized route policies of N partial paths is output as the calculation result of the trained supervised learning model. The value of the policy of each of the n routes of the other partial paths not corresponding to the current position of the moving object 2 is 0.

[0106] Next, the decision unit 25 selects a path along which the action a having the maximum value of the n action value functions Q output in step S11 is taken, thereby deciding a path to be taken in sequence (step S12). Next, the movement control unit 26 controls the movement of the device itself based on the path to be taken next decided in step S12 (step S13). More specifically, the movement control unit 26 can calculate a control command for the path to be taken next from the current position, and transmit the control command value to the motor 209.

[0107] The moving body 2 repeats the processes from step S10 to step S13 until it reaches the unit space of the specified waypoint from the starting point (step S14: NO). If it reaches the unit space of the specified waypoint (step S14: YES), it further repeats the processes from step S10 to step S14 until it reaches the destination point (step S15: NO). Thereafter, when the moving body 2 reaches a position in the unit space of the destination point (step S15: YES), the process ends. In this way, the moving body 2 can move from the starting point to the destination point via the specified waypoint and perform a predetermined task by executing the processes from step S9 to step S15 using the trained supervised learning model, which is control information.

[0108] Next, the first learning process (step S4 in FIG. 8) by the route management device 1 will be described with reference to the flowcharts in FIG. 9 and FIG. 10. First, step S3 described in FIG. 8 is executed. In addition, below, the learning process is performed for each of the multiple partial routes. First, the first learning unit 12 inputs the position of the unit space where the moving object 2 is currently located, which is the current state of the moving object 2 acquired in step S3, to the neural network model, performs calculations on the neural network model, and outputs a first estimated value Q1 of an action value function that represents an expected value of a cumulative value of future rewards to be obtained when the moving object 2 takes each action related to movement in a predetermined direction relative to the traveling direction as a course to be followed next from the current position in the unit space (step S20).

[0109] Next, the first acquisition unit 10 acquires the unit space position of the moving object 2 at the next time t as the next state s' (step S21). The unit space position where the moving object 2 next arrives is determined based on the GPS position of the moving object 2 acquired by the first acquisition unit 10 for each time step. Furthermore, the first learning unit 12 provides the unit space position where the moving object 2 next arrives, acquired in step S21, as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function (step S22).

[0110] Next, the first learning unit 12 calculates a target value from the second estimated value Q2 (step S23). Then, the first learning unit 12 learns weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S24). Specifically, the first learning unit 12 updates the weight parameters of the neural network model so as to minimize the loss function of the above formula (1).

[0111] Thereafter, the processes from step S3 to step S24 are repeated until a trained reinforcement learning model is constructed for all of the partial paths (step S25: NO). If training is completed for all of the partial paths (step S25: YES), the first storage unit 14 stores the trained reinforcement learning model obtained in step S24 (step S4).

[0112] Next, the first learning process by the first learning unit 12 in the case where a Fixed Target Q-Network using two neural networks, a main QN 121 and a target QN 123, is adopted will be described with reference to FIG.

[0113] The process of step S3 is similar to the step of the first learning process described in Fig. 9. After that, the first learning unit 12 provides the position of the unit space where the moving object 2 is currently located, acquired in step S3, as an input to the main QN121, performs calculations of the neural network model, outputs an action value function Q, and calculates a course a to be taken next (step S120).

[0114] Next, the first learning unit 12 returns the behavior of the moving object 2 on the course a obtained in step S120 to the environment 120, and obtains the next state s' of the moving object 2, that is, the position in the unit space to which the moving object 2 has advanced, and the reward r (step S121).

[0115] The first learning unit 12 stores the experience (s, a, r, a') obtained in step S121 in the experience data 124 (step S122). Next, the first learning unit 12 calculates a loss function L in a DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L (step S123). The first learning unit 12 repeats the processes from step S120 to step S123 a set number of times.

[0116] After that, the first learning unit 12 periodically copies the weights of the main QN 121 to the target QN 123 to synchronize them (step S124). The synchronization of the target QN 123 is performed less frequently than the update frequency of the weights of the main QN 121. Next, the first learning unit 12 extracts experience from the experience data 124, inputs past states to the target QN 123, and calculates the estimated value max a’ Q(s', a';θ - ) is output (step S126).

[0117] Next, the first learning unit 12 calculates the estimated value max a’ Q(s', a';θ - ) based on the target value r+γmax a’ Q(s', a';θ - ) (step S127). Next, the first learning unit 12 calculates a loss function L in the DQN loss calculation 122 using the target value calculated in step S127 (step S128). Next, the first learning unit 12 learns the weights of the main QN 121 so as to minimize the loss given by the loss function L (step S129). After that, the first storage unit 14 stores the respective trained reinforcement learning models for all of the multiple partial paths (step S4).

[0118] As described above, according to the route management device 1 of the present embodiment, a route from a starting point to a destination point via a specified waypoint is divided into a plurality of partial routes, an optimal course plan for the moving body 2 is learned for each partial route by reinforcement learning, and the course plan for the plurality of partial routes obtained by reinforcement learning is used as teacher data to learn the relationship between the current position of the moving body 2 and the course plan for the sequentially moving path using a supervised learning model. Furthermore, the learned supervised learning model is set in the moving body 2 as control information for controlling the course. Therefore, the route of the moving body moving to the destination point via the specified waypoints can be managed with a simpler configuration.

[0119] Furthermore, according to the route management device of this embodiment, the route from the starting point to the destination point via the designated waypoints is divided into a plurality of partial routes, and an optimal course strategy for each of the moving objects 2 is learned by reinforcement learning. Therefore, even if a plurality of waypoints are designated, it is possible to suppress the complication of the reward design.

[0120] In addition, according to the route management system of this embodiment, by setting control information in the moving body 2, it is possible to realize a moving body 2 that is capable of moving along an optimal route to a destination point via specified intermediate points, thereby simplifying the configuration of the moving body 2.

[0121] In the embodiment described above, the reinforcement learning model used by the first learning unit 12 is exemplified as DQN related to Fixed Target Q-Network composed of a multi-layer neural network, but the reinforcement learning model can be CNN, multi-layer perceptron, etc. In addition to DQN exemplified as a reinforcement learning model, Double DQN, Dueling DQN, Actor-Critic (AC) method, Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Q-learning, etc. can be used.

[0122] In the embodiment described above, a multilayer neural network is used as the supervised learning model used by the second learning unit 13. However, the supervised learning model may be a multilayer perceptron, a decision tree-based model such as a random forest, a support vector machine, or the like.

[0123] The above describes embodiments of the route management device, route management method, and route management system of the present invention. However, the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can make are possible within the scope of the invention described in the claims. [Explanation of symbols]

[0124] 1...route management device, 10...first acquisition unit, 11...division unit, 12...first learning unit, 13...second learning unit, 14...first memory unit, 15...second memory unit, 16...third memory unit, 17...setting unit, 2...mobile body, 20...fourth memory unit, 21...second acquisition unit, 22...fifth memory unit, 23...third acquisition unit, 24...calculation unit, 25...determination unit, 26...mobility control unit, 101, 201...bus, 102, 202...processor, 103, 203...main memory device, 104, 204...communication interface, 105, 205...auxiliary memory device, 106, 206...input / output I / O, 107...display device, 207...GPS receiver, 120...environment, 121...main QN, 122...DQN loss calculation, 123...target QN, 124...empirical data, NW...network.

Claims

1. A division unit configured to divide a route of a moving object from a starting point to a destination point via waypoints into a plurality of partial routes; a first acquisition unit configured to acquire a current location of the mobile object from the mobile object via a network; a first learning unit configured to apply a reward function to an estimation result of calculating a course that the moving body should take sequentially from an initial point position constituting each of the plurality of partial paths until the moving body reaches a final point position, for each of the plurality of partial paths based on the current position of the moving body acquired by the first acquisition unit, and to update the course so as to maximize a reward for the moving body to reach the final point position, and to learn a policy for the moving body to take sequentially from the current position using a reinforcement learning model; a second learning unit configured to learn, using a supervised learning model, a relationship between the current position of the moving object and a course plan for the plurality of partial routes that the moving object should take from the current position, the course plan having been obtained by learning by the first learning unit; A storage unit configured to store the trained supervised learning model constructed by the second learning unit; Equipped with The reward function has a variable representing the degree of arrival of the moving object at the final point. A route management device comprising:

2. 2. The route management device according to claim 1, The waypoints include a plurality of waypoints in a predetermined order, The plurality of partial routes include a route from the departure point to the first waypoint that the moving body passes through, a route between the waypoints that the moving body passes through in sequence, and a route from the last waypoint that the moving body passes through to the destination point. A route management device comprising:

3. 2. The route management device according to claim 1, The method further includes a setting unit configured to set the trained supervised learning model to the moving body as control information for controlling a route of the moving body from the starting point to the destination point via the intermediate points. A route management device comprising:

4. 2. The route management device according to claim 1, A moving space in which the moving object moves is defined as a unit space divided into a plurality of spaces, The course plan for the moving object to proceed from the current position is a course plan for the moving object to proceed from a position in the unit space corresponding to the current position. A route management device comprising:

5. 5. The route management device according to claim 4, The reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer; the first learning unit provides a position in a unit space corresponding to the current position as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value of an action value function representing an expectation of a cumulative value of the future reward to be obtained when the moving body takes each action related to movement in a predetermined direction relative to a traveling direction as a course to be followed next from the position in the unit space corresponding to the current position; The first learning unit further provides a next position in the unit space reached by the moving object as an input to the neural network model, performs a calculation on the neural network model, and outputs a second estimated value of the action-value function; The first learning unit learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value. A route management device comprising:

6. A dividing step of dividing a route of the moving object from a starting point to a destination point via intermediate points into a plurality of partial routes; a first acquisition step of acquiring a current location of the mobile object from the mobile object via a network; a first learning step of applying a reward function to an estimation result of a course that the moving body should take sequentially from an initial point position constituting each of the plurality of partial paths until the moving body reaches a final point position, based on the current position of the moving body acquired in the first acquisition step, and updating the course so as to maximize a reward for the moving body to reach the final point position, and learning a course plan for the moving body to take sequentially from the current position using a reinforcement learning model; a second learning step of learning, using a supervised learning model, a relationship between the current position of the moving object and a course plan for the plurality of partial routes that the moving object should take from the current position in sequence, the course plan being obtained by learning in the first learning step; a storage step of storing the trained supervised learning model constructed in the second learning step in a storage unit; Equipped with The reward function has a variable representing the degree of arrival of the moving object at the final point. A route management method comprising:

7. 7. The route management method according to claim 6, The waypoints include a plurality of waypoints in a predetermined order, The plurality of partial routes include a route from the departure point to the first waypoint that the moving body passes through, a route between the waypoints that the moving body passes through in sequence, and a route from the last waypoint that the moving body passes through to the destination point. A route management method comprising:

8. 7. The route management method according to claim 6, The method further includes a setting step of setting the trained supervised learning model to the mobile body as control information for controlling a route of the mobile body from the departure point to the destination point via the intermediate points. A route management method comprising:

9. 7. The route management method according to claim 6, A moving space in which the moving object moves is defined as a unit space divided into a plurality of spaces, The course plan for the moving object to proceed from the current position is a course plan for the moving object to proceed from a position in the unit space corresponding to the current position. A route management method comprising:

10. 10. The route management method according to claim 9, The reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer; the first learning step provides a position in a unit space corresponding to the current position as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value of an action value function representing an expected value of a cumulative value of the future reward to be obtained when the moving body takes each action related to movement in a predetermined direction relative to a traveling direction as a course to be followed next from the position in the unit space corresponding to the current position, The first learning step further includes providing a next position in the unit space reached by the moving object as an input to the neural network model, performing a calculation on the neural network model, and outputting a second estimated value of the action-value function; The first learning step learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value. A route management method comprising:

11. A route management device according to any one of claims 1 to 5, The moving body and A route management system comprising: The moving body is A second acquisition unit configured to acquire the trained supervised learning model constructed by the route management device; A third acquisition unit configured to acquire a current location of the device; A calculation unit configured to provide the current position of the own device acquired by the third acquisition unit as an unknown input to the trained supervised learning model, perform calculations on the trained supervised learning model, and output a course plan for a course to be taken from the current position of the own device; a movement control unit configured to control movement of the device from the starting point to the destination point via the waypoints based on a course plan to be taken from the current position of the device output by the calculation unit; A route management system comprising:

Images