Route management device, route management method, and route management system

Abstract

The object is to manage the routes of a group of moving objects moving from different departure points to a common destination point with a simpler configuration. [Solution] The route management device 1 applies a reward function to the estimated results of calculating the sequential routes that the moving body 2 should take from its current position until it reaches the destination position, and updates the results so as to maximize the reward for the moving body 2 to reach the destination position.The route management device 1 is equipped with a first learning unit 11 that uses a reinforcement learning model to learn, for each moving body 2, the sequential route measures that the moving body 2 should take from its current position, and a second learning unit 12 that uses a supervised learning model to learn the relationship between the current position of the moving body 2 and the sequential route measures that the moving body 2 should take from its current position, obtained by learning for each moving body 2 by the first learning unit, and the current position includes the position of the starting point within the movement space, and the starting point positions for each moving body 2 include positions that are different from each other.

Description

[Technical Field]

[0001] The present invention relates to a route management device, a route management method, and a route management system. [Background technology]

[0002] BACKGROUND ART Conventionally, there has been known a technique for learning an optimal movement route for a moving object from a fixedly set starting point to a destination point using reinforcement learning (see Patent Document 1).

[0003] It is possible to learn a route for a group of mobile objects to move collectively from the same starting point to a destination point based on the route management technology for a single mobile object described in Patent Document 1. However, when the mobile objects constituting the group of mobile objects move collectively from different starting points to a common destination point, it is necessary to learn a route for each mobile object, which makes the system configuration complicated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-173643 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, with conventional technology, it has been difficult to manage the routes of a group of moving objects moving from different departure points to a common destination point with a simpler configuration.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to manage the routes of a group of moving objects moving from different departure points to a common destination point with a simpler configuration. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the route management device of the present invention is a route management device that manages the route of a group of moving objects to a destination position set in a moving space, and includes: a first acquisition unit configured to acquire the current position of each moving object constituting the group of moving objects; a first learning unit configured to apply a reward function to an estimated result of calculating the sequential path that the moving object should take from its current position to the destination position, and update the result so as to maximize the reward for the moving object to reach the destination position, and to learn, for each moving object, a course plan for the moving object to take sequentially from its current position using a reinforcement learning model; a second learning unit configured to learn, using a supervised learning model, the relationship between the current position of the moving object and the course plan for the moving object to take sequentially from its current position, obtained by learning for each moving object by the first learning unit; and a memory unit configured to store the learned supervised learning model constructed by the second learning unit, wherein the current position includes the position of a starting point in the moving space, and the starting point positions for each moving object include positions that are different from each other.

[0008] In addition, the route management device of the present invention may further include a setting unit configured to set the learned supervised learning model to each of the group of moving objects as control information for controlling the routes of the group of moving objects.

[0009] In addition, in the route management device of the present invention, the movement space may be defined by a unit space obtained by dividing the movement space into a plurality of spaces, and the course plan for the moving body to proceed sequentially from the current position may be a course plan for the unit space for the moving body to proceed sequentially from a position in the unit space corresponding to the current position.

[0010] In addition, in the route management device of the present invention, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the first learning unit provides a position in the unit space corresponding to the current position as input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action value function that represents an expected value of the cumulative value of the future reward to be obtained when the moving body takes each action related to movement in a predetermined direction relative to its direction of travel as the next course to be taken from the current position in the unit space, the first learning unit further provides a position in the unit space that the moving body will next reach as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action value function, and the first learning unit may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate.

[0011] In order to solve the above-mentioned problems, the route management method of the present invention is a route management method for managing the route of a group of moving objects to a destination position set in a mobile space, and includes: a first acquisition step for acquiring the current position of each moving object constituting the group of moving objects; a first learning step for applying a reward function to an estimated result of calculating the sequential route that the moving object should take from its current position to the destination position, and updating the result so as to maximize the reward for the moving object to reach the destination position, and learning a sequential route plan for the moving object from its current position using a reinforcement learning model for each moving object; a second learning step for learning, using a supervised learning model, the relationship between the current position of the moving object and the sequential route plan for the moving object that has been obtained by learning for each moving object in the first learning step; and a storage step for storing the learned supervised learning model constructed in the second learning step in a memory unit, wherein the current position includes the position of a starting point in the mobile space, and the starting point positions for each moving object include positions that are different from each other.

[0012] In addition, the route management method of the present invention may further include a setting step of setting the learned supervised learning model to each of the moving bodies as control information for controlling the routes of the group of moving bodies.

[0013] In addition, in the route management method of the present invention, the movement space may be defined by a unit space obtained by dividing the movement space into a plurality of spaces, and the course plan for the moving body to proceed sequentially from the current position may be a course plan for the unit space for the moving body to proceed sequentially from a position in the unit space corresponding to the current position.

[0014] Furthermore, in the route management method according to the present invention, the reinforcement learning model may be a neural network model including an input layer, a hidden layer, and an output layer, and the first learning step may provide a position in a unit space corresponding to the current position as an input to the neural network model, perform calculations on the neural network model, and output a first estimate of an action value function representing an expected value of the cumulative value of the future reward to be obtained when the moving body takes each action relating to movement in a predetermined direction relative to its direction of travel as a next course to be taken from the current position in the unit space, the first learning step may further provide a position in the unit space to be reached next by the moving body as an input to the neural network model, perform calculations on the neural network model, and output a second estimate of the action value function, and the first learning step may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate.

[0015] In order to solve the above-mentioned problems, the route management system of the present invention is a route management system comprising the above-mentioned route management device and the group of mobile objects, wherein each of the mobile objects constituting the group of mobile objects comprises a second acquisition unit configured to acquire the learned supervised learning model constructed by the route management device, a third acquisition unit configured to acquire the current position of the device itself, a calculation unit configured to provide the current position of the device itself acquired by the third acquisition unit as an unknown input to the learned supervised learning model, perform calculations on the learned supervised learning model, and output a course plan for the device to follow sequentially from its current position, and a movement control unit configured to control the movement of the device itself from the departure point to the destination point based on the course plan for the device to follow sequentially from its current position output by the calculation unit. [Effects of the Invention]

[0016] According to the present invention, the relationship between the current position of a moving object and the course plan for the moving object to follow from its current position, which is obtained through learning for each moving object by the first learning unit, is learned using a supervised learning model. As a result, the routes of a group of moving objects moving from different starting points to a common destination can be managed with a simpler configuration. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing the configuration of a route management system including a route management device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram for explaining the learning process by the first learning unit included in the route management device according to the present embodiment. [Figure 3] FIG. 3 is a block diagram showing the configuration of the first learning unit included in the route management device according to this embodiment. [Figure 4] FIG. 4 is a diagram for explaining the learning process by the second learning unit included in the route management device according to the present embodiment. [Figure 5]FIG. 5 is a block diagram showing the hardware configuration of the path management device according to this embodiment. [Figure 6] FIG. 6 is a block diagram showing the configuration of a mobile object provided in the route management system according to this embodiment. [Figure 7] FIG. 7 is a block diagram showing the hardware configuration of a mobile object provided in the route management system according to this embodiment. [Figure 8] FIG. 8 is a sequence diagram showing the operation of the route management system according to the present embodiment. [Figure 9] FIG. 9 is a flowchart showing the first learning process of the route management device according to the present embodiment. [Figure 10] FIG. 10 is a flowchart showing the first learning process of the route management device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to FIGS.

[0019] [Route management system configuration] First, with reference to FIG. 1, an overview of a route management system including a route management device 1 and a plurality of mobile objects 2 according to an embodiment of the present invention will be described.

[0020] The route management system according to this embodiment includes a route management device 1 and a group of mobile objects which are multiple mobile objects 2. The route management device 1 and the multiple mobile objects 2 are connected to each other so that they can communicate with each other via a wireless communication network NW that complies with a predetermined communication standard such as LTE / 4G, 5G, or 6G. The route management system manages routes that the multiple mobile objects 2 use to move together to a common destination set in a mobile space A. As shown in FIG. 1, the mobile space A in which the mobile objects 2 move is capable of communication using, for example, a 5G wireless communication system.

[0021] The mobile objects 2 include drones, unmanned aerial vehicles, and other flying objects capable of autonomous flight, as well as self-driving vehicles, mobile robots, ships, etc. Each mobile object 2 constituting the mobile object group starts from an arbitrary position in the mobile space A or a different position set in advance, and moves collectively from each starting point to the same destination position to perform a set task.

[0022] In the following, an example will be described in which the moving body 2 is a drone. The moving body 2 controls its autonomous flight using a flight controller that processes information from a sensor 208 (described later) and controls the rotation speed of a motor 209 and the angle of a propeller 210. The moving body 2 also acquires its own GPS position using a GPS receiver 207. The moving body 2 is also configured as an IoT terminal having an IP address. Each IP address can uniquely identify the moving body 2. Furthermore, in this embodiment, the moving body 2 controls its autonomous flight to a destination point based on control information stored in an auxiliary storage device 205. Each of the multiple moving bodies 2 has the same configuration, and the functional blocks and hardware configuration thereof will be described in detail below.

[0023] As shown in Figure 1, the mobile space A in which multiple mobile objects 2 move is a three-dimensional matrix-like space composed of multiple unit spaces divided into spaces. Each unit space that makes up the mobile space A has the same volume. Furthermore, each unit space has a node ID, and each unit space is represented by a single position (x, y, z). The location information can be three-dimensional GPS position coordinates consisting of latitude, longitude, and altitude. For example, a representative value such as the center position of the unit space can be used as the position of the unit space.

[0024] As shown in Fig. 1, n (n is an integer equal to or greater than 2) mobile bodies 2 move in a group from positions in the unit space corresponding to different starting positions S1, S2, ..., Sn, using each unit space as a waypoint, to the position in the unit space of the destination point G. The position of the destination point G of the mobile bodies 2 is set arbitrarily and is a common destination point for multiple mobile bodies 2.

[0025] The route management system according to this embodiment uses reinforcement learning to learn a course policy for each of multiple moving bodies 2 moving from different starting points to a destination, based on the current unit space. Furthermore, using the course policies for the multiple moving bodies 2 obtained through learning as training data, the system learns the relationship between the current unit space position of each moving body 2 and the course policy for each moving body 2. Furthermore, the system sets the learned supervised learning model to each moving body 2 as common control information for controlling the courses of the multiple moving bodies 2. Based on the set control information, each moving body 2 performs calculations using the learned supervised learning model, using its own current unit space position as an unknown input, and outputs a course policy for each moving body 2. Then, based on the output course policy for each moving body 2, the system determines a course from the starting point to the destination and controls movement to the destination.

[0026] Each moving object 2 for which control information has been set moves in the direction it should go for each unit space by changing its course as indicated by the arrows at the starting point S1 in FIG. 1 according to the course determined based on the course strategy obtained by the calculation of the trained supervised learning model. The course can include various courses, i.e., movement directions. In FIG. 1, the moving space A is illustrated as a two-dimensional plane, but the course of the moving object 2 can be a three-dimensional course. Here, the movement from a position in each unit space to a position in an adjacent unit space is referred to as a course, and the movement from the position of the starting point to the position of the destination point via multiple unit spaces is referred to as a route. The common control information set by the route management device 1 for each moving object 2 allows multiple moving objects 2 to collectively reach the unit space of the destination point G from the unit spaces of their respective different starting points.

[0027] [Route management device functional block] 1, the route management device 1 includes a first acquisition unit 10, a first learning unit 11, a second learning unit 12, a first memory unit (memory unit) 13, a second memory unit 14, and a setting unit 15. The route management device 1 manages routes of a group of moving objects to a destination position set in a movement space A.

[0028] The first acquisition unit 10 acquires the current position of each moving body 2 constituting the moving body group. Specifically, the first acquisition unit 10 acquires the position in the unit space where each of the multiple moving bodies 2 is currently located as the current position. Furthermore, the current position is the position in the unit space where the moving body 2 exists for each set time t, and includes the unit space position of the starting point. In this embodiment, the starting point positions of the moving bodies 2 include positions that are different from each other.

[0029] The first acquisition unit 10 acquires a position in the unit space corresponding to the GPS position of the moving object 2 for each time t as the current position of the moving object 2. Note that there are cases where a plurality of position coordinates are included in the unit space. In such cases, the first acquisition unit 10 can acquire the position in the unit space that is closest to the GPS position received by the GPS receiver 207 of the moving object 2 as the current position of the moving object 2. In this embodiment, the first acquisition unit 10 acquires the current positions of all moving objects 2 that constitute the moving object group.

[0030] The first learning unit 11 applies a reward function to the estimation result of calculating the sequential paths that the moving body 2 should take from its current position until it reaches the destination position, updates the results so as to maximize the reward for the moving body 2 to reach the destination position, and learns the path policy that the moving body 2 should take from its current position using a reinforcement learning model for each moving body 2. In this way, the first learning unit 11 performs reinforcement learning for each of the multiple moving bodies 2 each having a different starting point, and learns the path policy.

[0031] In this embodiment, as a course of action for the moving body 2 to move sequentially from the position of each unit space, actions a related to movement in predetermined n directions (n ​​is an integer of 2 or more) relative to the moving direction are set. n The traveling direction is a direction based on the position of the unit space where the moving object 2 was just before.

[0032] The first learning unit 11 uses a neural network model including an input layer s, a hidden layer h, and an output layer q as a reinforcement learning model, as shown in Fig. 2. In addition, as the neural network model, a state s t , and all the action value functions Q(s t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n We use Deep Q-Network (DQN), a neural network that outputs

[0033] More specifically, the first learning unit 11 provides the current position of the moving object 2 in the unit space as an input to the neural network model, performs calculations on the neural network model, and learns actions a related to each of the n directions as a course of action that the moving object 2 should take next from the current position in the unit space. n The first estimate Q1 of the action value function, which represents the expected cumulative value of future rewards obtained when

[0034] The reward is a state s indicating the current position of the mobile object 2, and an action a indicating that the mobile object 2 moves in a predetermined direction. n , and the next position of the moving object 2, i.e., the reward function r=r(s, a, s') for the next state s'. In this embodiment, the reward function includes as a variable the degree of arrival of the moving object 2 at a position in the unit space corresponding to the destination point. In addition, it can also include as a variable the degree of arrival at a position in the unit space corresponding to a space containing obstacles such as high-rise buildings or steel towers. For example, if the action of moving the moving object 2 in a predetermined direction brings it closer to the destination point or if it reaches the destination point in the shortest distance, the reward, which is a scalar quantity, is set to a larger value.

[0035] On the other hand, when the moving object 2 moves away from the destination point or reaches a unit space where an obstacle exists, a negative reward value (for example, r=-1) can be given. In this way, by setting the reward for a unit space where an obstacle exists as a negative value, the moving object 2 can reach the destination point while avoiding these points.

[0036] Furthermore, the first learning unit 11 provides the next position in the unit space reached by the moving object 2 as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function. The first learning unit 11 learns weight parameters of the neural network model so that the first estimated value Q1 becomes a target value calculated from the second estimated value Q2.

[0037] If the weight parameter of the neural network model is θ and the action value function is Q(s, a; θ), the minimization loss function for learning is given by the following equation (1). L(θ)=1 / 2{r+γmax a’ Q(s',a';θ)-Q(s,a;θ)} 2 ···(1)

[0038] In the above equation (1), r is the reward (immediate reward) and γ is the discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value at the state s' one step forward, that is, the second estimated value Q2. The target value is r+γmax a’ It is expressed as Q(s',a';θ).

[0039] The first learning unit 11 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above formula (1).

[0040] More specifically, the first learning unit 11 can employ a Fixed Target Q-Network that uses two neural networks, a main QN 111 and a target QN 113, as shown in FIG. 3. The main QN 111 selects the optimal action and updates the action-value function Q. Meanwhile, the target QN 113 estimates and evaluates the value of the action a' to be taken in the next state s' as a result of the action. The main QN 111 and the target QN 113 have neural networks with the same layer structure, but the parameter of the main QN 111 is "θ" and the parameter of the target QN 113 is "θ". - " is given by

[0041] The main QN 111 receives the current position of the moving object 2 as state s from the environment 110. The environment 110 is a system of movement space in which the moving object 2 is placed, and in this environment 110, the moving object 2 moves to another unit space by taking action a related to movement in a predetermined direction, and transitions to the next state s', while simultaneously obtaining a reward r from the environment 110.

[0042] The first learning unit 11 inputs the state s relating to the current position of the moving object 2 to the main QN 111 and calculates the action value function Q(s, a; θ). The first learning unit 11 calculates the action a using, for example, the ε-greedy method, or calculates the optimal action argmax at the current time. a Q(s, a; θ) is calculated. In the environment 110, the moving object 2 takes the action argmax a Q(s, a; θ). The environment 110 is such that the mobile unit 2 takes action argmax a As a result of performing Q(s, a; θ), the position in the unit space to which the robot has moved is observed as the next state s', and a reward r is output. The experience data 114 stores the experience (s, a, r, s') output from the environment 110.

[0043] The first learning unit 11 calculates a loss function L in a DQN loss calculation 112, and updates the weight of the main QN 111 with the gradient of the loss function L.

[0044] The first learning unit 11 periodically copies the weights of the main QN 111 to the target QN 113 for synchronization. The synchronization of the target QN 113 is performed less frequently than the update frequency of the weights of the main QN 111. The first learning unit 11 extracts experience from the experience data 114, inputs past states to the target QN 113, and calculates the estimated value max a’ Q(s',a';θ - The first learning unit 11 outputs the estimated value max a’ Q(s',a';θ - ) based on the target value r+γmax a’ Q(s',a';θ - ) is used to learn the weights of the main QN 111 in the DQN loss calculation 112.

[0045] The strategies for the paths that multiple moving bodies 2 with different starting points should take from their current positions to their destinations, i.e., the learned reinforcement learning models, obtained through learning by the first learning unit 11, are stored in the first storage unit 13. The learned reinforcement learning models constructed for each moving body 2 are used as training data for learning by the second learning unit 12.

[0046] The second learning unit 12 uses a supervised learning model to learn the relationship between the current position of the moving body 2 and the course plan that the moving body 2 should take sequentially from its current position, obtained by learning for each moving body 2 by the first learning unit 11.

[0047] FIG. 4 shows the structure of a neural network model employed as an example of a supervised learning model used by the second learning unit 12. The neural network model includes an input layer x, a hidden layer h, and an output layer y. The second learning unit 12 provides the current unit space position of the mobile object 2, i.e., the unit space position corresponding to the GPS position of the mobile object 2 at each time t, to the input layer of the neural network model, applies an activation function to the weighted sum of the input, and passes the output determined by threshold processing to the output layer. Each output node in the output layer outputs a predicted output of the model corresponding to n action value functions Q.

[0048] The second learning unit 12 introduces an objective function E shown in the following equation (2) to learn the parameters of the neural network model so that the course plan to be taken sequentially from the current position, which is the predicted value from the neural network model for the current position of the moving body 2, becomes the value of the course plan to be taken sequentially for each moving body 2 from its current position, obtained by learning for each moving body 2 by the first learning unit 11.

[0049]

number

[0050] In the above equation (2), y1, y2, , y n indicates the predicted output value of each output node. Also, Y1, Y2, , Y n is training data, and in this embodiment, n action-value functions Q(s) for the current position obtained by reinforcement learning by the first learning unit 11 are t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n ) Furthermore, all of the action value functions Q obtained by the trained reinforcement learning model, from which multiple moving objects 2 each having a different starting point reach a common destination point, are used as training data.

[0051] The value of the objective function E in the above equation (2) is the above input value x of the supervised learning model, and the output values ​​y1, y2, . . . , y for the position in the unit space corresponding to the GPS position of the mobile unit 2 at time t. n are the target outputs of the training data Y1,Y2,...,Y n The second learning unit 12 adjusts the weight parameters of the neural network associated with the supervised learning model so that the objective function E is minimized, i.e., becomes 0. The second learning unit 12 can optimize the objective function E using backpropagation or the like.

[0052] The first storage unit 13 stores trained reinforcement learning models constructed by adversarial learning by the first learning unit 11. A trained reinforcement learning model is constructed for each moving body 2 having a different starting point, and therefore, if n moving bodies 2 have different starting points, n trained reinforcement learning models will be stored.

[0053] The second storage unit 14 stores the trained supervised learning model constructed by the second learning unit 12 through supervised learning.

[0054] The setting unit 15 sets the trained supervised learning model as control information for controlling the paths of the group of moving objects to each moving object 2. For example, the setting unit 15 can transmit the control information to each moving object 2 via the network NW.

[0055] [Hardware configuration of the route management device] Next, an example of a hardware configuration for realizing the path management device 1 having the above-described functions will be described with reference to FIG.

[0056] 5, the path management device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output (I / O) 106 connected via a bus 101, and a program that controls these hardware resources. Furthermore, the path management device 1 can include a display device 107 connected via the bus 101.

[0057] The processor 102 is realized by a CPU, a GPU, an FPGA, an ASIC, or the like.

[0058] The main memory device 103 pre-stores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 implement the functions of the route management device 1, such as the first acquisition unit 10, the first learning unit 11, the second learning unit 12, and the setting unit 15 shown in FIG.

[0059] The communication interface 104 is an interface circuit for connecting the route management device 1 to various external electronic devices via a network.

[0060] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or flash memory as the storage medium.

[0061] The auxiliary storage device 105 has a program storage area for storing the route management program executed by the route management device 1. It also has a program storage area for storing the reinforcement learning program executed by the route management device 1. The auxiliary storage device 105 also has an area for storing the supervised learning program. The first storage unit 13 and the second storage unit 14 described in FIG. 1 are realized by the auxiliary storage device 105. The auxiliary storage device 105 also has an area for storing the destination points of each moving object 2 constituting the group of moving objects. The auxiliary storage device 105 also has an area for storing position coordinates of the movement space and position coordinates of the unit space. The auxiliary storage device 105 also has an area for storing identification information such as the IP address of the moving object 2. It may also have, for example, a backup area for backing up the above-mentioned data and programs.

[0062] The input / output I / O 106 is an input / output device that inputs signals from external devices and outputs signals to external devices.

[0063] The display device 107 is configured by an organic EL display, a liquid crystal display, etc. The display device 107 can display a map of the moving space, and the current position, progress, route, and destination position information of each moving object 2 constituting the moving object group.

[0064] [Functional blocks of mobile units] Next, the functional blocks of the moving object 2 will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the moving object 2. As described above, each of the multiple moving objects 2 is configured with the same functional blocks.

[0065] The moving body 2 includes a third storage unit 20, a second acquisition unit 21, a fourth storage unit 22, a third acquisition unit 23, a calculation unit 24, a determination unit 25, and a movement control unit 26. Each of the multiple moving bodies 2 determines the next route that the device should take based on the control information set by the route management device 1, and controls the flight of the device to the destination position. As described above, the departure points of the multiple moving bodies 2 include different points from each other.

[0066] The third storage unit 20 stores the control information set by the setting unit 15 of the route management device 1. The control information is a trained supervised learning model in which the second learning unit 12 of the route management device 1 has learned the course measures to be taken sequentially from the current position until the route reaches the same destination point from different starting points.

[0067] The second acquisition unit 21 acquires the control information set by the route management device 1. Specifically, the second acquisition unit 21 loads the control information stored in the third storage unit 20.

[0068] The fourth storage unit 22 stores map data including position coordinates of the moving space, and information associating position coordinates of unit spaces constituting the moving space with node IDs of the unit spaces. The fourth storage unit 22 also stores position information of destination points.

[0069] The third acquisition unit 23 acquires the current position of the device. More specifically, the third acquisition unit 23 acquires the position of the unit space in which the device is located at each time t based on the GPS position of the device. The third acquisition unit 23 refers to the fourth storage unit 22 and acquires the position in the unit space corresponding to the current GPS position received by the GPS receiver 207 as the current position of the device.

[0070] The calculation unit 24 provides the current position of the device acquired by the third acquisition unit 23 as an unknown input to the trained supervised learning model, performs calculations on the trained supervised learning model, and outputs a course plan for the device to follow sequentially from its current position.

[0071] The determination unit 25 determines the next course to be taken from the current position of the device itself, based on the course plan that the device itself should take from the current position, output by the calculation unit 24. More specifically, based on the course plan output by the calculation unit 24, the determination unit 25 determines the current position in the unit space as a state s t As a result, for each state s t The path to be taken is determined by selecting the path that is action a with the largest value of the action value function Q.

[0072] The movement control unit 26 controls the movement of the own device based on the plan of the course that the own device should take from its current position, output by the calculation unit 24. Specifically, the movement control unit 26 controls the movement of the own device based on the course that the own device should take next, output by the calculation unit 24 and determined by the determination unit 25. The movement control unit 26 can calculate a control command for the course that the own device should take next from its current position, and transmit the control command value to the motor 209. In this way, each of the multiple moving bodies 2 can be controlled in each state s t By selecting the action a with the largest value of the action value function Q, a group can move from different starting points to a common destination.

[0073] [Hardware configuration of mobile devices] Next, an example of a hardware configuration for realizing the moving object 2 having the above-described functions will be described with reference to FIG.

[0074] 7, the moving body 2 can be realized by, for example, a microcomputer including a processor 202, a main memory device 203, a communication interface 204, an auxiliary memory device 205, and an input / output (I / O) 206, all of which are connected via a bus 201, a program for controlling these hardware resources, a GPS receiver 207, a sensor 208, a motor 209, a propeller 210, and a battery 211. A flight controller that controls the autonomous flight of the moving body 2 is realized by a computer such as a microcomputer and the program. The moving body 2 can mount these components on a frame of a predetermined shape.

[0075] The main memory device 203 pre-stores programs for the processor 202 to control movement and perform calculations. The processor 202 and the main memory device 203 implement the functions of the moving object 2, such as the second acquisition unit 21, the calculation unit 24, the determination unit 25, and the movement control unit 26, shown in FIG.

[0076] The communication interface 204 is an interface circuit for connecting the mobile unit 2 and the route management device 1 via a network.

[0077] The auxiliary storage device 205 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 205 can use a semiconductor memory such as a hard disk or flash memory as the storage medium.

[0078] The auxiliary storage device 205 has a program storage area for storing a movement control program executed by the mobile object 2. The auxiliary storage device 205 also has an area for storing a calculation program for performing calculations on a trained supervised learning model. The auxiliary storage device 205 realizes the third storage unit 20 and the fourth storage unit 22 described in FIG. 6. The auxiliary storage device 205 also has an area for storing identification information such as the IP address of the mobile object 2. The auxiliary storage device 205 also has an area for storing an application that notifies the route management device 1 of the GPS position. Furthermore, the auxiliary storage device 205 may have, for example, a backup area for backing up the above-mentioned data and programs.

[0079] The input / output I / O 206 is an input / output device that inputs signals from external devices and outputs signals to external devices.

[0080] The GPS receiver 207 has an antenna for receiving GPS signals built in. The GPS receiver 207 realizes the third acquisition unit 23 in FIG.

[0081] The sensor 208 is composed of various sensors such as an altitude sensor, an attitude sensor, a camera, LiDAR, and RADAR. In addition to the GPS receiver 207, the altitude sensor realizes the third acquisition unit 23 in Fig. 6. Furthermore, the flight controller performs flight control of the moving object 2 based on the various sensor data measured by the sensor 208.

[0082] The motor 209 rotates by the rotation drive, and rotates the propeller 210 attached to the rotation shaft of the motor 209 .

[0083] The battery 211 is a lithium ion battery or the like, and supplies power to the components of the moving object 2.

[0084] The mobile object 2 may be provided with a SIM and may have an International Mobile Subscriber Identity (IMSI) of the SIM.

[0085] [Operation of the route management system] Next, the operation of the route monitoring system including the route management device 1 having the above-described configuration and multiple moving objects 2 will be described with reference to the sequence in Fig. 8. The multiple moving objects 2 are placed at different unit space positions within the moving space, and these positions are used as starting points.

[0086] The route management device 1 acquires setting information of a destination point to which the group of mobile objects move collectively (step S1). Next, the first acquisition unit 10 acquires the position of the unit space where the mobile object 2 is currently located as the current position of the mobile object 2 (step S2). For example, the first acquisition unit 10 can acquire the current position of each mobile object 2 by having each mobile object 2 notify its current GPS position via a push notification to the IP addresses of the target mobile objects 2.

[0087] The first acquisition unit 10 acquires the unit space position where each moving body 2 currently resides for each time t from all moving bodies 2 constituting the moving body group. As described above, each moving body 2 departs from a unit space position at a different starting point. Therefore, the unit space position where each moving body 2 currently resides at the initial time t is different from each other. The first acquisition unit 10 acquires the unit space position corresponding to the current GPS position received by the GPS receiver 207 of each moving body 2 as the current position of the moving body 2.

[0088] Next, the first learning unit 11 applies a reward function to the estimation result obtained in step S2, which calculates the sequential paths that the moving body 2 should take from its current position to the destination position, and updates the results so as to maximize the reward for the moving body 2 to reach the destination position, and learns a strategy for the sequential paths that the moving body 2 should take from its current position using a reinforcement learning model (first learning process) (step S3). In step S3, the learning process is performed for all moving bodies 2 with different starting points. The first learning process will be described in detail later.

[0089] Thereafter, the first storage unit 13 stores the trained reinforcement learning model obtained in step S3 (step S4). Next, the second learning unit 12 uses a supervised learning model to learn the relationship between the current position of the moving object 2 and the course measures for the moving object 2 to take sequentially from the current position, which are obtained in the first learning process for each moving object 2 in step S3 (second learning process) (step S5).

[0090] Specifically, the second learning unit 12 repeatedly adjusts and updates parameters such as weights and thresholds to determine the values ​​of these parameters so that the error between the predicted output value of the policy for the sequential course to be taken when the position in the unit space corresponding to the current GPS position of the mobile object 2, i.e., the current state, is given as an input value to the supervised learning model and the teacher data minimizes the objective function E in the above equation (2). In step S5, the second learning unit 12 can determine the parameters that minimize the objective function E by using an error backpropagation method or the like.

[0091] The training data used in step S5 is a course of action to be taken from the current position in the unit space, which is obtained by the trained reinforcement learning model constructed in the first learning process of step S3. More specifically, as shown in the schematic diagram of the neural network in Figure 4, n action-value functions Q for the current position are calculated based on the training data Y1 to Y n In step S5, all of the action value functions Q obtained by the trained reinforcement learning model in step S3, from when multiple moving objects 2 with different starting points reach their destination points, are used as training data.

[0092] Next, second storage unit 14 stores the trained supervised learning model constructed in step S5 (step S6). After that, setting unit 15 sets the trained supervised learning model as control information for each moving object 2 constituting the moving object group (step S7). In step S7, setting unit 15 can transmit the trained supervised learning model to each moving object 2 via network NW.

[0093] Thereafter, the second acquisition unit 21 of the mobile object 2 constituting the mobile object group acquires the trained supervised learning model constructed by the route management device 1 (step S8). The second acquisition unit 21 reads out the control information transmitted from the route management device 1 and stored in the third storage unit 20, i.e., the trained supervised learning model.

[0094] Next, the third acquisition unit 23 acquires the current unit space position of the own device as the current position (step S9). Specifically, the unit space position corresponding to the GPS position received by the GPS receiver 207 can be acquired as the current position of the own device. Next, the calculation unit 24 uses the control information acquired in step S8, provides the current unit space position of the own device acquired in step S9 as an unknown input, performs calculations on the trained supervised learning model, and outputs a course of action to be taken from the current unit space position (step S10). For example, in the movement space A of FIG. 1, if the starting point of the moving object 2 is S2, the position of the starting point S2 is input to the trained supervised learning model as the current position at time t=1, and outputs n action value functions Q to be taken from S2.

[0095] Next, the determination unit 25 determines the next course to take by selecting a course along which the action a having the largest value of the n action-value functions Q output in step S11 is taken (step S11). Next, the movement control unit 26 controls the movement of the device itself based on the next course to take determined in step S11 (step S12). More specifically, the movement control unit 26 can calculate a control command for the next course to take from the current position and transmit the control command value to the motor 209.

[0096] The moving object 2 is in the state s t(step S9), calculates the trained supervised learning model (step S10), determines the next course a to be taken at each time t (step S11), and performs movement control based on the determined course a (step S44). The process is repeated. Thereafter, when the moving body 2 reaches the unit space position of the destination point (step S13: YES), the process ends. In this way, each moving body 2 constituting the moving body group executes the processes from step S8 to step S13 using the trained supervised learning model, which is common control information, so that multiple moving bodies 2 with different starting points can move collectively to a common destination point and perform a predetermined task.

[0097] Next, the first learning process (step S3 in FIG. 8) by the route management device 1 will be described with reference to the flowcharts in FIGS. 9 and 10. First, steps S1 and S2 described in FIG. 8 are executed. After that, the first learning unit 11 provides the position in the unit space where the moving object 2 is currently located, which is the current state of the moving object 2 acquired in step S2, as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value Q1 of an action value function that represents an expected value of the cumulative value of future rewards that will be obtained when the moving object 2 takes each action related to movement in a predetermined direction relative to the traveling direction as the next course to be taken from the current position in the unit space (step S20).

[0098] Next, the first acquisition unit 10 acquires the unit space position of the moving object 2 at the next time t as the next state s' (step S21). The unit space position where the moving object 2 will next arrive is determined based on the GPS position of the moving object 2 acquired by the first acquisition unit 10 for each time step. Furthermore, the first learning unit 11 provides the unit space position where the moving object 2 will next arrive, acquired in step S21, as input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function (step S22).

[0099] Next, the first learning unit 11 calculates a target value from the second estimated value Q2 (step S23). Subsequently, the first learning unit 11 learns weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S24). Specifically, the first learning unit 11 updates the weight parameters of the neural network model so as to minimize the loss function of the above equation (1).

[0100] Thereafter, the processes from step S2 to step S24 are repeated until trained reinforcement learning models are constructed for all of the multiple moving bodies 2 each having a different starting point (step S25: NO). After that, when training has been completed for all of the moving bodies 2 (step S25: YES), the first storage unit 13 stores the trained reinforcement learning models obtained in step S24 (step S4).

[0101] Next, with reference to FIG. 10, the first learning process by the first learning unit 11 when a Fixed Target Q-Network using two neural networks, a main QN 111 and a target QN 113, is adopted will be described.

[0102] The processing from step S1 to step S2 is the same as the steps of the first learning processing described in Fig. 9. Thereafter, the first learning unit 11 provides the main QN 111 with the current position of the moving object 2 in the unit space acquired in step S2 as an input, performs calculations on the neural network model, outputs an action value function Q, and calculates the next course a to be taken (step S120).

[0103] Next, the first learning unit 11 returns the behavior of the moving body 2 along the path a obtained in step S120 to the environment 110, and obtains the next state s' of the moving body 2, that is, the position in the unit space to which the moving body 2 has advanced, and the reward r (step S121).

[0104] The first learning unit 11 stores the experience (s, a, r, a') obtained in step S121 in the experience data 114 (step S122). Next, the first learning unit 11 calculates the loss function L in the DQN loss calculation 112, and updates the weight of the main QN 111 with the gradient of the loss function L (step S123). The first learning unit 11 repeats the processes from step S120 to step S123 a set number of times.

[0105] Thereafter, the first learning unit 11 periodically copies the weights of the main QN 111 to the target QN 113 for synchronization (step S124). The synchronization of the target QN 113 is performed less frequently than the update frequency of the weights of the main QN 111. Next, the first learning unit 11 extracts experience from the experience data 114, inputs past states to the target QN 113, and calculates the estimated value max a’ Q(s',a';θ - ) is output (step S126).

[0106] Next, the first learning unit 11 calculates the estimated value max output by the target QN 113. a’ Q(s',a';θ - ) based on the target value r+γmax a’ Q(s',a';θ - ) (step S127). Next, the first learning unit 11 calculates the loss function L in the DQN loss calculation 112 using the target value calculated in step S127 (step S128). Next, the first learning unit 11 learns the weights of the main QN 111 so as to minimize the loss given by the loss function L (step S129). Thereafter, the first storage unit 13 stores the trained reinforcement learning models for all of the moving objects 2 having different starting points (step S4).

[0107] As described above, the route management device 1 according to this embodiment learns, by reinforcement learning, the optimal path plan for each of multiple moving objects 2 moving from different starting points to a common destination, and uses the path plan obtained by reinforcement learning as training data to learn the relationship between the current position of the moving object 2 and the path plan to be taken sequentially using a supervised learning model. Furthermore, the learned supervised learning model is set in each moving object 2 as control information common to the multiple moving objects 2. Therefore, the routes of a group of moving objects moving from different starting points to a common destination can be managed with a simpler configuration.

[0108] Furthermore, according to the route management system of this embodiment, common control information is set for a group of moving objects, thereby realizing moving objects 2 that can move collectively from different starting points to a common destination point. Therefore, the configuration of each moving object 2 can be simplified, and the moving objects 2 can perform movement control for group movement.

[0109] In the embodiment described above, the reinforcement learning model used by the first learning unit 11 is exemplified as a DQN related to a Fixed Target Q-Network configured as a multilayer neural network, but the reinforcement learning model can also be a CNN, a multilayer perceptron, etc. In addition to the DQN exemplified as a reinforcement learning model, Double DQN, Dueling DQN, Actor-Critic (AC) method, Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG), Q-learning, etc. can also be used.

[0110] In the embodiment described above, a multilayer neural network is used as the supervised learning model used by the second learning unit 12. However, the supervised learning model may be a multilayer perceptron, a decision tree-based model such as a random forest, or a support vector machine.

[0111] The above describes embodiments of the route management device, route management method, and route management system of the present invention, but the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can imagine are possible within the scope of the invention described in the claims. [Explanation of symbols]

[0112] 1...route management device, 10...first acquisition unit, 11...first learning unit, 12...second learning unit, 13...first memory unit, 14...second memory unit, 15...setting unit, 2...mobile body, 20...third memory unit, 21...second acquisition unit, 22...fourth memory unit, 23...third acquisition unit, 24...calculation unit, 25...determination unit, 26...mobility control unit, 101, 201...bus, 102, 202...processor, 103, 203...main memory device, 104, 204...communication interface, 105, 205...auxiliary memory device, 106, 206...input / output I / O, 107...memory device, 207...GPS receiver, 110...environment, 111...main QN, 112...DQN loss calculation, 113...target QN, 114...empirical data, NW...network.

Claims

1. A route management device that manages routes of a group of moving objects to a destination point set in a moving space, a first acquisition unit configured to acquire a current position of each of the moving objects constituting the group of moving objects; a first learning unit configured to apply a reward function to an estimation result obtained by calculating a course that the moving body should take sequentially from the current position to the destination position, update the course so as to maximize a reward for the moving body to reach the destination position, and learn, for each moving body, a course plan that the moving body should take sequentially from the current position using a reinforcement learning model; a second learning unit configured to learn, using a supervised learning model, a relationship between the current position of the moving object and a course plan for the moving object to move sequentially from the current position, the course plan being obtained through learning for each moving object by the first learning unit; a storage unit configured to store the trained supervised learning model constructed by the second learning unit; and Equipped with The current position includes a position of a starting point in the moving space, and the positions of the starting points for the moving bodies include positions different from each other. A route management device characterized by:

2. 2. The route management device according to claim 1, The system further includes a setting unit configured to set the trained supervised learning model to each of the moving objects as control information for controlling the paths of the group of moving objects. A route management device characterized by:

3. 2. The route management device according to claim 1, the movement space is defined by a unit space obtained by dividing the movement space into a plurality of spaces, The course plan for the moving body to proceed sequentially from the current position is a course plan for the moving body to proceed sequentially in unit space from a position in unit space corresponding to the current position. A route management device characterized by:

4. 4. The route management device according to claim 3, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer; the first learning unit provides a position in a unit space corresponding to the current position as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value of an action value function representing an expected value of a cumulative value of the future reward to be obtained when the moving object takes each action related to movement in a predetermined direction relative to a traveling direction as a course to be taken next from the position in the current unit space; the first learning unit further provides a next position in the unit space reached by the moving object as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value of the action-value function; The first learning unit learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value. A route management device characterized by:

5. A route management method for managing routes of a group of moving objects to a destination point set in a moving space, comprising: a first acquisition step of acquiring a current position of each of the moving objects constituting the group of moving objects; a first learning step of applying a reward function to an estimation result of calculating a course that the moving body should take sequentially from the current position to the destination position, updating the course so as to maximize a reward for the moving body to reach the destination position, and learning a course policy that the moving body should take sequentially from the current position using a reinforcement learning model for each moving body; a second learning step of learning, using a supervised learning model, a relationship between the current position of the moving object and a course plan for the moving object to move sequentially from the current position, which is obtained by learning for each moving object in the first learning step; a storage step of storing the trained supervised learning model constructed in the second learning step in a storage unit; Equipped with The current position includes a position of a starting point in the moving space, and the positions of the starting points for the moving bodies include positions different from each other. A route management method comprising:

6. 6. The route management method according to claim 5, The method further includes a setting step of setting the trained supervised learning model to each of the moving objects as control information for controlling the paths of the group of moving objects. A route management method comprising:

7. 6. The route management method according to claim 5, the movement space is defined by unit spaces obtained by dividing the movement space into a plurality of spaces, The course plan for the moving body to proceed sequentially from the current position is a course plan for the moving body to proceed sequentially in unit space from a position in unit space corresponding to the current position. A route management method comprising:

8. 8. The route management method according to claim 7, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer; the first learning step provides a position in a unit space corresponding to the current position as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value of an action value function representing an expected value of a cumulative value of the future reward to be obtained when the moving body takes each action relating to movement in a predetermined direction relative to a traveling direction as a next course to be taken from the current position in the unit space; The first learning step further includes providing a next position in the unit space reached by the moving object as an input to the neural network model, performing calculations on the neural network model, and outputting a second estimated value of the action-value function; The first learning step learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value. A route management method comprising:

9. A route management device according to any one of claims 1 to 4; the group of moving objects; A route management system comprising: Each of the moving bodies constituting the group of moving bodies is a second acquisition unit configured to acquire the trained supervised learning model constructed by the route management device; a third acquisition unit configured to acquire a current location of the device; a calculation unit configured to provide the current position of the host device acquired by the third acquisition unit as an unknown input to the trained supervised learning model, perform calculations on the trained supervised learning model, and output a course plan for the host device to proceed sequentially from the current position; a movement control unit configured to control movement of the device from the departure point to the destination point based on a course plan to be followed sequentially from the current position of the device output by the calculation unit; A route management system comprising:

Images