Control device, control method, and control system

Abstract

To control a group behavior while reducing the complexity of reward design.SOLUTION: A control system includes: a second learning unit 12 that performs adversarial learning of a generative model having a generator 121 that generates pseudo-policy data similar to true policy data, using a policy of a course that the first UAV2a should take sequentially from its current position obtained by learning by a first learning unit 11 as the true policy data, and a classifier 122 that distinguishes between the pseudo-policy data generated by the generator 121 and the true policy data; a generation unit 14 that generates the pseudo-policy data similar to the true policy data by using a learned generator 121' obtained by the adversarial learning of the second learning unit 12; and a setting unit 15 that sets information including the pseudo-policy data generated by the generation unit 14 to each moving body UAV2 as control information for controlling the courses of multiple UAV2s.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a control system, and more particularly to a control technique for a group of moving objects. [Background technology]

[0002] Conventionally, in systems aimed at single agents that control mobile objects such as drones, a technology has been known that employs Deep Q-Network (DQN) to guide a mobile object to a destination point using an action value function that represents the expected cumulative value of future rewards obtained when an action indicating the next course of action is taken based on the current position of the mobile object (see Patent Document 1).

[0003] Independent Learners (ILs), an algorithm in which each agent has an independent action value function and learns independently, is known as a method of applying single-agent reinforcement learning to multi-agent reinforcement learning (see Non-Patent Document 1). In ILs, when each agent learns independently using DQN, the policy changes and the environment becomes non-stationary, which can cause learning to become unstable and make it difficult to incorporate cooperative behavior across the entire system.

[0004] To address this issue, a cooperative multi-agent system is known that employs multi-agent reinforcement learning to learn the policy of each agent that maximizes a shared reward among the agents (see Non-Patent Document 2). However, since the reward design requires understanding the state of the entire system, if the agent's own behavior is not linked to the reward, the policy cannot be learned correctly. Therefore, when using a conventional cooperative multi-agent system to control the collective behavior of multiple mobile objects, it can be difficult to control the collective behavior. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7407329 [Non-patent literature]

[0006] [Non-Patent Document 1] Tan, Ming. "Multi-agent reinforcement learning: Independent vs. cooperative agents." Proceedings of the tenth international conference on machine learning. 1993. [Non-patent document 2] Gu, Shangding, et al. “Multi-agent constrained policy optimization.” arXiv preprint arXiv:2110.02793(2021). Summary of the Invention [Problem to be solved by the invention]

[0007] With conventional techniques, it has been difficult to control group behavior while reducing the complexity of reward design.

[0008] The present invention has been made to solve the above-mentioned problems, and aims to control collective behavior while reducing the complexity of reward design. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, a control device according to the present invention is a control device that controls a course of a group of moving objects to a destination position set in a moving space, and includes a first acquisition unit configured to acquire the current position of a first moving object selected from the group of moving objects, a first learning unit configured to apply a reward function to an estimation result obtained by calculating a course that the first moving object should take sequentially from the current position of the first moving object until it reaches the destination position, and update the course so as to maximize a reward for the first moving object to reach the destination position, and to learn a policy of the course that the first moving object should take sequentially from the current position using a reinforcement learning model, and a first learning unit configured to learn a policy of the course that the first moving object should take sequentially from the current position using a reinforcement learning model, and a first learning unit configured to learn a policy of the course that the first moving object should take sequentially from the current position using a reinforcement learning model, and a first learning unit configured to learn a policy of the course that the first moving object should take sequentially from the current position using a reinforcement learning model, and a first learning unit configured to learn a policy of the course that the first moving object should take sequentially from the current position using a reinforcement learning model, and a second ... the first moving body group comprises: a generator configured to generate pseudo policy data similar to the true policy data, using a policy of a course that the first moving body should take sequentially from the current position obtained by adversarial learning; and a second learning unit configured to perform adversarial learning of a generative model having a classifier that distinguishes between the pseudo policy data generated by the generator and the true policy data; a generation unit configured to generate the pseudo policy data similar to the true policy data, using the trained generator obtained by the adversarial learning of the second learning unit; and a setting unit configured to set information including the pseudo policy data generated by the generation unit to each moving body as control information for controlling the course of the moving bodies.

[0010] In addition, in the control device of the present invention, the movement space may be defined by a unit space obtained by dividing the movement space into a plurality of spaces, and the course plan for the first moving body to proceed sequentially from the current position may be a course plan for the unit space for the first moving body to proceed sequentially from a position in the unit space corresponding to the current position.

[0011] In addition, in the control device of the present invention, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the first learning unit provides the current unit space position as input to the neural network model, performs calculations on the neural network model, and outputs a first estimate of an action value function that represents an expected value of the cumulative value of the future reward to be obtained when the first moving body takes each action related to movement in a predetermined direction relative to its direction of travel as a next course to be taken from the current unit space position, the first learning unit further provides the next unit space position to be reached by the first moving body as input to the neural network model, performs calculations on the neural network model, and outputs a second estimate of the action value function, and the first learning unit may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate.

[0012] In order to solve the above-mentioned problems, a control method according to the present invention is a control method for controlling a course of a group of moving objects to a position of a destination point set in a moving space, the control method including: a first acquisition step of acquiring the current position of a first moving object selected from the group of moving objects; a first learning step of applying a reward function to an estimation result of a course that the first moving object should take sequentially from the current position of the first moving object until it reaches the position of the destination point, updating the course so as to maximize a reward for the first moving object to reach the position of the destination point, and learning a policy of the course that the first moving object should take sequentially from the current position using a reinforcement learning model; and The method includes a second learning step of performing adversarial learning of a generative model having a generator that generates pseudo-policy data similar to the true policy data, using the obtained policy of a course that the first moving body should take sequentially from the current position as true policy data, and a classifier that distinguishes between the pseudo-policy data generated by the generator and the true policy data; a generation step of generating the pseudo-policy data similar to the true policy data using the trained generator obtained by the adversarial learning in the second learning step; and a setting step of setting information including the pseudo-policy data generated in the generation step in each moving body as control information for controlling the course of the group of moving bodies.

[0013] Furthermore, in the control method according to the present invention, the movement space may be defined by a unit space obtained by dividing the movement space into a plurality of spaces, and the course plan for the first moving body to proceed sequentially from the current position may be a course plan for the unit space for the first moving body to proceed sequentially from a position in the unit space corresponding to the current position.

[0014] Furthermore, in the control method according to the present invention, the reinforcement learning model may be a neural network model including an input layer, a hidden layer, and an output layer, and the first learning step may provide the current unit space position as an input to the neural network model, perform calculations on the neural network model, and output a first estimate of an action value function representing an expected value of the cumulative value of the future reward to be obtained when the first moving body takes each action relating to movement in a predetermined direction relative to its direction of travel as a next course to be taken from the current unit space position to the neural network model, the first learning step may further provide the next unit space position to be reached by the first moving body as an input to the neural network model, perform calculations on the neural network model, and output a second estimate of the action value function, and the first learning step may learn weight parameters of the neural network model so that the first estimate becomes a target value calculated from the second estimate.

[0015] In order to solve the above-mentioned problems, the control system of the present invention is a control system comprising the above-mentioned control device and the group of moving bodies, wherein the group of moving bodies includes the first moving body and a plurality of second moving bodies, and each of the first moving body and the plurality of second moving bodies comprises a second acquisition unit configured to acquire the control information set by the control device, a third acquisition unit configured to acquire the current position of the device, a determination unit configured to determine a next course to take from the current position of the device acquired by the third acquisition unit based on a course plan for the device to take sequentially from its current position, as indicated by the control information, and a movement control unit configured to control the movement of the device based on the next course to take determined by the determination unit, wherein the control information includes the true course plan data and the pseudo course plan data, and the determination unit of the first moving body determines the next course to take based on the true course plan data, and the determination unit of each of the plurality of second moving bodies determines the next course to take based on the pseudo course plan data. [Effects of the Invention]

[0016] According to the present invention, a policy of a path that a first moving object should take sequentially from its current position, obtained through learning by a first learning unit, is used as true policy data, and adversarial learning of a generative model is performed, the generative model having a generator that generates pseudo-policy data similar to the true policy data and a classifier that distinguishes between the pseudo-policy data generated by the generator and the true policy data, and information including the generated pseudo-policy data is set to each moving object as control information for controlling the path of the group of moving objects. As a result, it is possible to control collective behavior while suppressing the complexity of reward design. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing the configuration of a control system including a control device and multiple UAVs according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the configuration of a UAV according to this embodiment. [Figure 3] FIG. 3 is a block diagram showing the hardware configuration of the UAV according to this embodiment. [Figure 4] FIG. 4 is a diagram for explaining the first learning unit included in the control device according to the present embodiment. [Figure 5] FIG. 5 is a block diagram showing the configuration of the first learning unit included in the control device according to this embodiment. [Figure 6] FIG. 6 is a block diagram showing the configuration of the second learning unit included in the control device according to this embodiment. [Figure 7] FIG. 7 is a diagram for explaining the second learning unit included in the control device according to the present embodiment. [Figure 8] FIG. 8 is a diagram for explaining the second learning unit included in the control device according to the present embodiment. [Figure 9] FIG. 9 is a block diagram showing the hardware configuration of the control device according to this embodiment. [Figure 10] FIG. 10 is a flowchart showing the operation of the control device according to this embodiment. [Figure 11]FIG. 11 is a flowchart showing the reinforcement learning process performed by the control device according to this embodiment. [Figure 12] FIG. 12 is a flowchart showing the reinforcement learning process performed by the control device according to this embodiment. [Figure 13] FIG. 13 is a flowchart showing the adversarial learning process performed by the control device according to this embodiment. [Figure 14] FIG. 14 is a flowchart showing the operation of the UAV according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to FIGS.

[0019] [Control system configuration] First, with reference to FIG. 1, an overview of a control system including a control device 1 and a plurality of UAVs 2 according to an embodiment of the present invention will be described.

[0020] The control system according to this embodiment includes a control device 1 and multiple UAVs 2 (a group of moving objects), and the control device 1 and the multiple UAVs 2 are connected to each other so that they can communicate with each other via a wireless communication network NW that conforms to a predetermined communication standard such as LTE / 4G, 5G, or 6G. The control system controls the path of the multiple UAVs 2 to fly (move) together to a set destination position in a moving space A. As shown in FIG. 1, the moving space A in which the UAVs 2 fly is capable of communication using, for example, a 5G wireless communication system.

[0021] The UAV2 is a mobile object such as a drone or unmanned aerial vehicle, and is capable of autonomous flight. The autonomous flight of the UAV2 is controlled by a flight controller that processes information from a sensor 208 (described later) and controls the rotation speed of a motor 209 and the angle of a propeller 210. In this embodiment, the UAV2 controls its autonomous flight to a destination point based on route control information stored in an auxiliary storage device 205. Each of the multiple UAV2 has the same configuration, and when there is no need to distinguish between them, they are collectively referred to as UAV2. The functional blocks and hardware configuration of the UAV2 will be described in detail below.

[0022] In this embodiment, among the multiple UAVs 2, the UAV 2 that flies ahead in the movement space A is selected as the first UAV 2a (first moving body). The first UAV 2a can be randomly selected from the UAVs 2 that fly ahead. Furthermore, among the multiple UAVs 2, those other than the first UAV 2a are referred to as the second UAV 2b (second moving body).

[0023] As shown in Figure 1, the mobile space A in which multiple UAVs 2 fly is a three-dimensional matrix-like space composed of unit spaces divided into multiple spaces.

[0024] Each unit space that makes up the mobile space A has the same volume. Furthermore, each unit space has a node ID, and each unit space is represented by a single position (x, y, z). The position information can be three-dimensional GPS position coordinates consisting of latitude, longitude, and altitude. For example, a representative value such as the center position of the unit space can be used as the position of the unit space.

[0025] Also, as shown in Figure 1, multiple UAVs 2 use each unit space as a waypoint to move from a position in the unit space corresponding to the current position S of each UAV 2 to the unit space of the destination point G. The position of the destination point G of the UAV 2 is set arbitrarily and is a common destination point for multiple UAVs 2. For example, it is conceivable that multiple UAVs 2 fly in a group to capture an image of the destination point G.

[0026] The control system according to this embodiment uses reinforcement learning to learn a course policy for a first UAV 2a, selected from among multiple UAVs 2 and flying ahead, to follow sequentially from the current unit space. Furthermore, a generator is constructed using adversarial learning to generate pseudo-policy data similar to the true policy data, using the course policy for the first UAV 2a obtained through learning as true policy data. The control system sets the pseudo-policy data generated by the trained generator 121′ as course control information (control information) for each second UAV 2b, and sets the true policy data for the first UAV 2a as course control information for the first UAV 2a. Furthermore, each UAV 2 determines a course to a destination point based on the course control information and performs flight control.

[0027] Each UAV2 for which route control information has been set changes its route by maneuvering itself, as shown by the arrows in Figure 1, based on the route strategy indicated by the route control information, and moves in the direction it should go for each unit space. The route indicated by the route control information can include various routes, i.e., movement directions. In Figure 1, the movement space A is illustrated as a two-dimensional plane, but the route of the UAV2 can be a three-dimensional route. The route control information set for each UAV2 by the control device 1 allows multiple UAV2 to arrive collectively at the unit space of the destination point G.

[0028] [Controller function block] As shown in FIG. 1, the control device 1 includes an acquisition unit 10 (first acquisition unit), a first learning unit 11, a second learning unit 12, a storage unit 13, a generation unit 14, and a setting unit 15.

[0029] The acquisition unit 10 acquires the current position of a first UAV2a selected from among a plurality of UAVs 2. Specifically, the acquisition unit 10 acquires the position of the unit space in which the first UAV2a is currently located as the current position of the UAV2. The acquisition unit 10 acquires the position of the unit space in which the first UAV2a is currently located via the wireless communication network NW. In addition, in this embodiment, the acquisition unit 10 acquires the current position of the first UAV2a for each set time step. As described above, the first UAV2a is selected as a representative from among a plurality of UAVs 2; for example, a UAV2 flying ahead of the other UAVs 2 is selected as the first UAV2a.

[0030] More specifically, the acquisition unit 10 can acquire the position in the unit space corresponding to the GPS position received by the GPS receiver 207 of the first UAV 2a as the current position of the first UAV 2a. Note that the unit space may contain multiple position coordinates. In such a case, the acquisition unit 10 can acquire the position in the unit space that is closest to the GPS position received by the GPS receiver 207 of the first UAV 2a as the current position of the first UAV 2a.

[0031] The first learning unit 11 applies a reward function to the estimated result of calculating the course that the first UAV2a should take from its current position to the destination position, updates it so as to maximize the reward for the first UAV2a to reach the destination position, and learns the course policy that the first UAV2a should take from its current position using a reinforcement learning model. The first learning unit 11 passes the course policy that the first UAV2a should take from its current position obtained by learning to the second learning unit 12.

[0032] In this embodiment, as a course of action that the first UAV 2a should take from the position of each unit space, an action a related to movement in a predetermined n (n is a positive integer) directions relative to the direction of travel is n The direction of travel is based on the position of the unit space where UAV2 was just before.

[0033] The first learning unit 11 uses a neural network model including an input layer, a hidden layer, and an output layer as a reinforcement learning model, as shown in Fig. 4. In addition, as the neural network model, a state s t , and all action values ​​Q(s t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n The first learning unit 11 employs a Deep Q-Network (DQN), which is a neural network that outputs a CNN.

[0034] More specifically, the first learning unit 11 provides the current position of the unit space, which indicates the current position of the first UAV 2a, as an input to the neural network model, performs calculations on the neural network model, and learns actions a related to each movement in n directions as a course to be taken by the first UAV 2a from the current position in the unit space. n The first estimate Q1 of the action value function, which represents the expected cumulative value of future rewards obtained when

[0035] The reward is a state s indicating the current position of the first UAV 2a, an action a indicating that the first UAV 2a moves in a predetermined direction, and n , and the reward function r=r(s, a, s') for the next position of the first UAV2a, i.e., the next state s'. In this embodiment, the reward function includes as a variable the degree of arrival of the first UAV2a at a position in the unit space corresponding to the destination point. In addition, it can also include as a variable the degree of arrival at a position in the unit space corresponding to a space containing obstacles such as high-rise buildings or steel towers. For example, if the action of moving the first UAV2a in a predetermined direction brings it closer to the destination point or if it reaches the destination point in the shortest distance, the reward, which is a scalar quantity, is set to a larger value.

[0036] On the other hand, if the first UAV 2a moves away from the destination point or reaches a unit space where an obstacle exists, a negative reward value (for example, r = -1) can be given. In this way, by setting the reward for a unit space where an obstacle exists as a negative value, the first UAV 2a can reach the destination point while avoiding these points.

[0037] Furthermore, the first learning unit 11 provides the position in the unit space where the first UAV 2a next arrives as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function. The first learning unit 11 learns weight parameters of the neural network model so that the first estimated value Q1 becomes a target value calculated from the second estimated value Q2.

[0038] If the weight parameter of the neural network model is θ and the action value function is Q(s, a; θ), the minimization loss function for learning is given by the following equation (1). L(θ)=1 / 2{r+γmax a’ Q(s',a';θ)-Q(s,a;θ)} 2 ···(1)

[0039] In the above equation (1), r is the reward (immediate reward) and γ is the discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value at the state s' one step forward, that is, the second estimated value Q2. The target value is r+γmax a’ It is expressed as Q(s',a';θ).

[0040] The first learning unit 11 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above formula (1).

[0041] More specifically, the first learning unit 11 can employ a Fixed Target Q-Network that uses two neural networks, a main QN 111 and a target QN 113, as shown in FIG. 5. The main QN 111 selects the optimal action and updates the action-value function Q. Meanwhile, the target QN 113 estimates and evaluates the value of the action a' to be taken in the next state s' as a result of the action. The main QN 111 and the target QN 113 have neural networks with the same layer structure, but the parameter of the main QN 111 is "θ" and the parameter of the target QN 113 is "θ" - " is given by

[0042] The main QN 111 receives the current position of the first UAV 2a as state s from the environment 110. The environment 110 is a system of moving space A in which the first UAV 2a is placed, and in this environment 110, the first UAV 2a moves to another unit space by taking action a related to movement in a predetermined direction, transitioning to the next state s' and simultaneously obtaining a reward r from the environment 110.

[0043] The first learning unit 11 inputs the state s relating to the current position of the first UAV 2a to the main QN 111 and calculates the action value function Q(s, a; θ). The first learning unit 11 calculates the action a using, for example, the ε-greedy method, or calculates the optimal action argmax at the current time. a Q(s, a; θ) is calculated. In the environment 110, the first UAV 2a calculates the action argmax related to the optimal course at the current time. a Q(s, a; θ). The environment 110 is such that the first UAV 2a takes action argmax a As a result of performing Q(s, a; θ), the position in the unit space to which the robot has moved is observed as the next state s', and a reward r is output. The experience data 114 stores the experience (s, a, r, s') output from the environment 110.

[0044] The first learning unit 11 calculates a loss function L in a DQN loss calculation 112, and updates the weight of the main QN 111 with the gradient of the loss function L.

[0045] The first learning unit 11 periodically copies the weights of the main QN 111 to the target QN 113 for synchronization. The synchronization of the target QN 113 is performed less frequently than the update frequency of the weights of the main QN 111. The first learning unit 11 extracts experience from the experience data 114, inputs past states to the target QN 113, and calculates the estimated value max a’ Q(s',a';θ - The first learning unit 11 outputs the estimated value max a’ Q(s',a';θ - ) based on the target value r+γmax a’ Q(s',a';θ - ) is used to learn the weights of the main QN 111 in the DQN loss calculation 112.

[0046] The trained DQN constructed by the first training unit 11 is a state s t Based on this, the action value Q(s t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n ) can be output. At this time, actions a1 to a n In this embodiment, the action with the highest probability value among all the action values ​​Q(s) output from the trained DQN is selected as the policy. t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n ) is called policy data.

[0047] Returning to Figure 1, the second learning unit 12 performs adversarial learning of a generative model having a generator 121 that generates pseudo-policy data similar to the true policy data, which is the policy of the path that the first UAV 2a should take sequentially from its current position obtained by learning by the first learning unit 11, and a classifier 122 that distinguishes between the pseudo-policy data generated by the generator 121 and the true policy data.

[0048] As shown in Fig. 6, the second learning unit 12 adversarially learns a GAN (Generative Adversarial Network) having a generator 121 and a classifier 122. A trained generator 121' is constructed by the learning of the second learning unit 12. In this manner, in this embodiment, the policy for the path of the first UAV 2a learned by reinforcement learning by the first learning unit 11 is used as training data to learn the generator 121 that generates pseudo-policy data similar to the policy for the path of the first UAV 2a.

[0049] 7 and 8 are diagrams schematically illustrating the neural network configurations of the generator 121 and the classifier 122 of the GAN used by the second learning unit 12. As shown in FIG. 7, the generator 121 is configured as a neural network having an input layer, a hidden layer, and an output layer. The generator 121 is a model that generates pseudo-policy data from random noise. For example, m randomly sampled Gaussian noise vectors (z1 to z m ).

[0050] The generator 121 outputs the output G(z) after performing a product-sum operation on the input and weight parameters and threshold processing using an activation function. The output G(z) from the generator 121 is the position of the first UAV 2a, that is, the state s t The output from the trained DQN, i.e., all action values ​​Q(s t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,an ) is pseudo-policy data similar to the above. CNN or ResNet can be used as the neural network that configures the generator 121.

[0051] The classifier 122 shown in Fig. 8 is configured with a neural network having an input layer, a hidden layer, and an output layer. In the example of Fig. 8, the training data input is a policy x for the path of the first UAV a obtained by learning in the first learning unit 11. The training data for the true policy data is the position of the first UAV 2 a, which is the state s t The output from the trained DQN, i.e., all action values ​​Q(s t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n )

[0052] The classifier 122 outputs a binary value of 1 or 0 after performing a product-sum operation on the input and weight parameters and threshold processing using an activation function. The classifier 122 outputs an output y=1 when it correctly identifies the training data related to the input true policy data as true policy data. On the other hand, it outputs an output y=0 when it correctly identifies the training data related to the input pseudo policy data as pseudo policy data. In this way, the classifier 122 is a model that distinguishes the model distribution generated by the generator 121 from the data distribution of the training data, which is the true distribution. A CNN can be used as the neural network that constitutes the classifier 122.

[0053] FIG. 6 is a block diagram for explaining adversarial learning of GAN by the second learning unit 12. The generator 121 of the GAN adopted by the second learning unit 12 is represented as a function G, and the discriminator 122 is represented as a function D. Furthermore, the true policy data is represented as x, the predicted value output by the discriminator 122 is represented as y, and the correct label is represented as t. The correct label t is set to 1 for the true policy data and 0 for the pseudo policy data generated by the generator 121. In this case, the discriminator 122 calculates the cross entropy E CE It can be expressed as:

[0054]

number

[0055] The first term in the brace of the above equation (2) represents t n lny n In this case, the predicted value y n is the correct label of the true policy data, t n = 1. On the other hand, the second term in the braces represents (1-t n )ln(1-y n ), the predicted value y n is the correct label value (1-t n ) = 0. In this way, the cross entropy E CE is the maximum value when the predicted value matches the correct label value.

[0056] Here, the generator 121 that constitutes the GAN has parameters w G ,θ G and the function G(w G ,θ G ) The classifier 122 also uses the parameter w D ,θ D and function D(w D ,θ D ) The cross entropy E in the above equation (2) CE The objective function E of the GAN including the generator 121 and the discriminator 122 based on the above can be expressed by the following equation (3).

number

[0057] The first term in the above equation (3) represents E D(x)=1 lnD(w D ,θ D ) is the expected value that the classifier 122 classifies the true policy data as the true policy data. D(x)=0 ln(1-D(G(w G ,θ G ),w D ,θ D )) is the expected value at which the classifier 122 identifies the pseudo policy data generated by the generator 121 as pseudo policy data. In GAN training, the generator 121 and the classifier 122 are trained adversarially through min-max optimization of the objective function E. Therefore, the generator 121 is trained to generate pseudo policy data that can fool the classifier 122, and the classifier 122 is trained to identify the pseudo policy data generated by the generator 121 as pseudo policy data.

[0058] In learning, when true policy data is given, the classifier 122 maximizes the first term of the objective function E in the above equation (3) by outputting an output close to y=1. On the other hand, when pseudo policy data is given, the classifier 122 maximizes the second term of the objective function E by outputting an output close to y=0.

[0059] In the learning of the generator 121, D(G(w G ,θ G ),w D ,θ D ) (D(G(z)) in Figure 6) is close to 1. G ,θ G ) (G(z) in FIG. 6 ) to minimize the objective function E. The second learning unit 12 uses a learning procedure that alternately updates the parameters of the generator 121 and the classifier 122. Details of the learning procedure of the generator 121 and the classifier 122 by the second learning unit 12 will be described later.

[0060] When the objective function E of the GAN is optimized, the second learning unit 12 passes the trained generator 121′ to the generation unit 14.

[0061] The storage unit 13 stores the parameters of the trained reinforcement learning model and the trained generator 121′.

[0062] The generation unit 14 uses a trained generator 121′ constructed by GAN learning to generate pseudo-policy data similar to the policy data of the path of the first UAV 2a.

[0063] The setting unit 15 sets information including the pseudo-strategy data generated by the generation unit 14 as route control information for controlling the routes of the multiple UAVs 2 to each UAV 2. More specifically, the setting unit 15 sets the pseudo-strategy data generated by the generation unit 14 as route control information for the second UAV 2b among the multiple UAVs 2. Furthermore, the setting unit 15 sets the true strategy data obtained by learning by the first learning unit 11 as route control information for the first UAV 2a. The setting unit 15 transmits the route control information to each UAV 2 via, for example, a wireless communication network NW, and each UAV 2 that receives the route control information loads the route control information. Alternatively, the setting unit 15 can be configured to download the route control information to each UAV 2 offline.

[0064] [Control device hardware configuration] Next, an example of a hardware configuration for realizing the control device 1 having the above-described functions will be described with reference to FIG.

[0065] As shown in FIG. 9, the control device 1 can be realized, for example, by a computer having a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output (I / O) 106 connected via a bus 101, and a program that controls these hardware resources.

[0066] The processor 102 is realized by a CPU, a GPU, an FPGA, an ASIC, or the like.

[0067] The main memory device 103 pre-stores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 implement the functions of the control device 1, such as the acquisition unit 10, the first learning unit 11, the second learning unit 12, the generation unit 14, and the setting unit 15 shown in FIG.

[0068] The communication interface 104 is an interface circuit for connecting the control device 1 to various external electronic devices via a network.

[0069] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or flash memory as the storage medium.

[0070] The auxiliary storage device 105 has a program storage area for storing the control program executed by the control device 1. The auxiliary storage device 105 also has an area for storing a first learning program for performing reinforcement learning using DQN and a second learning program for performing adversarial learning using GAN. The auxiliary storage device 105 realizes the memory unit 13 described in FIG. 1. The auxiliary storage device 105 also has an area for storing map data including the position coordinates of the movement space A, as well as the position coordinates and node IDs of the unit spaces that make up the movement space A, in association with each other. The auxiliary storage device 105 also has an area for storing identification information of the UAV 2. Furthermore, the auxiliary storage device 105 may have, for example, a backup area for backing up the above-mentioned data, programs, etc.

[0071] The input / output I / O 106 is an input / output device that inputs signals from external devices and outputs signals to external devices.

[0072] The display device 107 is configured by an organic EL display, a liquid crystal display, etc. The display device 107 can display a map display of the moving space A, the current position of the UAV 2, the traveling route, and the position information of the destination point.

[0073] [UAV functional blocks] Next, the functional blocks of the UAV 2 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the UAV 2. As described above, the multiple UAVs 2 include a first UAV 2a selected by the control device 1 and multiple second UAVs 2b other than the first UAV 2a, and each UAV 2 is configured with the same functional blocks.

[0074] The UAV 2 includes a control information storage unit 20, a control information acquisition unit (second acquisition unit) 21, a flight information storage unit 22, a position information acquisition unit (third acquisition unit) 23, a determination unit 24, and a flight control unit (movement control unit) 25. Each of the multiple UAVs 2 determines the next course that the UAV should take based on the course control information set by the control device 1, and controls the flight of the UAV to the destination position.

[0075] The control information storage unit 20 stores the route control information set by the setting unit 15 of the control device 1. The route control information includes true policy data obtained by reinforcement learning by the first learning unit 11 of the control device 1 and pseudo-policy data generated by the generation unit 14 based on adversarial learning by the second learning unit 12 of the control device 1. More specifically, the control information storage unit 20 of the first UAV 2a stores the true policy data as route control information. Meanwhile, the control information storage unit 20 of each of the multiple second UAVs 2b stores the pseudo-policy data as route control information.

[0076] The control information acquisition unit 21 acquires the route control information set by the control device 1. Specifically, the control information acquisition unit 21 loads the route control information stored in the control information storage unit 20. The control information acquisition unit 21 of the first UAV 2a acquires true strategy data, and the control information acquisition unit 21 of each of the multiple second UAVs 2b acquires pseudo strategy data.

[0077] The flight information storage unit 22 stores map data including the position coordinates of the movement space A, and information associating the position coordinates of the unit spaces constituting the movement space A with the node IDs of the unit spaces. The flight information storage unit 22 also stores the position information of the destination point.

[0078] The position information acquisition unit 23 acquires the current position of the aircraft. More specifically, the position information acquisition unit 23 acquires the position of the unit space in which the aircraft is currently located for each time step based on the GPS position of the aircraft. The position information acquisition unit 23 can refer to the flight information storage unit 22 and acquire the position in the unit space corresponding to the current GPS position received by the GPS receiver 207 as the current position of the aircraft.

[0079] The determination unit 24 determines the next route to be taken from the current position of the aircraft acquired by the position information acquisition unit 23, based on the course plan for the aircraft to take from its current position, as indicated by the course control information.

[0080] More specifically, the decision unit 24 of the first UAV 2a determines the current position in the unit space based on the route control information of the true strategy data. t As a result, for each state s t In this case, the route to be taken is determined by selecting the route that is the action a with the largest value of the action value function Q. Similarly, the determination unit 24 provided in each of the second UAVs 2b determines the route to be taken in each state s based on the route control information of the pseudo-strategy data. t The path to be taken is determined by selecting the path that is action a with the largest value of the action value function Q.

[0081] More specifically, the decision unit 24 of the first UAV 2a performs a trained DQN calculation corresponding to the route control information of the true policy data, inputs the current unit space position to the trained neural network model, performs calculations on the trained neural network model, and determines the optimal action among actions related to movement in a predetermined n direction relative to the direction of travel of the first UAV 2a as the next route to take. Similarly, the decision unit 24 of the second UAV 2b performs a DQN calculation corresponding to the route control information of the pseudo policy data to determine the route.

[0082] The flight control unit 25 controls the flight of the aircraft based on the next course determined by the determination unit 24. The flight control unit 25 can calculate a control command for the next course from the current position and transmit the control command value to the motor 209. In this way, each of the multiple UAVs 2 including the first UAV 2a and the second UAV 2b can determine each state s based on the course control information. t By selecting the action a with the largest value of the action value function Q, the group can fly to the destination.

[0083] [UAV hardware configuration] Next, an example of a hardware configuration for realizing the UAV2 having the above-described functions will be described with reference to FIG.

[0084] 3, the UAV 2 can be realized by, for example, a microcomputer including a processor 202, a main memory device 203, a communication interface 204, an auxiliary memory device 205, and an input / output (I / O) 206, all connected via a bus 201, a program for controlling these hardware resources, a GPS receiver 207, a sensor 208, a motor 209, a propeller 210, and a battery 211. A flight controller that controls the autonomous flight of the UAV 2 is realized by a computer such as a microcomputer and the program. The UAV 2 can mount these components on a frame of a predetermined shape.

[0085] The main memory device 203 pre-stores programs for the processor 202 to perform flight control and calculations. The processor 202 and the main memory device 203 implement the functions of the UAV 2, such as the control information acquisition unit 21, the decision unit 24, and the flight control unit 25, shown in FIG.

[0086] The communication interface 204 is an interface circuit for network connection between the UAV 2 and the control device 1.

[0087] The auxiliary storage device 205 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 205 can use a semiconductor memory such as a hard disk or flash memory as the storage medium.

[0088] The auxiliary storage device 205 has a program storage area for storing a flight control program executed by the UAV 2. The auxiliary storage device 205 also has an area for storing a calculation program for determining a route based on route control information. The auxiliary storage device 205 realizes the control information storage unit 20 and the flight information storage unit 22 described in FIG. 2. The auxiliary storage device 205 also has an area for storing identification information for the UAV 2. Furthermore, the auxiliary storage device 205 may have, for example, a backup area for backing up the above-mentioned data and programs.

[0089] The input / output I / O 206 is an input / output device that inputs signals from external devices and outputs signals to external devices.

[0090] The GPS receiver 207 has an antenna for receiving GPS signals built in. The GPS receiver 207 realizes the position information acquisition unit 23 in FIG.

[0091] The sensor 208 is composed of various sensors such as an altitude sensor, an attitude sensor, a camera, LiDAR, and RADAR. In addition to the GPS receiver 207, the altitude sensor realizes the position information acquisition unit 23 in Figure 2. Furthermore, based on the various sensor data measured by the sensor 208, the flight controller performs flight control of the UAV 2.

[0092] The motor 209 rotates by the rotation drive, and rotates the propeller 210 attached to the rotation shaft of the motor 209 .

[0093] The battery 211 is a lithium ion battery or the like, and supplies power to the components of the UAV2.

[0094] The UAV 2 may include a SIM and have the SIM's International Mobile Subscriber Identity (IMSI). The UAV 2 may also be configured as an IoT terminal with an IP address. These IMSI and IP address may be used as identification information unique to the UAV 2.

[0095] [Control device operation] Next, the operation of the control device 1 having the above-described configuration will be described with reference to the flowcharts of FIGS.

[0096] First, the overall operation of the control device 1 will be described with reference to Fig. 10. First, the control device 1 sets a destination point to which a plurality of UAVs 2 are to move collectively in the movement space A (step S1).

[0097] Next, the first learning unit 11 performs a reinforcement learning process (step S2). After that, the second learning unit 12 performs an adversarial learning process (step S3) using the policy of the course of the unit space that the first UAV 2a should move sequentially from the position of the unit space corresponding to the current position obtained in step S2 as training data.

[0098] Next, the generator 14 generates pseudo-policy data similar to the policy of the path of the unit space that the first UAV 2a should take sequentially from the position of each unit space, using the trained generator 121' constructed by the adversarial learning in step S3 (step S4). After that, the setting unit 15 sets information including the pseudo-policy data generated in step S4 to each UAV 2 as the path control information of the multiple UAVs 2 (step S5). More specifically, the setting unit 15 sets the path control information of the pseudo-policy data to each of the second UAVs 2b. The setting unit 15 sets the path control information of the true policy data to the first UAV 2a.

[0099] Next, the reinforcement learning process by the first learning unit 11 of the control device 1 will be described with reference to Figures 11 and 12. First, the acquisition unit 10 acquires the position of the unit space where the first UAV 2a is currently located as the current position of the first UAV 2a (step S20). Specifically, the acquisition unit 10 acquires the position of the unit space where the first UAV 2a is currently located for each time step based on the GPS position of the first UAV 2a received via the wireless communication network NW. Furthermore, the acquisition unit 10 can refer to the memory unit 13 and acquire the position of the unit space corresponding to the current GPS position received by the GPS receiver 207 of the first UAV 2a as the current position of the first UAV 2a.

[0100] Next, the first learning unit 11 provides the current state of the first UAV2a, which is the position of the unit space where the first UAV2a is currently located, acquired in step S20, as input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value Q1 of the action value function, which represents the expected value of the cumulative value of future rewards that will be obtained when the first UAV2a takes each action related to movement in a specified direction relative to the direction of travel as the next course to be taken from the current position in the unit space (step S21).

[0101] Furthermore, the first learning unit 11 acquires the position of the unit space where the first UAV2a next arrives (step S22). Next, the first learning unit 11 provides the position of the unit space where the first UAV2a next arrives acquired in step S22 as input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value Q2 of the action value function (step S23). The position of the unit space where the first UAV2a next arrives is determined based on the GPS position of the first UAV2a acquired by the acquisition unit 10 for each time step.

[0102] Next, the first learning unit 11 calculates a target value from the second estimated value Q2 (step S24). Subsequently, the first learning unit 11 learns weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S5). Specifically, the first learning unit 11 updates the weight parameters of the neural network model so as to minimize the loss function of the above equation (1).

[0103] The storage unit 13 stores the learned parameters obtained in step S7 (step S26). The learned parameters stored in step S26 correspond to the course plan for the first UAV 2a to follow.

[0104] Next, with reference to FIG. 12, the learning process by the first learning unit 11 when a Fixed Target Q-Network using two neural networks, a main QN 111 and a target QN 113, is adopted will be described.

[0105] The processing of step S20 is the same as the learning processing step described in Fig. 11. Thereafter, the first learning unit 11 provides the main QN 111 with the position of the unit space where the first UAV 2a is currently located, acquired in step S20, as input, performs neural network calculations, outputs the action value function Q, and calculates the next course a to be taken (step S120).

[0106] Next, the first learning unit 11 returns the behavior of the first UAV2a along the path a obtained in step S120 to the environment 110, and obtains the next state s' of the first UAV2a, which is the position in the unit space to which the first UAV2a has progressed, and the reward r (step S121).

[0107] The first learning unit 11 stores the experience (s, a, r, a') obtained in step S121 in the experience data 114 (step S122). Next, the first learning unit 11 calculates the loss function L in the DQN loss calculation 112, and updates the weight of the main QN 111 with the gradient of the loss function L (step S123). The first learning unit 11 repeats the processes from step S120 to step S123 a set number of times.

[0108] Thereafter, the first learning unit 11 periodically copies the weights of the main QN 111 to the target QN 113 for synchronization (step S124). The synchronization of the target QN 113 is performed less frequently than the update frequency of the weights of the main QN 111. Next, the first learning unit 11 extracts experience from the experience data 114, inputs past states to the target QN 113, and calculates the estimated value max a’ Q(s',a';θ - ) is output (step S126).

[0109] Next, the first learning unit 11 calculates the estimated value max output by the target QN 113. a’ Q(s',a';θ - ) based on the target value r+γmax a’ Q(s',a';θ - ) (step S127). Next, the first learning unit 11 calculates the loss function L in the DQN loss calculation 112 using the target value calculated in step S127 (step S128). Next, the first learning unit 11 learns the weights of the main QN 111 so as to minimize the loss given by the loss function L (step S129). Thereafter, the learned parameters are stored in the storage unit 13 (step S26).

[0110] Next, the adversarial learning process by the second learning unit 12 of the control device 1 will be described with reference to Fig. 13. First, the second learning unit 12 acquires, as true policy data, the policy of the path that the first UAV 2a should take sequentially from its current position to the destination position, obtained by the learned parameters, i.e., the learned DQN, obtained by the reinforcement learning process by the first learning unit 11 in step S2 of Fig. 11 (step S30).

[0111] Here, as shown in the GAN learning process by the second learning unit 12 in FIG. 6, the true policy data collected in step S30 is used as training data 124 to be input when learning the classifier 122. In this embodiment, all policies obtained by the learned DQN are used as training data. All policies are the action values ​​Q(s) adjusted to enable the first UAV 2a to transition from its current position to the destination position as expected. t ,a1), Q(s t ,a2), Q(s t ,a3),···,Q(s t ,a n-1 ), Q(s t ,a n ) is a career path strategy.

[0112] Next, the second learning unit 12 inputs the true policy data to the classifier 122 as training data 124, and adjusts the parameter w of the classifier 122 so that the true policy data is distinguished from the true policy data (y=1). D ,θ D (Step S31). In Step S31, the second learning unit 12 can cause the classifier 122 to learn the true policy data using, for example, backpropagation. By Step S31, the classifier 122 that can distinguish the true policy data from the true policy data is constructed in advance.

[0113] Next, the second learning unit 12 generates Gaussian noise 120 and provides a random vector of the generated Gaussian noise 120 as an input to the generator 121 (step S32). Subsequently, the generator 121 calculates a random vector of the input z and the weight parameter w based on the provided Gaussian noise 120.G ,θ G Then, a product-sum operation and a threshold process using an activation function are performed to generate pseudo policy data G(z) (step S33).

[0114] Next, the second learning unit 12 learns the classifier 122. The learning of the classifier 122 is performed by using the parameter w D ,θ D First, the second learning unit 12 provides the training data 124 of the true policy data acquired in step S30 as input to the classifier 122, and adjusts the parameter w by backpropagation or the like so that the objective function E in the above formula (3) is maximized. D ,θ D (Step S34). The label of the training data 124 is set to 1 (true policy data).

[0115] Next, the second learning unit 12 provides the pseudo-policy data generated by the generator 121 in step S33 to the discriminator 122 as an input, and calculates the parameter w by backpropagation or the like so that the objective function E in the above equation (3) is maximized. D ,θ D That is, in steps S34 and S35, in order to maximize the objective function E in the above equation (3), the first term is updated as D(w D ,θ D )=1 is output, and the second term is D(G(w G ,θ G ),w D ,θ D )=0. Note that the label 0 (pseudo policy data) is set for the training data 124.

[0116] The training of the classifier 122 in steps S34 and S35 corresponds to the dashed arrows in FIG. 6, which indicate that a classifier error is calculated in block 125 of the objective function E based on the output 123 from the classifier 122, and then the error is backpropagated to the classifier 122.

[0117] Next, the second learning unit 12 trains the generator 121. The training of the generator 121 is performed with the parameters of the discriminator 122 fixed. The second learning unit 12 trains the generator 121 so that pseudo-policy data is generated when random Gaussian noise 120 is given to the generator 121. Specifically, the second learning unit 12 trains the parameters w G ,θ G is updated (step S36).

[0118] The learning in step S36 corresponds to the dashed arrow shown in Fig. 6, which indicates backpropagation of error to the generator 121. That is, step S36 corresponds to the dashed arrow shown in Fig. 6, in which the pseudo-policy data generated by the generator 121 is input to the discriminator 122, the generator error is calculated from the output 123 in the block 125 of the objective function E, and the error is further backpropagated to the generator 121.

[0119] Thereafter, the learning of the discriminator 122 and the generator 121 from step S33 to step S36 is repeated until the value of the objective function E reaches a Nash equilibrium and converges (step S37: NO). On the other hand, if the value of the objective function E converges (step S37: YES), the processing from step S31 to step S37 is repeated until the learning of the generator 121 and the discriminator 122 is performed using the remaining n-1 pieces of true policy data out of all n pieces of true policy data (step S38: NO).

[0120] Thereafter, when the generator 121 and the classifier 122 are trained using the remaining n-1 pieces of true policy data (step S38: YES), the second learning unit 12 stores the trained generator 121' in the storage unit 13 (step S39). The trained generator 121' is constructed by the above-described processes from step S30 to step S39.

[0121] [UAV operation] Next, the operation of the UAV 2 having the above-described configuration will be described with reference to the flowchart of FIG.

[0122] FIG. 14 is a flowchart showing flight control to a destination point based on route control information by a UAV 2. First, a destination point is set for each of multiple UAVs 2, including a first UAV 2a and a second UAV 2b (step S40). Next, the control information acquisition unit 21 of each UAV 2 acquires route control information (step S41). More specifically, the control information acquisition unit 21 of the first UAV 2a reads the route control information of the true strategy data set by the control device 1 and stored in the control information storage unit 20. Meanwhile, the control information acquisition unit 21 of the second UAV 2b reads the route control information of the pseudo strategy data stored in the control information storage unit 20.

[0123] Next, the position information acquisition unit 23 acquires the current unit space position of the host vehicle as the current position (step S42). Specifically, the position in the unit space corresponding to the GPS position received by the GPS receiver 207 can be acquired as the current position of the host vehicle. Next, the determination unit 24 determines the next route that the host vehicle should take based on the strategy data of the route control information acquired in step S41 (step S43).

[0124] More specifically, the decision unit of the first UAV 2a determines the current unit space position based on the route control information of the true strategy data. t As a result, for each state s t In the above, the route to be taken is determined by selecting the route that is the action a with the largest value of the action value function Q. Similarly, the determination unit 24 of the second UAV 2b determines the route to be taken in each state s based on the route control information of the pseudo-strategy data. t The path to be taken is determined by selecting the path that is action a with the largest value of the action value function Q.

[0125] The flight control unit 25 controls the flight of the aircraft based on the next course determined by the determination unit 24 (step S44). More specifically, the flight control unit 25 calculates a control command for the next course from the current position and transmits the control command value to the motor 209. Each of the first UAV 2a and the second UAV 2b maintains the current position in the unit space, that is, the state s t (step S42), the next course a to be followed is determined for each time step (step S43), and flight control is performed based on the determined course a (step S44).

[0126] Thereafter, when each of the first UAV 2a and the second UAV 2b reaches the unit space position of the destination point (step S45: YES), the process ends. In this way, each of the multiple UAVs 2 including the first UAV 2a and the second UAV 2b determines each state s based on the route control information. t By selecting the action a with the largest value of the action value function Q, the UAVs 2 can fly together to the destination point. At the destination point, the UAVs 2, including the first UAV 2a and the second UAV 2b, can perform pre-set tasks such as taking pictures.

[0127] As described above, according to the control device 1 of this embodiment, pseudo-policy data that simulates the path policy acquired by the trained reinforcement learning model is generated by the trained generator 121′ constructed by adversarial learning based on the current position of the first UAV 2a selected from among the multiple UAVs 2. Furthermore, information including the generated pseudo-policy data is set for each UAV 2 as path control information for moving the multiple UAVs 2 collectively to the destination point. Therefore, collective behavior can be controlled while reducing the complexity of the reward design for each UAV 2.

[0128] In the embodiment described above, the control device 1 generates a pseudo-DQN similar to a trained DQN by reinforcement learning using a DQN and adversarial learning using a GAN, and sets the pseudo-DQN to each UAV 2 as control information for guiding the UAVs 2 to their destinations. However, the control device 1 may also employ generative adversarial imitation learning (GAIL) in the first learning unit 11 and the second learning unit 12 described above.

[0129] GAIL uses a generator and a classifier, similar to GAN. The generator takes the current position as input and outputs a policy, which is the probability of a selected path (action). The classifier takes policy data consisting of a state and an action as input and outputs the probability that the policy data is that of an expert or is pseudo-policy data generated by the generator. The generator learns by considering the output from the classifier for the pseudo-policy data consisting of a state and an action generated by the generator as a reward. When GAIL is used, sampled data of the expert's policy data, which is training data, is input to the classifier. On the other hand, the classifier 122 according to the described embodiment inputs all of the policy data of the first UAV 2a's path corresponding to all of the expert's policy data as training data.

[0130] The above describes embodiments of the control device, control method, and control system of the present invention, but the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can make within the scope of the invention described in the claims are possible. [Explanation of symbols]

[0131] 1...control device, 2...UAV, 2a...first UAV, 2b...second UAV, 10...acquisition unit, 11...first learning unit, 12...second learning unit, 13...memory unit, 14...generation unit, 15...setting unit, 20...control information memory unit, 21...control information acquisition unit, 22...flight information memory unit, 23...position information acquisition unit, 24...determination unit, 25...flight control unit, 201, 101...bus, 202, 102...processor, 203, 103...main memory unit, 204, 104...communication interface, 205, 105... Auxiliary storage device, 206, 106...input / output I / O, 107...display device, 110...environment, 111...main QN, 112...DQN loss calculation, 113...target QN, 114...empirical data, 120...noise, 121...generator, 122...discriminator, 123...output, 124...training data, 125...objective function E block, 207...GPS receiver, 208...sensor, 209...motor, 210...propeller, 211...battery, A...mobile space, NW...wireless communication network.

Claims

1. A control device that controls a course of a group of moving objects to a destination position set in a moving space, a first acquisition unit configured to acquire a current position of a first moving object selected from the group of moving objects; a first learning unit configured to apply a reward function to an estimation result of calculating a course that the first moving body should take sequentially from the current position of the first moving body until it reaches the destination position, update the course so as to maximize a reward for the first moving body to reach the destination position, and learn a policy of a course that the first moving body should take sequentially from the current position using a reinforcement learning model; a second learning unit configured to perform adversarial learning of a generative model including: a generator that generates pseudo-policy data similar to true policy data, using a policy of a course that the first moving body should take sequentially from the current position obtained by learning by the first learning unit as true policy data; and a classifier that distinguishes between the pseudo-policy data generated by the generator and the true policy data; a generation unit configured to generate the pseudo policy data similar to the true policy data using a trained generator obtained by adversarial training of the second learning unit; a setting unit configured to set information including the pseudo-strategy data generated by the generation unit to each moving object as control information for controlling the paths of the group of moving objects; A control device comprising:

2. 2. The control device according to claim 1, the movement space is defined by a unit space obtained by dividing the movement space into a plurality of spaces, The course plan for the first moving body to proceed sequentially from the current position is a course plan for the first moving body to proceed sequentially in a unit space from a position in the unit space corresponding to the current position. A control device characterized by:

3. 3. The control device according to claim 2, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer; the first learning unit provides the current position in the unit space as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value of an action value function that represents an expected value of a cumulative value of the future reward to be obtained when the first moving body takes each action related to movement in a predetermined direction relative to a traveling direction as a course to be taken next from the current position in the unit space; the first learning unit further provides a next position in the unit space reached by the first moving object as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value of the action-value function; The first learning unit learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value. A control device characterized by:

4. A control method for controlling a course of a group of moving objects to a destination position set in a moving space, comprising: a first acquisition step of acquiring a current position of a first moving object selected from the group of moving objects; a first learning step of applying a reward function to an estimation result of calculating a course that the first moving body should take sequentially from the current position of the first moving body until it reaches the destination position, updating the course so as to maximize a reward for the first moving body to reach the destination position, and learning a course policy that the first moving body should take sequentially from the current position using a reinforcement learning model; a second learning step of performing adversarial learning of a generative model having a generator that generates pseudo-policy data similar to true policy data, using a policy of a course that the first moving body should take sequentially from the current position obtained by learning in the first learning step, and a classifier that distinguishes between the pseudo-policy data generated by the generator and the true policy data; a generation step of generating the pseudo policy data similar to the true policy data using a trained generator obtained by adversarial training in the second training step; a setting step of setting information including the pseudo-policy data generated in the generating step in each moving object as control information for controlling the routes of the group of moving objects; A control method comprising:

5. 5. The control method according to claim 4, the movement space is defined by a unit space obtained by dividing the movement space into a plurality of spaces, The course plan for the first moving body to proceed sequentially from the current position is a course plan for the first moving body to proceed sequentially in a unit space from a position in the unit space corresponding to the current position. A control method comprising:

6. 6. The control method according to claim 5, the reinforcement learning model is a neural network model including an input layer, a hidden layer, and an output layer; the first learning step provides the current position in the unit space as an input to the neural network model, performs calculations on the neural network model, and outputs a first estimated value of an action value function that represents an expected value of a cumulative value of the future reward to be obtained when the first moving body takes each action related to movement in a predetermined direction relative to a traveling direction as a next course to be taken from the current position in the unit space; The first learning step further includes providing a next position in the unit space reached by the first moving object as an input to the neural network model, performing a calculation on the neural network model, and outputting a second estimated value of the action-value function; The first learning step learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value. A control method comprising:

7. The control device according to any one of claims 1 to 3; the group of moving objects; A control system comprising: the group of moving bodies includes the first moving body and a plurality of second moving bodies; The first moving body and each of the plurality of second moving bodies are a second acquisition unit configured to acquire the control information set by the control device; a third acquisition unit configured to acquire a current location of the device; a determination unit configured to determine a next route to be taken from the current position of the host device acquired by the third acquisition unit, based on a route strategy to be taken by the host device from the current position, the route strategy being indicated by the control information; a movement control unit configured to control movement of the host device based on the next course determined by the determination unit; and Equipped with the control information includes the true strategy data and the pseudo strategy data; the determination unit of the first moving body determines the next course to be taken based on the true policy data; The determination unit of each of the plurality of second moving bodies determines the next course to be taken based on the pseudo-strategy data. A control system comprising:

Images