Control device, learning device, control method, learning method, and program

Abstract

This control device for a moving body includes: an acquiring means for acquiring state information including observation information of another moving body; a topological feature determining means for determining a target value of a topological feature for the other moving body on the basis of the state information; and an action determining means for determining the next action for reaching a destination such that the topological feature becomes the target value.

Description

Control device, learning device, control method, learning method, and program

[0001] This disclosure relates to the control of mobile devices, and more particularly to the control of distributed multi-agent systems.

[0002] The multi-agent distributed collision avoidance problem is a problem in which multiple agents move to a shared space to reach a destination, and each agent autonomously avoids collisions with other agents without centralized control or instructions. Each agent can observe its own surroundings and the position and velocity of other agents, but the destinations of other agents are unknown, and it decides its own actions under these conditions.

[0003] As a previous study on distributed collision avoidance problems in multi-agent systems, Non-Patent Document 1 proposes a phase-based method that considers the relative positional relationships and topological characteristics between agents. Specifically, the method proposed in Non-Patent Document 1 models collision avoidance between agents as a Hamiltonian dynamical system, predicts the number of rotations (winding number) for the trajectories between agents from observed information, and performs movement control considering this. The sign of the rotation number indicates whether collision avoidance is performed in the left or right direction, and collision avoidance becomes possible if each agent predicts a similar rotation number. This method makes predictions by enumerating and comparing possible interactions between multiple agents, so the computational cost increases significantly as the number of agents increases, resulting in a lack of scalability.

[0004] C. Mavrogiannis and RA Knepper, “Hamiltonian coordination primitives for decentralized multiagent navigation,” The International Journal of Robotics Research, vol. 40, no. 10-11, pp. 1234-1254, 2021.<URL: https: / / chrismavrogiannis.com / pdfs / mavrogiannis2021hamiltonian.pdf>

[0005] Conventional methods struggle to achieve both collision avoidance along smooth paths and scalability with respect to the number of agents.

[0006] This disclosure aims to provide a scalable movement control method that enables collision avoidance along smooth paths in a distributed multi-agent environment.

[0007] One aspect of the present disclosure is a control device for a moving object, comprising: acquisition means for acquiring state information including observation information of another moving object; topological feature determination means for determining a target value for a topological feature for the other moving object based on the state information; and action determination means for determining the next action to reach a destination so that the topological feature becomes the target value.

[0008] Topological features include homological features and homotopy features. Homological features are features that quantify and represent the structure of an orbit. In one embodiment, homological topological features can be the number of rotations (winding number) or the angle of rotation of the relative position with respect to other moving objects. The number of rotations or the angle of rotation can also be expressed as the number of times or angles by which two agents rotate around each other. Other homological topological features include the number of links, which represents how many times one orbit encloses another orbit, and the number of intersections, which represents how many times one orbit intersects another orbit. Homotopy features are features that represent whether multiple orbits coincide due to continuous deformation. For example, blade groups and homotopy classes can be used. In addition, quantities related to knots and links may be used as topological features.

[0009] In one embodiment, the phase feature determination means has a machine learning model that takes the state information as input and outputs the target value, and the machine learning model is trained by reinforcement learning.

[0010] In one embodiment, the action decision means determines the next action by model predictive control, which is used as an objective function that includes a penalty term based on the difference between the current value and the destination, a penalty term based on the relative distance to other agents, and a penalty term based on the difference between the phase feature and the target value.

[0011] Thus, this disclosure employs a two-stage control process: determining phase features and then controlling based on those determined phase features. Since the action is determined using phase features, collision avoidance along a smooth path is possible. Furthermore, the action decision process can be implemented, for example, by a computationally intensive model predictive control, making it applicable even with a large number of agents.

[0012] Another aspect of the present disclosure is a learning device comprising: a machine learning model; a phase feature determination means that determines a target value for a phase feature for another moving object based on state information including observation information of the other moving object; an action determination means that determines an action to reach a destination such that the phase feature becomes the target value; and a learning means that learns the machine learning model by reinforcement learning based on the reward for the action.

[0013] This disclosure can be understood as a control device or learning device comprising at least a part of the above means. This disclosure can also be understood as a method comprising at least a part of the above process, or as a program for implementing such a method or a recording medium on which such program is non-temporarily recorded. Furthermore, this disclosure includes devices or methods in which each of the above means and processes is combined with each other as far as possible.

[0014] According to this disclosure, even in a distributed multi-agent environment where a large number of agents are present, collision avoidance through smooth paths becomes possible.

[0015] A diagram showing an overview of the agent and its control device related to this disclosure. A diagram explaining the rotation speed related to this disclosure. A flowchart showing the flow of the movement control process in this disclosure. A diagram showing an overview of the learning device related to this disclosure. A flowchart showing the flow of the learning process in this disclosure. A diagram showing the experimental results in this disclosure. A diagram showing the experimental results in this disclosure. A diagram showing the experimental results in this disclosure. A diagram showing the experimental results in this disclosure. A diagram showing the configuration of the information processing device related to this disclosure.

[0016] This embodiment deals with route planning in a multi-agent environment with autonomous distributed control. Specifically, it is assumed that there is no centralized server that oversees the entire system and issues instructions to each agent, and that agents do not communicate with each other. Each agent can obtain the position and velocity of other agents through observation, but it is assumed that it cannot obtain the destination of other agents. In this embodiment, agent movement control is performed to ensure that agents reach their destinations without colliding with each other in such an environment.

[0017] <Agent and Control Device> In this embodiment, the agent is a mobile device such as an AMR (Autonomous Mobile Robot) that performs autonomous and distributed movement control.

[0018] Figure 1 shows the functional configuration of agent 100 in this embodiment. Agent 100 includes a control device 110 that controls the mobile device based on the state observation results, a sensor 120 that observes the environmental state, and a mobile device 130.

[0019] The control device 110 includes a state information acquisition unit 111, a target rotation speed determination unit 112, an action determination unit 113, and a movement control unit 114.

[0020] The status information acquisition unit 111 acquires status information from the sensor 120. Examples of status information include the position and speed of the own agent, and the positions and speeds of other agents and surrounding objects.

[0021] The target rotation speed determination unit 112 determines a target value for the rotation speed (Winding Number) for the trajectory based on state information. Figures 2(A) to 2(D) illustrate the rotation speed. Figures 2(A) and 2(B) show two agents A1 and A2 moving while avoiding collision. Two methods are assumed for the two agents to avoid collision: clockwise rotation and counterclockwise rotation. Figures 2(C) and 2(D) show the movement trajectory of agent B shown in Figures 2(A) and 2(B) as the relative displacement of agent A. In this embodiment, the rotation speed is defined between the two agents as the sum of the changes in the angularity of the other agent as seen from one agent. In this embodiment, the rotation speed is defined as follows.

[0022] Thus, the rotational speed λi is defined based on the observable state variables of agents i and j from time step k to l. Δθ on the right-hand side ｋ ｉ，ｊ This represents the change in the argument k of agent j as seen from agent i at time step k. By convention, counterclockwise rotation is considered positive for the argument, but the opposite is also acceptable. Here, the number of rotations is calculated by dividing the sum of the changes in the argument by 2π, but the sum of the changes in the argument, i.e., the rotation angle, may also be used as a topological feature.

[0023] The target rotation speed determination unit 112 has a policy function that has been pre-learned by reinforcement learning, and determines the target rotation speed value by inputting observed state variables into this policy function. In this embodiment, the policy function is constructed by a machine learning model such as a neural network. Note that the target rotation speed is determined for each agent if there are multiple agents in the vicinity.

[0024] The action decision-making unit 113 determines the next action based on the state variables acquired by the state information acquisition unit 111 and the rotational speed target value determined by the rotational speed target determination unit 112. In this embodiment, under the constraints of not colliding with other agents and the rotational speed of the trajectory with other agents matching the rotational speed target value, the next action is determined so that the agent reaches the destination in the shortest possible time.

[0025] In this embodiment, the action decision-making unit 113 is implemented using the method of model predictive control (MPC: Model Predictive Control). Formally, the action decision-making unit 113 is defined as follows.

[0026] Here, s is the state variable, y is the destination, and w is the rotational speed target value. The MPC controller π ｉ ＭＰＣ solves the optimal control problem represented by the following formula at each time step k with the prediction horizon length being K, and uses the initial value of the obtained optimal input sequence as the input to the system. Note that f is the state transition function, p is the position of the agent, r is the radius of the agent, and the variables with bars indicate the predicted values of the internal model possessed by the action decision-making unit 113.

[0027] The objective function J is defined as follows.

[0028] J ｇ is the penalty term for the distance to the destination, J Ｏ is the penalty term for collision with obstacles, and J ｗ is the penalty term for the deviation between the predicted rotational speed and the rotational speed target value. Also, α ｇ and α Ｏ and α ｗ are the weights for their respective penalty terms.

[0029] By using this type of model predictive control, it is possible to achieve control that reaches the destination as quickly as possible while matching the target rotation speed value, without colliding with other agents. Note that the implementation of the action decision unit 113 and the selection of the objective function are not limited to the above example and may be changed as appropriate depending on the purpose of the implementation.

[0030] The movement control unit 114 generates a control command for the movement device 130 to take the action determined by the action decision unit 113 and outputs it to the movement device 130.

[0031] Sensor 120 is a device for observing the environment surrounding agent 100, and includes cameras, lidars, radars, ultrasonic sensors, etc. Sensor 120 also includes a device for observing the state of agent 100 itself, acquiring information such as the position of agent 100 and the control status of the mobile device 130.

[0032] The moving device 130 includes a mechanism for moving the agent, for example, a mechanism for controlling the speed of movement such as an engine or motor, and a mechanism for controlling the direction of movement such as a steering angle control device.

[0033] Figure 3 is a flowchart showing the movement control process of agent 100 in this embodiment. Note that this flowchart is intended to simplify the overall process and may differ from the actual process flow. For example, some processes may be executed repeatedly or in parallel with other processes, and the execution order of the processes does not necessarily have to match the flowchart.

[0034] In step S1, the control device 110 acquires the destination of the agent 100. In step S2, the state information acquisition unit 111 acquires state information based on data from the sensor 120. The state information includes the position and velocity of the self-device and the positions and velocities of other devices. In step S3, it is determined whether the agent has reached the destination. If the agent has reached the destination, the process ends; otherwise, the process proceeds to step S4. In step S4, the target rotation speed determination unit 112 inputs the state information into a policy function pre-learned by reinforcement learning to determine a rotation speed target value. In step S5, the action determination unit 113 determines the next action to be taken using a model predictive controller based on the state variables and the rotation speed target value. In step S6, the movement control unit 114 controls the movement device 130 so that the agent 100 moves according to the determined action. Then, the process returns to step S2 and the above process is repeated.

[0035] Note that in the figure, it is described that the control by the movement control unit (S5 - S6) and the update of the rotation speed target value by the target rotation speed determination unit 112 (S4) are executed in the same cycle. However, the update of the rotation speed target value may be performed at a cycle longer than the control cycle. That is, instead of calculating the rotation speed target value every time the state information is acquired, the rotation speed target value may be calculated every time the state information is acquired multiple times.

[0036] The control device according to the present embodiment realizes collision avoidance on a smooth path by considering a phase feature quantity such as the rotation speed, and uses model predictive control with relatively light processing for movement control, so scalability with respect to an increase in the number of agents can also be ensured.

[0037] <Learning device> Hereinafter, a method for learning a policy for determining the rotation speed target value used by the target rotation speed determination unit 112 will be described. In the present embodiment, reinforcement learning is used to learn the policy.

[0038] In this embodiment, reinforcement learning is performed using the Proximal Policy Optimization (PPO) method. PPO is a method based on the Actor-Critic method, and it is a method of learning a policy function that determines the agent's action (in this embodiment, the rotational speed target value) and a state value function that evaluates the expected reward sum for the action, respectively. In PPO, the stability and efficiency are improved by restricting the update range of the policy.

[0039] FIG. 4 shows the functional configuration of a learning device 400 for learning a policy for determining a rotational speed target value used in the target rotational speed determination unit 112. The learning device 400 includes a state information acquisition unit 401, a reward calculation unit 402, a learning unit 403, a target rotational speed determination policy 405, a value function 404, and an action determination unit 406.

[0040] The target rotational speed determination policy 405 corresponds to the Actor and outputs a rotational speed target value based on the current state and the value function 404. The target rotational speed determination policy 405 is implemented by a machine learning model such as a neural network.

[0041] The value function 404 corresponds to the Critic and is a function representing the sum of the cumulative rewards expected to be obtained in the future in a specific state. The state value function is implemented by a machine learning model such as a neural network.

[0042] The state information acquisition unit 401 acquires the state information after taking the action determined by the action determination unit 406 based on the current state. When the reinforcement learning is performed using simulation, the state information acquisition unit 401 acquires the state information as the simulation result, and when the reinforcement learning is performed based on the actions in the real world, the state information acquisition unit 401 acquires the state information from the agent's sensors and the like.

[0043] The reward calculation unit 402 calculates the reward for an action based on the state information. Based on the determined rotation speed, the reward calculation unit 402 gives a negative reward if the robot collides with or approaches an obstacle, and a high reward if it reaches its destination. As an example, the reward calculation unit 402 calculates the reward for each action based on the following reward function. Here, d min is the distance to the nearest other agent, p is the position of the current agent, p g is the destination of the agent. Note that the agent's radius is set to 0.15, d min The value is the distance between the centers of the nearest agent minus 0.3.

[0044] The learning unit 403 learns the target rotation rate determination policy 405 and the value function 404 to maximize the objective function based on the cumulative expected reward, based on the reward obtained as a result of the action. The objective function is defined to promote reaching the destination in a short time by giving a discount rate γ as shown in the following equation. The discount rate γ is a value less than 1 that is close to 1, for example, γ = 0.95. Alternatively, a step penalty may be given instead of a discount rate.

[0045] Furthermore, in this embodiment, the agent trajectories obtained from one episode of execution for each agent are collected in a common buffer and used for training a common Actor and Critic.

[0046] The action decision unit 406 has the same function as the movement control unit 114 described above, and determines the next action the agent should take based on the state variable and the rotation speed target value. Although not shown in the diagram, the agent is controlled to take the determined action.

[0047] Figure 5 is a flowchart showing the flow of reinforcement learning processing by the learning device 400. In step S11, the state information acquisition unit 401 acquires state information. In step S12, it is determined whether the episode termination conditions have been met. If the episode has not ended, the process proceeds to step S13; if the episode has ended, the process proceeds to step S15. An episode ends when any of the following conditions are met: the agent reaches the destination, a collision occurs between agents, or a predetermined number of steps have elapsed. In step S13, the target rotation speed target value is determined using the target rotation speed determination policy 405. In step S14, the action determination unit 406 determines the next action to take based on the state variables and the target rotation speed value, and controls the agent to take the determined action. After the action, the process returns to step S11 and repeats from the acquisition of state information. If the episode ends in step S12, the process proceeds to step S15, where the learning unit 403 updates the target rotation speed determination policy 405 and the value function 404 based on the reward for each step calculated by the reward calculation unit 402.

[0048] The target rotational speed determination policy 405 learned in this way is used by the target rotational speed determination unit 112. This enables the target rotational speed determination unit 112 to determine a target rotational speed value that can realize an appropriate travel path in each state.

[0049] <Experimental Results> The results of the simulation experiment conducted to demonstrate the effectiveness of this method are described below. In the experiment, a setting was used in which multiple agents, arranged at high density, intersected simultaneously. The starting position and destination of each agent were generated on the same circumference, and the destination was placed on the opposite side of the starting point, with the center of the circle in between. Furthermore, in order to generate diverse environments, perturbations were applied to these starting positions and destinations during environment generation.

[0050] Figure 6 shows the trajectory of an agent in a single evaluation episode. Here, experimental results with five agents are shown, with P0 to P4 being the starting positions of each agent and G0 to G4 being the destinations. As mentioned above, the starting and target positions are set on opposite sides of the same circle, but a deviation from the circumference occurs due to perturbation. The trajectory T0 to T4 can be determined from the position of each agent at each step. As can be seen from these results, this method enables movement along a very smooth path. Furthermore, avoidance behavior is initiated when the agents are at a certain distance from each other, and stable and efficient collision avoidance is achieved by avoiding in a consistent direction.

[0051] Figure 7 shows the success rate of navigation to the destination for cases with 3, 5, 7, and 9 agents. Here, success is defined as all agents reaching the destination within a predetermined time without colliding with other agents. WNumMPC (the rightmost of the four graphs) in the figure represents the proposed method. ORCA, CADRL, and Normal MPC are conventional methods for comparison. ORCA is a response-based method, and CADRL is a deep reinforcement learning-based method; both are described in the following literature.

[0052] ORCA: Jur van den Berg, Stephen J. Guy, Jamie Snape, Ming C. Lin, and Dinesh Manocha, Reciprocal n-body collision, Robotics Research: The 14th International Symposium ISRR, Springer Tracts in Advanced Robotics, vol. 70, Springer-Verlag, May 2011, pp. 3-19,<URL:https: / / gamma.cs.unc.edu / ORCA / publications / ORCA.pdf> CADRL: YF Chen, M. Liu, M. Everett, and JP How, “Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 285-292.<URL: https: / / arxiv.org / abs / 1609.07845>

[0053] Normal MPC is a model predictive control method that does not consider rotational speed, and in the objective function J (Equation 4) of the model predictive control in this embodiment, the weight α of the penalty term related to the rotational speed target value is ｗ This is equivalent to setting it to zero.

[0054] As these experimental results show, unlike the comparative example, the proposed method maintains a high success rate even under settings with a large number of agents. This result indicates that the proposed method can perform safe navigation in a scalable manner with respect to the number of agents.

[0055] Figure 8 shows the timeout rate during navigation to the destination for cases with 3, 5, 7, and 9 agents. A timeout is counted when no agent collisions occur, but any agent fails to reach the destination within a predetermined time. The comparative examples are the same as above.

[0056] As these experimental results show, unlike the comparative example, the proposed method maintains a low timeout rate even under settings with a large number of agents, enabling rapid navigation completion. This result demonstrates that the proposed method can perform scalable and rapid navigation with respect to the number of agents.

[0057] Figures 9(A) to 9(C) show the 50th, 75th, and 90th percentile values ​​of the Average Extra Time to Goal, respectively. The Average Extra Time is the average difference between the time taken to reach the destination and the time taken to reach the destination at maximum speed. The time taken to reach the destination varies depending on the distance between the starting point and the destination, but the influence of this variability is eliminated by using the Average Extra Time as an indicator.

[0058] As these experimental results show, the proposed method exhibits significantly less average surplus time compared to conventional methods, demonstrating highly efficient navigation. Furthermore, the proposed method maintains a consistent average surplus time even as the number of agents increases. Additionally, there is little difference between the 50th and 90th percentile values ​​when comparing methods with the same number of agents. This indicates that stable navigation with nearly the same average surplus time is possible regardless of the environment settings. This result suggests that the proposed method appropriately considers inter-agent interactions and handles positional distortions even in complex, dynamic environments with a large number of agents. In particular, the scalability of the proposed method with respect to the number of agents is a noteworthy and significant effect.

[0059] Figure 10 shows the Average Cumulative Heading Change (hereinafter referred to as Average CHC). Therefore, if the agent travels along an ideal path that is perfectly straight, the CHC will be close to zero, and Average CHC is an indicator of how close the travel path is to a straight line.

[0060] As these experimental results show, the proposed method exhibits a significantly smaller average CHC compared to the other two methods excluding ORCA. This indicates that there is no fluctuation in the direction of agent movement, and collision avoidance is achieved with minimal evasive action. Furthermore, the proposed method shows that the average CHC increases only slightly even as the number of agents increases, demonstrating its scalability to an increasing number of agents.

[0061] <Hardware Configuration> Figure 11 shows the hardware configuration of an information processing device (computer) 1100 that functions as a control device 110 or a learning device 400. As shown in Figure 11, the information processing device 1100 according to this embodiment is a computer (information processing device) in which a control unit 1101, a storage unit 1102, an input device 1105, an output device 1106, a communication interface 1107, and a drive 1108 are electrically connected.

[0062] The control unit 1101 includes a hardware processor such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), and is configured to perform information processing based on programs and various data. The control unit 1101 (CPU) is an example of processor resources.

[0063] The memory unit 1102 is an example of a memory resource and is composed of, for example, a hard disk drive, a solid-state drive, etc. In this embodiment, the memory unit 1102 stores various information such as a control program 1103 and a reinforcement learning program 1104 related to the movement (navigation) of the agent 100.

[0064] The control program 1103 determines a target rotational speed value from environmental state information and navigates the agent to its destination so that the actual rotational speed matches the target value and avoids collisions with other agents. When the information processing device 1100 executes the control program 1103, the information processing device 1100 functions as the control device 110.

[0065] The reinforcement learning program 1104 is a program for executing a reinforcement learning process for determining a policy (strategy) for determining the target value of the rotation speed. By executing the reinforcement learning program 1104, the information processing device 1100 functions as a learning device 400.

[0066] The input device 1105 is a device for inputting data, such as a mouse, keyboard, keypad, or touch panel device. The output device 1106 is a device for outputting data, such as a display or speaker. The user can operate the control device 110 by using the input device 1105 and the output device 1106. The input device 1105 and the output device 1106 may be integrated into a single unit, such as a touch panel display.

[0067] The communication interface 1107 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for performing wired or wireless communication over a network.

[0068] The drive 1108 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading various information such as programs stored in the storage medium 1109. At least one of the control program 1103 and the reinforcement learning program 1104 may be stored in the storage medium 1109.

[0069] The storage medium 1109 is a medium that stores information such as stored programs by electrical, magnetic, optical, mechanical, or chemical means so that a computer or other device or machine can read the stored information such as programs. The control device 110 or the learning device 400 may obtain at least one of the control program 1103 and the reinforcement learning program 1104 from the storage medium 1109.

[0070] In Figure 11, a disk-type storage medium such as a CD or DVD is shown as an example of a storage medium 1109. However, the type of storage medium 1109 is not limited to disk type and may be other types. Examples of storage media other than disk type include semiconductor memory such as flash memory. The type of drive 1108 may be appropriately selected according to the type of storage medium 1109.

[0071] Regarding the specific hardware configuration of the control device 110 and the learning device 400, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 1101 may include multiple hardware processors. The hardware processors may consist of a microprocessor, FPGA (field-programmable gate array), DSP (digital signal processor), etc. The storage unit 1102 may consist of RAM and ROM included in the control unit 1101. At least one of the input device 1105, output device 1106, communication interface 1107, and drive 1108 may be omitted. The control device 110 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the control device 110 and the learning device 400 may be information processing devices designed specifically for the services provided, as well as general-purpose server devices, general-purpose PCs (personal computers), industrial PCs, etc.

[0072] <Other Embodiments> The embodiments described above are merely examples, and this disclosure may be modified as appropriate without departing from its essence.

[0073] In the above embodiment, rotation number or rotation angle is used as a topological feature, but other features may be used as long as they represent the change in relative position between two or more agents. Homological features or homotopy features can be used as such topological features. Homological features include the number of links and the number of intersections, in addition to the rotation number. Homotopy features include the blade group and homotopy classes. The blade group is a topological feature that represents the exchange of positions between two or more agents. In addition, quantities related to knots and links can also be used as topological features.

[0074] In the above embodiment, an example was shown where the agent was an AMR (Anti-Molecular Responder), but this method can be applied to other mobile objects besides AMRs. Examples of mobile objects to which this method can be applied include automobiles, flying objects (drones), ships, underwater exploration robots, cleaning robots, and food delivery robots. In addition to these, this method can also be applied to self-propelled capsule endoscopes and laparoscopic cameras that move inside the body or blood vessels. Furthermore, the agents to which this method can be applied are not limited to devices in the real world, but can also be applied to mobile objects in virtual worlds such as simulations and games. Such mobile objects in virtual worlds include not only the devices described above, but also NPCs (Non-Player Characters).

[0075] In the above embodiment, we showed an example where only agents moving according to the same policy exist in the environment, but other objects may also be present. For example, the environment may contain stationary obstacles, or agents moving according to different policies. Other obstacles may include people or vehicles driven by people. In any case, if the reinforcement learning method described above is applied in an environment similar to the actual application scenario, appropriate navigation can be achieved.

[0076] In the above embodiment, we described an example in which there is no centralized server in the system and each agent performs control based only on information it has measured itself. However, a centralized server may exist in the system. For example, each agent may communicate with each other through this server, or information about the environment may be sent from the server to the agents. Agents may also communicate with each other. With such a configuration, each agent can perform control based on a more accurate knowledge of the state of the environment.

[0077] <Note> 1. A control device (110) for a mobile body (100), comprising: an acquisition means (111) for acquiring state information including observation information of another mobile body; a phase feature determination means (112) for determining a target value for the phase feature of the other mobile body based on the state information; and an action determination means (113) for determining the next action to reach the destination so that the phase feature becomes the target value.

[0078] 2. A learning device (400) comprising: a phase feature determination means (405, 112) having a machine learning model and determining a target value for a phase feature for another moving object based on state information including observation information of the other moving object using the machine learning model; an action determination means (406, 112) determining an action to reach a destination so that the phase feature becomes the target value; and a learning means (403) learning the machine learning model by reinforcement learning based on the reward for the action.

[0079] 100: Agent (mobile unit) 110: Control device 111: State information acquisition unit 112: Target rotation speed determination unit 113: Action determination unit 114: Movement control unit 400: Learning device 401: State information acquisition unit 402: Reward calculation unit 403: Learning unit 404: Value function 405: Target rotation speed determination policy 406: Action determination unit

Claims

1. A control device for a moving object, comprising: acquisition means for acquiring state information including observation information of another moving object; phase feature determination means for determining a target value for a phase feature for the other moving object based on the state information; and action determination means for determining the next action to reach a destination so that the phase feature becomes the target value.

2. The control device according to claim 1, characterized in that the phase feature quantity is the number of rotations or rotation angle of the relative position with respect to another moving body.

3. The control device according to claim 1, wherein the phase feature determination means has a machine learning model that takes the state information as input and outputs the target value, and the machine learning model is learned by reinforcement learning.

4. The control device according to claim 1, characterized in that the action determination means determines the next action by model predictive control, which uses the difference between the phase feature quantity and the target value as an objective function.

5. The control device according to claim 4, wherein the objective function further includes at least one of a penalty term based on the difference between the current value and the destination, and a penalty term based on the relative distance to other agents.

6. A learning device comprising: a phase feature determination means having a machine learning model, which determines a target value for a phase feature for another moving object based on state information including observation information of the other moving object; an action determination means determining an action to reach a destination so that the phase feature becomes the target value; and a learning means learning the machine learning model by reinforcement learning based on the reward for the action.

7. The learning device according to claim 6, characterized in that the phase feature quantity is the number of rotations or the rotation angle of the relative position with respect to another moving body.

8. A method for controlling a moving object, comprising: an acquisition step of acquiring state information including observation information of another moving object; a phase feature determination step of determining a target value for a phase feature for the other moving object based on the state information; and an action determination step of determining the next action to reach a destination so that the phase feature becomes the target value.

9. The control method according to claim 8, characterized in that the phase feature quantity is the number of rotations or rotation angle of the relative position with respect to another moving body.

10. The control method according to claim 8, characterized in that, in the phase feature determination step, the state information is input to a machine learning model to determine the target value, and the machine learning model is trained by reinforcement learning.

11. The control method according to claim 8, characterized in that in the action decision step, the next action is determined by model predictive control, which is used as an objective function that includes a penalty term based on the difference between the current value and the destination, a penalty term based on the relative distance to other agents, and a penalty term based on the difference between the phase feature quantity and the target value.

12. A learning method comprising: a phase feature determination step in which a machine learning model determines a target value for a phase feature for another moving object based on state information including observation information of the other moving object; an action determination step in which an action is determined to reach a destination such that the phase feature becomes the target value; and a learning step in which the machine learning model is trained by reinforcement learning based on the reward for the action.

13. The learning method according to claim 12, characterized in that the phase feature quantity is the number of rotations or rotation angle of the relative position with respect to another moving body.

14. A program for causing a computer to perform each step of the control method described in any one of claims 8 to 11.

15. A program for causing a computer to perform each step of the learning method described in any one of claims 12 to 13.

Images