Behavior control planning device and behavior control planning method

Abstract

To plan the details of behavior control of a movable body relative to various behaviors of a target movable body in various environments.SOLUTION: A behavior control planning device planning the behavior of a movable body relative to a target movable body performs simulation using simulation conditions for the movable body and the target movable body and a preset machine learning model and outputs the positional relationship between the movable body and the target movable body as step data, provides the step data to the machine learning model to cause the machine learning model to learn the step data and accumulates episode data including the positional relationship between the movable body and the target movable body and a reward and a state as a learning result, screens the accumulated episode data according to a predetermined condition, creates a learning pattern using the episode data based on a result of tabulation of the accumulated episode data, and provides the screened episode data to the machine learning model in the created learning pattern to cause the machine learning model to learn every episode data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a behavior control planning device and method for automatically planning the behavior of a mobile body such as a ship or a vehicle relative to other mobile bodies. [Background technology]

[0002] A method for controlling the behavior of a mobile object in response to the environment must take into account various objectives and requirements. The environment includes fixed limitations such as a limited range of movement and obstacle avoidance, as well as the positional relationship with other mobile objects (targets). In particular, the positional relationship between the target mobile object and the mobile object itself requires active objectives such as avoidance, tracking, or pursuing, to maintain a constant distance from the target mobile object.

[0003] Furthermore, even when using a movement method to achieve various actions toward a goal, it is necessary to reflect in the action plan the merits and demerits of the movement method relative to the goal, such as positional relationships that are likely to result in failure to achieve the goal, positional relationships that are likely to result in success due to environmental changes, and even positional relationships that need to be considered after the action has been taken, in order to achieve realistic movement.

[0004] Deciding how a moving object should move is equivalent to determining the behavioral control content that can be set in the control system of the moving object. Conventionally, to determine the behavioral control content of a moving object to be controlled, a control method is designed based on control theory, but behavioral control content can also be planned using reinforcement learning.

[0005] In particular, in recent years, the learning capabilities of deep reinforcement learning, which trains deep neural networks defined by the relationship between state variables, actions (policies), and action values ​​or rewards, have improved, and its application to the design and planning of control methods is being considered.

[0006] Patent Document 1 proposes a technology for measuring disturbances in the event that the environment in which a robot previously learned differs from the environment after learning, and an action is selected based on the amount of change in the disturbance compared to the learning period. Patent Document 2 proposes a technology for determining the behavior of a moving object by deep reinforcement learning in an environment where obstacles are set.

[0007] Patent Document 3 proposes a technology that uses trajectory data of a moving object as training data to determine a cost function related to the movement of the moving object through inverse reinforcement learning. A behavior control method can be learned by setting a reward for reinforcement learning based on the cost function and performing reinforcement learning for the environment. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-320997 [Patent Document 2] Japanese Patent Application Publication No. 2018-198012 [Patent Document 3] International Publication No. 2019 / 167457 Summary of the Invention [Problem to be solved by the invention]

[0009] When using reinforcement learning technology to determine the behavioral control of a mobile object depending on the environment, the first problem is whether the learning itself is successful. If the conditions, such as the environment, are complex, it becomes difficult to achieve the learning itself.

[0010] In the behavior control planning of a moving object, the positional relationship between the moving object and the dynamically changing target is important, especially for purposes such as tracking and pursuit, and it becomes difficult to determine the behavioral value and reward in advance, or it becomes a factor that complicates the environmental conditions.

[0011] It is also desirable to learn behavior control methods that are effective for achieving goals, based on behavior control obtained from the learning results and unexpected behaviors. The same applies to behavior control methods when there are two or more mobile units.

[0012] In Patent Document 1, the behavior control method after learning involves providing additional measurement means in the actual environment to correctively change the behavior selection, but does not disclose a learning method for various situations, such as when the environment changes.

[0013] In Patent Document 2, the environment is set to consist of stationary obstacles or paths, and no learning method is shown that can adapt to various environments based on the learning results of one moving object.

[0014] In Patent Document 3, the reward is determined from fixed information such as obstacles in the surrounding space as the environment for trajectory data, which is the trajectory of a moving object. Since the design content of reinforcement learning itself is modified, there are concerns that this may affect abilities such as successful learning.

[0015] Furthermore, in Patent Document 3, the use of inverse reinforcement learning itself does not directly reflect what has been found as a result of reinforcement learning (whether an action is good or bad) and does not involve learning by efficiently utilizing that data. Patent Document 3 does not disclose a means for creating a cost function for the dynamic positional relationship between a target moving object and its own moving object, and does not describe learning such as tracking behavior.

[0016] An object of the present invention is to plan behavior control content for a mobile object in response to various behaviors of target mobile objects in various environments. [Means for solving the problem]

[0017] The present invention provides a behavior control planning device having a processor and a memory, which plans the behavior of a self moving body with respect to the behavior of a target moving body, and which includes: a simulator that receives simulation conditions that are preset for the behavior of the self moving body and the target moving body, and outputs the positional relationship between the self moving body and the target moving body as step data at predetermined time intervals based on the simulation conditions and a preset machine learning model; a reinforcement learning unit that acquires the step data, provides the step data to the machine learning model to learn, and accumulates episode data in a data accumulation unit as the learning result, the episode data including state quantities indicating the positional relationship between the self moving body and the target moving body, rewards, and states; a screening unit that selects the episode data accumulated in the data accumulation unit based on predetermined conditions; a learning pattern generation unit that aggregates the episode data in the data accumulation unit and generates a learning pattern including an order in which the episode data will be used based on the aggregation result; and a transfer learning unit that provides the episode data selected by the screening unit to a machine learning model using the learning pattern generated by the learning pattern generation unit, thereby allowing learning for each of the episode data. [Effects of the Invention]

[0018] According to the present invention, it is possible to learn behavioral control to achieve goals such as tracking and pursuit in various environments, especially when the positional relationship between the target moving object and the moving object varies.The results of reinforcement learning are accumulated as episode data, and the accumulated episode data is utilized in transfer learning, thereby improving the quality of the behavioral control plan by utilizing the machine learning model obtained from transfer learning.

[0019] The details of at least one implementation of the subject matter disclosed herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosed subject matter will become apparent from the following disclosure, drawings, and claims. [Brief explanation of the drawings]

[0020] [Figure 1]FIG. 2 is a diagram illustrating an example of processing performed in the behavior control planning system according to the embodiment of the present invention. [Figure 2] 4 is a flowchart illustrating an example of processing performed by the behavior control planning device according to the embodiment of the present invention. [Figure 3A] FIG. 1 illustrates an embodiment of the present invention and is a diagram illustrating an example of an environment setting in which a target moving object is tracked by one moving object. [Figure 3B] FIG. 1 illustrates an embodiment of the present invention and is a diagram illustrating an example of an environment setting in which a target moving object is tracked by two moving objects. [Figure 4] 1A and 1B are diagrams illustrating an example of a behavior pattern in which one moving object is tracked against a target moving object according to an embodiment of the present invention. [Figure 5] FIG. 1 shows an embodiment of the present invention and is a diagram illustrating an example of a behavior pattern in which two moving objects track a target moving object. [Figure 6] 1A and 1B are diagrams illustrating an example of a trajectory when one moving object is tracking a target moving object according to an embodiment of the present invention. [Figure 7] FIG. 2 shows an embodiment of the present invention and is a diagram illustrating an example of trajectories when two moving bodies are tracking a target moving body. [Figure 8] 1 is a flowchart illustrating an example of processing from reinforcement learning to transfer learning according to an embodiment of the present invention. [Figure 9] FIG. 10 illustrates an example of the success and failure of an episode and the number of steps in the learning process of reinforcement learning according to the embodiment of the present invention. [Figure 10] FIG. 10 illustrates an example of a diagram illustrating the characteristics of a learning process of reinforcement learning according to an embodiment of the present invention. [Figure 11] FIG. 10 illustrates an example of a learning pattern according to the embodiment of the present invention. [Figure 12] 1 is a block diagram illustrating an example of a behavior control planning device according to an embodiment of the present invention. [Figure 13] 1A and 1B are diagrams illustrating an example of the positional relationship between a target moving body and a moving body according to an embodiment of the present invention. [Figure 14] 10A to 10C are diagrams illustrating examples of graphs of trajectories, distances, directions, relative velocities, and distance changes of successful and unsuccessful episodes according to an embodiment of the present invention. [Figure 15A] FIG. 10 is a diagram illustrating an example of the relationship between the change in distance and the judgment index with respect to the number of steps in an episode when the learning result is “capture,” according to an embodiment of the present invention. [Figure 15B] FIG. 10 is a diagram illustrating an example of the relationship between the change in distance and the judgment index with respect to the number of steps in an episode when the learning result is “lost,” according to an embodiment of the present invention. [Figure 16] FIG. 10 illustrates an example of a graph of teaching data and predicted values ​​of a determination index model according to an embodiment of the present invention. [Figure 17A] FIG. 10 illustrates an example of prediction by a decision index model for individual episode 1 according to an embodiment of the present invention. [Figure 17B] FIG. 10 illustrates an example of prediction by a decision index model for individual episode 2, according to an embodiment of the present invention. [Figure 17C] FIG. 10 illustrates an example of prediction of a decision index model for a failure episode, according to an embodiment of the present invention. [Figure 18] FIG. 10 illustrates an example of the frequency of occurrence of output for input data in a decision index model using random forest and multiple regression analysis, according to an embodiment of the present invention. [Figure 19] 10A and 10B are diagrams illustrating an example of the positional relationship between a target moving body and a moving body itself in classification of the trajectories of the moving bodies according to an embodiment of the present invention. [Figure 20] 1 is a diagram illustrating an embodiment of the present invention and illustrating the difference in the direction of revolution and rotation of a moving body. FIG. [Figure 21] FIG. 10 illustrates an example of classification of trajectories based on trajectory feature amounts according to an embodiment of the present invention. [Figure 22] FIG. 10 illustrates an embodiment of the present invention, showing a classification of the relationship between episode success and failure for a target and a wingman when tracking two moving objects. [Figure 23] FIG. 10 illustrates an embodiment of the present invention, showing how episodes of success and failure occur depending on the factors involved in tracking two moving objects. [Figure 24] FIG. 1 illustrates an embodiment of the present invention and is a diagram showing an example of a learning process of reinforcement learning for tracking by two moving objects. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0022] In the present invention, when a moving body (or the moving body itself) such as a ship, vehicle, or aircraft performs tracking in which the body behaves in consideration of its positional relationship with another moving body (target moving body), the body is controlled to perform optimal behavior.

[0023] For automatic behavior control planning, reinforcement learning technology is used, which simulates the movement of the self moving body and the target moving body. In reinforcement learning, tracking begins from the initial positions of the self moving body and the target moving body, and they move one step at a time (a specified time interval), and if they continue to satisfy constraints or restrictions on the relationship between them, such as their distance, (or if they reach a specified number of steps), the goal of continuing tracking is achieved and the system is successful, or if they deviate from the constraints or restrictions along the way, the system is considered to have failed.

[0024] An episode is the period from the start of training a machine learning model using reinforcement learning to the point where it either succeeds or fails. By repeating episodes, the model learns how to control its behavior to achieve its goal.

[0025] For reinforcement learning, the relationship between state variables, actions (policies), and state values ​​is defined in a deep neural network (DNN), and the DNN or machine learning model is trained using state, action, and reward data for each episode and step.

[0026] An episode consists of successive step data from the start to the end of success or failure. Multiple, typically many, episodes are passed before learning is completed. Successful learning is judged by testing the action selection on an episode-by-episode basis. Learning may fail if no action control method that satisfies the constraints is obtained.

[0027] Reinforcement learning is likely to be successful if the behavior pattern of the target moving object relative to the self moving object is simple or one of a kind. However, it is expected that the behavior pattern of the target moving object will vary. A learning method that achieves a certain goal by learning to achieve the goal in other environments and for other purposes is called transfer learning. Note that if the moving object whose behavior is to be controlled is easily confused with the target moving object, it will be called the self moving object.

[0028] In this invention, the transfer learning method is an episode data learning method that performs machine learning using episode data obtained when reinforcement learning is performed under individual environmental conditions for environmental conditions such as the behavior patterns of various target moving objects. Episode data is a sequence of state quantities, actions (policies), and state values ​​for each step. Episode data is the result of reinforcement learning under individual behavior patterns. It accumulates.

[0029] In the transfer learning method (episodic data learning method), the learning process for each episode in reinforcement learning, from success to failure, is analyzed in terms of the number of steps. Then, the episodes of failure or success according to the number of steps are arranged in the order of use of learning data from the start to the end of learning, and a learning pattern is set. Episodic data learning is a learning method in which episode data is searched and acquired according to the learning pattern, and those episodes are used in sequence to train the DNN.

[0030] In action control planning, it is desirable that the action itself be excellent for achieving the goal. For example, there may be cases where the risk of failure is low, the trajectory of the moving object itself is unlikely to fail, and no unnecessary energy (fuel, etc.) is used, which cannot be directly evaluated by the position or speed of the moving object at each step in the episode.

[0031] In order to realize learning that reflects the risk of failure and the quality of the trajectory, the present invention performs screening to select episode data. The screened episode data is searched and acquired for episode data learning, and transfer learning is then performed to improve the qualitative quality of the behavior control method that is the learning result, that is, the quality and learning ability.

[0032] Furthermore, it is possible that the target moving object may not be a single object but two or more objects. Since failure or success is determined by the relationship between the target moving object and the multiple object moving objects, and also by the relationship between the object moving objects themselves, the number of factors for failure increases. In such cases, screening episode data for episode data learning is also effective.

[0033] As a result, by using episode data resulting from reinforcement learning for each behavior pattern, and then screening the episode data and performing transfer learning, a machine learning model can be obtained that plans behavior control for multiple behavior patterns. Furthermore, it is possible to plan behavior control for multiple self-moving bodies. This makes it possible to learn high-quality behavior control methods for complex relationships between target moving bodies and self-moving bodies, which is difficult to achieve with conventional reinforcement learning alone.

[0034] Fig. 1 is a diagram showing an example of processing performed in a behavior control planning system. Fig. 1 shows an overview of processing of behavior control planning using reinforcement learning, episode screening, and transfer learning according to an embodiment of the present invention. Fig. 2 is a flowchart showing an example of processing performed in the behavior control planning device.

[0035] First, in step 201, the behavior control planning device performs reinforcement learning of behavior control of a moving object under conditions such as individual environments. Here, "individual" refers to each of the various conditions for the various environments and the behavior patterns of the target moving object, and refers to the range of conditions under which reinforcement learning is successful or likely to be successful.

[0036] In reinforcement learning, the action selection for the state of the mobile unit proceeds step by step, and if the goal of the mobile unit is achieved, the episode is successful; if not, it is a failure. In processing the episode, a function that defines the relationship between the state quantity, the policy, and the state value is trained.

[0037] The above-mentioned one step is step data including the positional relationship between the own moving body and other moving bodies at predetermined time intervals performed by the simulator. The behaviors of the own moving body and other moving bodies are set in advance in the simulator as simulation conditions.

[0038] The behavior control planning device repeats episode processing until there are no more failures or only successes in the episode processing, at which point the learning is considered successful and reinforcement learning is complete. In one reinforcement learning of individual conditions, many episodes are processed. The state quantities, actions, and reward data for each step of the episode are called episode data.

[0039] The failure or success of an episode is determined by the reward value of the final step, but information on success or failure may be added directly to the episode data. In step 202, the behavior control planning device stores the episode data in a database along with the order of the reinforcement learning process.

[0040] The database also stores trajectory data for the target moving object and the self moving object in each episode. Note that episode data is stored even if the reinforcement learning fails. Episode data for reinforcement learning under different conditions is also stored.

[0041] Then, in step 203, the behavior control planning device screens (selects) episode data for transfer learning, which learns behavior control corresponding to various environments and behavior patterns.

[0042] To screen episodes, a screening model is required, and the behavior control planning device classifies the success or failure of an episode and the factors of the failure. For example, a model for tracking a target moving object is a model for determining the risk of tracking failure (failure risk modeling 153), and another example is a model for classifying the characteristics of the global trajectory over an episode (global trajectory classification model 154).

[0043] The equations that define the model are prepared in advance, but the model coefficients and thresholds for selecting episodes are set based on the episode data and trajectory data accumulated in the database through reinforcement learning.The behavior control planning device then outputs two types of learning episode data: episode data selected based on failure risk (first learning episode data) and episode data selected based on trajectory characteristics (second learning episode data), and uses these in transfer learning.

[0044] The behavior control planning device determines the order of episodes to be applied in training order for episode data learning (transfer learning) that trains a deep neural network (DNN) with episode data.

[0045] This sequence of episodes during learning is called the learning process. In the beginning of the learning process, there are many failures and only a few steps in each episode, but as episodes are repeated, the number of steps gradually increases, and at the end when learning is successful, there are only successful episodes with the same number of steps.

[0046] In this way, the behavior control planning device performs transfer learning based on the definition of the relationship between the learning process and the number of steps. If the number of steps is not that of a successful episode, the episode is a failure, and the type of failure is also corresponding information. The definition of the relationship between the learning process and the number of steps, and between success and failure, is called a learning pattern. For transfer learning, the behavior control planning device sets a learning pattern in step 204.

[0047] In step 205, the behavior control planning device reads episode data from the database and selects or sets episodes to be used in transfer learning.

[0048] Then, in step 206, the behavior control planning device performs transfer learning. In transfer learning, the screened episodes are obtained from the database according to the episodes of the learning process set in the learning pattern, and learning proceeds.

[0049] The above is an explanation of the overall flow of processing behavioral control plans using reinforcement learning, episode screening, learning pattern design, and transfer learning, using the flowchart in Figure 2.

[0050] The diagram of the behavior control planning system in Figure 1 shows the functions of the computer system, the processing order, and the exchange of data using arrows.

[0051] The behavior control planning system is classified into processing by an operator 101, processing by a simulator 102, and processing by reinforcement learning 103, episode screening 105, and transfer learning 106 of the behavior control planning device 100.

[0052] Although this embodiment shows an example in which the simulator 102 is implemented by the behavior control planning device 100, the simulator 102 may also be implemented by a computer (not shown) connected to the behavior control planning device 100. Furthermore, the processing by the operator 101 can be implemented by a computer (not shown) connected to the behavior control planning device 100 or the input / output device 14 of the behavior control planning device 100.

[0053] The episode data and trajectory data of reinforcement learning are stored in the episode database 104. Unless otherwise specified, the actual processing performed hereinafter is automatic processing by a computer program (the behavior control planning device 100).

[0054] The operator 101 sets environmental information such as the initial positions and movement ranges of the moving body and the target moving body, the movement pattern of the target moving body, parameters of the reinforcement learning algorithm, information required for calculation, etc. (condition setting 111). The condition setting 111 by the operator 101 is a manual process.

[0055] The simulator 102 has an action selection and update function (action selection and update unit 121) that sets environmental information determined in the condition setting 111, calculates information on the target moving object and the own moving object for each step of an episode, and updates information on the action selection result at the next point in time series. The action selection and update information is used as an episode in the DNN training unit 131 of the reinforcement learning 103 and in the transfer learning 106.

[0056] In reinforcement learning 103, based on the information in condition setting 111, the action selection and update unit 121 of the simulator 102 obtains information on state quantities, action selection (policy), and rewards for each episode and step of processing, and the DNN training unit 131 trains the deep neural network 10.

[0057] The DNN training unit 131 provides the output of the action selection and update unit 121 to the deep neural network 10 (machine learning model) to perform reinforcement learning.

[0058] The DNN training unit 131 continues processing until reinforcement learning 103 succeeds or fails, and stores episode data and trajectory data in the learning process in the episode database 104. The DNN training unit 131 similarly performs reinforcement learning for various different environments and conditions such as behavioral patterns, and accumulates episode data in the episode database 104.

[0059] Next, episode screening 105 is performed on the episode data and trajectory data stored in episode database 104 .

[0060] Then, episode screening is performed in the episode screening unit 152. The episode screening unit 152 classifies or determines episodes of the episode data used for transfer learning using a model that uses trajectory data. Specific models include failure risk modeling 153 that determines the risk of failure, and a global trajectory classification model 154 that classifies the trajectory data.

[0061] In the modeling of the failure risk modeling 153 and the global trajectory classification model 154, the definition of the model formula, the definition of the data required for model fitting, and the setting of the threshold for screening must be done manually. Also, once the model formula, data format, and threshold settings are determined, the model fitting calculation and the screening process using the model prediction function can be automatically processed by computer.

[0062] Transfer learning 106 includes a learning pattern design unit 161 that designs learning patterns, an episode setting and selection unit 162 that sets episodes to be used in transfer learning execution unit 163 based on the episodes selected by episode screening 105 and the results of learning pattern design unit 161, and selects episodes from episode database 104, and transfer learning execution unit 163 that executes transfer learning.

[0063] The transfer learning execution unit 163 directly learns using episode data, but the training of the DNN in learning is the same as the algorithm of reinforcement learning 103, and can utilize the DNN training unit 131. The learning target of the transfer learning execution unit 163 may be the deep neural network 10 (machine learning model) that has undergone reinforcement learning 103, or a new deep neural network 10.

[0064] The above is an overview of the processing of the behavior control planning system shown in Figures 1 and 2.

[0065] <Transfer Learning> In this embodiment, a transfer learning target obtained by screening episode data and a transfer learning method to which episode data screening is applied will be described.

[0066] First, in learning the behavior control method, the environment settings for tracking a target moving object with the own moving object are shown in Figs. 3A and 3B.

[0067] Fig. 3A shows an environment 301 in which one moving object is tracking a target moving object as Activity Example 1, and Fig. 3B shows an environment 302 in which two moving objects are tracking a target moving object as Activity Example 2. In the illustrated environment, position and movement are expressed in the coordinate system x (302) and y (303).

[0068] In tracking by one own moving body 304, the own moving body 304 tracks the trajectory 306 of the movement of the target moving body 305 as shown by the trajectory 307 indicated by the dashed arrow. For tracking, the own moving body 304 needs to detect the position of the target moving body 305, and the condition is that the target moving body 305 exists within a range of a detection distance 308. If the own moving body 304 goes outside the range of the detection distance 308, it will enter a range (capture distance 309) where it will lose sight of the target moving body 305.

[0069] Furthermore, if the own moving body 304 gets too close to the target moving body 305, the own moving body 304 will be found by the target moving body 305 and will be captured. This distance is set in advance as a capture distance 309. Therefore, a necessary condition during tracking is that the own moving body 304 must be farther away from the target moving body 305 than the capture distance 309 and be within the range of the detection distance 308.

[0070] If the own moving body 304 approaches closer than the capture distance 309, it is "captured", and if the own moving body 304 moves farther away than the detection distance 308, it is "lost" and the episode fails.

[0071] When there are two own moving bodies, the relative positions of the wingmen also become a constraint on behavior. In Figure 3B, the target of moving body 1 (320) is the same target moving body 305 as in Figure 3A. The relationship between the detection distance 308 and capture distance 309 for own moving body 304 is also the same as in Figure 3A, and the relationship between moving body 2 (311) in Figure 3B and target moving body 305 is also the same as in Figure 3A.

[0072] It is assumed that moving body 1 (320) and moving body 2 (311) share information with each other through communication (wireless) while moving. There is a distance restriction on communication, and this distance is called communication limit 312.

[0073] If moving body 1 (320) and moving body 2 (311) deviate from the communication limit 312, the episode will fail. In addition, if the distance between wingmen becomes too close, there is a risk of contact, which is undesirable for the activity, so a proximity limit 313 is set in advance. If moving body 1 (320) and moving body 2 (311) get closer than the proximity limit 313, the episode will fail.

[0074] In the above environment settings, the episode is successful if moving body 1 (320) and moving body 2 (311) continue to track the target moving body 305 while satisfying the conditions for the specified number of steps. Note that although an example in which up to two moving bodies are involved is illustrated, the environment can be set up with similar settings even if there are three or more moving bodies involved.

[0075] The initial conditions for the episode are the position, moving direction, and speed of the target moving body 305, as well as the position of the own moving body 304. Obstacles may be set in the environment, in which case the distance to the boundary of the obstacle is set as a constraint.

[0076] In the reinforcement learning 103, behavior control is learned in the DNN training unit 131 with the environment set as described above. It is considered that the behavior of the moving bodies (the target moving body 305 and the own moving body 304) is restricted depending on the initial conditions of the target moving body 305 and the own moving body 304.

[0077] For example, if the target moving object 305 approaches, the moving object 304 has no choice but to avoid or take a detour, and if it moves away, it will chase after it to catch up. Therefore, the initial condition is called a behavior pattern.

[0078] 4 is a diagram showing an example of a behavior pattern in which one mobile unit tracks a target. A behavior pattern list 400 shows five examples of behavior patterns (behavior patterns 01, 02, 03, 04, and 05) for tracking a target.

[0079] In the figure, the initial position and direction of movement of the target moving object (white circle) are shown with the dotted line, and the movement trajectory of the moving object (black circle) is shown with the dotted line. Also, examples of behavior patterns (behavior patterns 11, 12, 13, 14, and 15) when two moving objects track a target moving object are shown in the behavior pattern list 450 of Figure 5.

[0080] As a result of reinforcement learning, a policy function that has learned a behavior control method for tracking a target moving object, that is, a DNN (deep neural network 10) that outputs an action to be selected for a state quantity, is obtained.

[0081] The trajectory of the own moving body relative to the trajectory of the target moving body according to the above behavior control method can also be obtained. An example of the trajectory of the own moving body relative to the target moving body is shown in Figure 6. Figure 6 is a diagram showing an example of the trajectory when one moving body tracks a target moving body.

[0082] In the illustrated trajectory list 500, there are five trajectories (trajectories 01, 02, 03, 04, and 05), and the trajectories of the target moving body are shown with solid lines and the trajectories of the self-moving body are shown with dashed lines. Each of the trajectories 01 to 05 has the following characteristics: the self-moving body spirals relative to the trajectory of the target moving body (trajectory 01), runs parallel to the target moving body (trajectory 03), runs parallel to the target moving body while rotating (trajectory 04), circles around the target moving body only once (trajectory 02), and even runs parallel to the target moving body overall but is difficult to describe as a clear movement (trajectory 05). All of these are examples of successful episodes in which the conditions for the relationship between the target moving body and the self-moving body are satisfied.

[0083] Furthermore, an example of trajectories when two moving objects are tracked is shown in Figure 7. In the illustrated trajectory list 550, the trajectories of the two moving objects are shown, one with a dashed line and the other with a dotted line. Globally, the trajectories in Figure 7 are similar to those shown in Figure 6.

[0084] In reinforcement learning, a behavior control method for each behavior pattern is learned by the deep neural network 10. Episode data is obtained during the learning process and stored in the episode database 104.

[0085] The behavior control planning device 100 uses the accumulated episode data to perform transfer learning, thereby causing the deep neural network 10 to learn a behavior control method that can respond to any condition for a variety of behavior patterns.

[0086] Transfer learning uses learning patterns that consist of episode failures, successes, and step numbers in the order of episodes obtained during the reinforcement learning learning process. The learning patterns will be explained after explaining the overall process flow from reinforcement learning to transfer learning.

[0087] Fig. 8 shows a flowchart of the processing from reinforcement learning to transfer learning performed in the behavior control planning device 100. The processing in the flowchart in Fig. 8 is divided into reinforcement learning, episode screening, learning pattern design, and transfer learning. Transfer learning is an episode data learning method that uses episode data.

[0088] In reinforcement learning, first, the DNN training unit 131 acquires environmental information, behavioral patterns (initial settings of the target and the own moving body), and parameter conditions for reinforcement learning (801).

[0089] In the process of training a DNN (deep neural network 10, the same applies below), which is a function of a policy and action value for a state quantity, the process is repeated from step 802 to step 805 of the episode repetition. In this embodiment, one episode is processed in one repetition.

[0090] For each repetition, the DNN training unit 131 learns the action selection and state changes for one episode (803). One episode is a continuous process of steps from the initial state until the episode ends with success or failure, and the DNN training unit 131 updates the state by selecting an action based on the state quantity at each step. The DNN training unit 131 also trains the DNN from data on the state quantity, selected action, and reward during the step processing.

[0091] When the processing of the episode is completed, the DNN training unit 131 stores the episode data, including the trajectory data, in the episode database 104 (804). When the reinforcement learning is completed, the DNN training unit 131 also stores the learning results in the episode database 104.

[0092] In steps 1202 to 1207, the episode screening process performed in the episode screening unit 152 will be described.

[0093] The episode screening unit 152 collects (1202) episode data stored in the episode database 104. The episode screening unit 152 may collect all episodes stored in the episode database 104.

[0094] If the number of data items is too large, the episode screening unit 152 may narrow down the search by the attributes of the data files themselves, specifically, the date, the file size, etc. The collection target may also be limited from the perspective of the conditions of the tracking environment targeted by transfer learning (such as the initial positions of the target and the moving object).

[0095] The episode screening unit 152 sorts the data from the perspective of whether the episode is a failure or success (1203). For example, when evaluating the risk of failure with the episode screening unit 152, if the target episode data does not contain information on the state or behavior at the time of failure, it is not possible to evaluate the circumstances in which failure occurs.

[0096] Conversely, if it is desired to evaluate an episode from the viewpoint that even if the episode is successful, a lower risk of failure is better, the episode screening unit 152 will collect successful episodes. Also, for example, when evaluating good, desirable trajectory data, the episode screening unit 152 can only evaluate bad trajectories when evaluating the trajectories of failed episodes.

[0097] In this case, it is necessary to compare multiple trajectory data for successful episodes in the episode screening unit 152. In any case, it is necessary for the episode screening unit 152 to sort the data from the perspective of whether the episode was successful or unsuccessful.

[0098] Furthermore, even if an episode fails, it can be classified as, for example, "captured," "lost," or, if there are two or more mobile objects, "too far away (deviation from communication limit)" or "too close (deviation from proximity limit)," and the number of steps leading to failure can also be compared, so these can also be used as classification targets.The episode screening unit 152 collects (1202) and classifies (1203) episode data according to the purpose of learning in transfer learning.

[0099] To perform the screening process, it is necessary to determine model parameters in the episode screening unit 152. To this end, the episode screening unit 152 sets, by machine learning, failure risk modeling 153 and global trajectory classification model 154, which are models for determining or classifying episode data (1204).

[0100] When screening episode data, the model format, formulas, theory, etc. are already established. Machine learning is a mathematical processing method that, when there is a large amount of data, uses it as variables to determine the values ​​of parameters used in formulas and calculations.

[0101] For example, multivariate analysis such as regression analysis and component analysis, as well as image recognition using deep neural networks and calculating the parameters of speech recognition models are all called machine learning.

[0102] Machine learning parameters include not only parameters for estimating values, but also thresholds for comparison and judgment. In some cases, machine learning also involves using these models to obtain analysis or recognition results from data as input.

[0103] As a result, the parameters of the models for screening (failure risk modeling 153, global trajectory classification model 154) are obtained.

[0104] When the episode screening unit 152 performs screening, it first acquires parameters of the failure risk modeling 153 and the global trajectory classification model 154, which are models for judgment and classification (1205).

[0105] Then, the episode screening unit 152 screens the episode data (1206) using the failure risk modeling 153 and the global trajectory classification model 154. In this embodiment, screening is the process of selecting episode data, that is, determining whether or not to use the data for transfer learning.

[0106] That is, the failure risk modeling 153 selects first learning episode data from the input episode data, and the global trajectory classification model 154 selects second learning episode data from the input episode data and outputs them respectively.

[0107] To achieve this, each model can output a discrete value, and it can be determined whether or not to use the discrete value. Alternatively, the model can output a continuous value, and the episode data can be classified based on a preset threshold for the continuous value to determine whether to use the data for transfer learning. In step 1206, the failure risk modeling 153 and the global trajectory classification model 154 screen the episode data to select the episode data to be used in transfer learning.

[0108] Then, the episode screening unit 152 outputs the episode data for transfer learning (1207). This completes the screening process for episode data.

[0109] For episode data screening, the format, formula, and theory of the model (failure risk modeling153 and global trajectory classification model154) must be determined in advance. Although this method is a creative matter, it is necessary for episode screening to determine the parameters of the above model using machine learning from accumulated episode data.

[0110] The content of these parameters is called model definition 1208. The episode screening unit 152 sets model definition 1208 using the episode data acquired in episode data collection in step 1202, and uses this in machine learning in step 1204, resulting in a flowchart as shown by the dashed arrow in Figure 12.

[0111] Furthermore, if the models (failure risk modeling 153 and global trajectory classification model 154) are mathematically configured as linear polynomials of arbitrary variables, and various methods such as multiple regression analysis, elastic net, and support vector regression can be selected as the parameter calculation method, the process of model definition 1208 itself can be automated. The episode data screening process (105) is also called episode statistical analysis because it handles episode data statistically.

[0112] This concludes the description of the flowchart of the episode screening process.

[0113] Next, the learning pattern design unit 161 first collects (for analysis) (806) episodes from the episode database 104 for each behavior pattern, and also acquires information on the learning process in reinforcement learning.

[0114] In step 807, the learning pattern design unit 161 counts the number of successes and failures and the number of steps in the learning process (in episode order) in which reinforcement learning was successful for the collected episodes.

[0115] Based on the above aggregation results, the learning pattern design unit 161 determines the number of successful or failed samples for each range of episode order in the learning process and the range of the number of steps in the episode, and generates a learning pattern (definition of the relationship between episode order, number of steps, and success and failure) (808).

[0116] In transfer learning 106, first, the conditions acquired in step 801 and further learning patterns are acquired for a plurality of behavioral patterns (809). Transfer learning will be explained below as an episode data learning method.

[0117] The episode setting and selection unit 162 searches the episode database 104 for episode data that falls within the range of the number of steps of success and failure in the order of the learning process specified by the learning pattern, and collects episodes (for learning) (810).

[0118] In episode data learning, in order to train the DNNs of the policy function and the state value function, the transfer learning execution unit 163 performs repeated processing for each episode from step 811 to step 813.

[0119] The transfer learning execution unit 163 trains the deep neural network 10 using one piece of episode data for each repetition (812). The learning here is not reinforcement learning that selects an action, but training of the DNN directly using the episode data. Therefore, the transfer learning execution unit 163 does not select an action for a state, but sets the state quantity, action, and reward of the episode data to the moving object at each step, and trains the DNN with this data.

[0120] The transfer learning execution unit 163 repeats the process for each episode, but may use data determined to be effective for training from the episode data used up to that point as the training data. The transfer learning execution unit 163 terminates the learning when it determines that the learning is successful or when the learning is not successful for all the given episode data.

[0121] This concludes the explanation of the process from reinforcement learning to transfer learning using the flowchart in Figure 8.

[0122] 9 is a diagram showing an example of the success or failure of an episode in the learning process of reinforcement learning and the number of steps. Note that FIG. 9 shows an example in which reinforcement learning was successful.

[0123] In the figure, the horizontal axis represents the number of episodes 901, and learning progresses toward the right side of the figure. The vertical axis represents the number of steps 902. As shown in legend 903, black squares represent failures due to "losing sight," white triangles represent failures due to "capture," and white circles represent plots of successes where the positional relationship with the target remained good.

[0124] In the example shown, if the positional relationship is good (open circle) around 400 steps, the episode is a success. Failures (black squares or open triangles) will continue until learning approaches success, but the number of steps is small at the beginning of learning and increases as learning progresses. Reinforcement learning is then successful by repeating successful episodes of 400 steps in a row.

[0125] Figure 10 shows the characteristics of the learning process in a diagram. In the diagram, failures such as losing sight 1001 and capturing 1002 gradually increase the number of steps from the start of learning. Then, an episode of success 1003 appears, and repeated successes lead to successful learning.

[0126] In actual episodes, the number of steps is not determined for the order of episodes as shown in the line in Figure 10, but there is a variation (range) in the number of steps for the order of episodes, as shown in Figure 9. Therefore, we define a learning pattern by associating a range of the number of steps with a range of the order of episodes.

[0127] An example of a learning pattern is shown in Fig. 11. A learning pattern 1100 includes an episode result 1101, a lower limit 1102 of the number of end steps, an upper limit 1103, and a number of episodes 1104 in one entry.

[0128] The learning pattern 1100 corresponds to the order of episodes from top to bottom, with the first 20 episodes being "lost" and the step number range being set from 1 at the lower limit 1102 to 100 at the upper limit 1103.

[0129] The next 20 steps represent a learning pattern in which episode result 1101 is "captured" with a step number range of 1 to 100, then episode result 1101 is "lost" with 20 steps and a range of 101 to 200, and so on, progressing through the episodes until finally resulting in 40 episodes of successful well-positioned outcomes.

[0130] The episode setting and selection unit 162 searches and acquires episode data that falls within the step number range (1102, 1103) for the corresponding episode order range from the episode database 104, and uses the acquired data in transfer learning .

[0131] This concludes the explanation of the learning patterns.

[0132] It should be noted that even if the number of own mobile objects is two or more, the content of the processing shown in this embodiment is the same. The above is a description of the target for transfer learning by episode data screening and the transfer learning method.

[0133] <System configuration> 1 shows an example of the configuration of a mobile object behavior control planning system according to the present invention.

[0134] An example of the method for planning behavior control of a moving object, or an overview thereof, is as explained above with reference to Figures 1 and 2. That is, the behavior control planning device 100 accumulates episode data that has undergone reinforcement learning in the episode database 104. The behavior control planning device 100 screens the episode data for transfer learning and designs learning patterns. This method then collects episode data and performs transfer learning, and the above series of processes constitutes the behavior control planning method.

[0135] In screening episode data, models are used to improve the performance of the learning process itself and the quality of the learning results. The models used are failure risk modeling153, which determines the risk of failure, and global trajectory classification model154, which classifies global trajectories.

[0136] The series of processes, reinforcement learning, episode screening, learning pattern design, and transfer learning, are as explained above with reference to FIGS.

[0137] FIG. 12 is a block diagram showing an example of the configuration of the behavior control planning device 100 according to the present invention.

[0138] The behavior control planning device 100 is a computer including a processor 11, a memory 12, a storage device 13, and an input / output device 14. Each of the functional units, namely, a simulator 1301, a reinforcement learning / transfer learning unit (episode data learning) 1302, an episode statistical analysis / screening unit 1303, and a learning planning unit (learning pattern design) 1304, is loaded as a program into the memory 12 and executed by the processor 11.

[0139] The processor 11 operates as a functional unit that provides a predetermined function by processing in accordance with the program of each functional unit. For example, the processor 11 functions as a simulator 102 by processing in accordance with a simulation program. The same applies to other programs. Furthermore, the processor 11 also operates as a functional unit that provides each function of multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units. The storage device 13 stores the episode database 104 and the deep neural network 10.

[0140] The simulator 1301 corresponds to the simulator 102 in Fig. 1. The simulator 102 includes a behavior environment setting function 1311, a mobile object behavior processing function 1312, a mobile object state quantity calculation function 1313, and a mobile object state determination function 1314.

[0141] The behavior environment setting function 1311 sets the behavior patterns of the target moving object and the own moving object and the conditions for calculation processing. The moving object behavior processing function 1312 simulates the behavior of the target and moving object for each step in the episode.

[0142] A moving object state quantity calculation function 1313 updates the state quantity of the moving object based on the result of the action. A moving object state determination function 1314 determines the state of the moving object based on the state quantity and determines the reward in reinforcement learning.

[0143] The reinforcement learning / transfer learning unit (episode data learning) 1302 corresponds to the DNN training unit 131 and the transfer learning execution unit 163 in FIG.

[0144] The reinforcement learning / transfer learning unit 1302 combines the functions of reinforcement learning 103 and transfer learning (episode data learning) 106, and the configuration, generation, and training of the DNN are common.

[0145] The reinforcement learning / transfer learning unit 1302 includes an action selection / value function DNN configuration function 1321 that generates a DNN configuration for action selection and value function, an episode processing function 1322 that performs learning processes such as action selection of a moving object for each step in an episode, state update, state determination, and DNN training, a reward setting function 1323 that sets a reward value for the state of the moving object, a DNN training function 1324 that optimizes DNN parameters, and an episode data accumulation function 1325 that stores and manages episode data in the episode database 104. Episode data 1326 can be loaded from the episode database 104 into the memory 12.

[0146] In addition, the reinforcement learning / transfer learning unit 1302 includes, for episode data learning, a learning pattern acquisition function 1327, an episode data collection function 1328, and an episode data learning function 1329 that performs episode processing for episode data learning, which corresponds to the episode processing function 1322 in reinforcement learning.

[0147] The episode statistical analysis and screening unit 1303 includes a failure / success sorting function 1331 that collects episode data and sorts it into failures or successes, a screening function 1332 that processes the screening of the episode data, and a judgment index model function 1333 and a global classification model function 1334 as models for judging and classifying the episode data to be screened.

[0148] The judgment index model function 1333 corresponds to the failure risk modeling 153 in Fig. 1 and judges the failure risk of episode data. The global classification model function 1334 corresponds to the global trajectory classification model 154 in Fig. 1 and is used to evaluate the quality of trajectory data.

[0149] The learning plan (learning pattern design unit) 1304 includes an episode data collection function 1341 that collects episodes for analysis by behavioral pattern, an episode data drawing function 1342 that obtains the failure or success of episodes and the progress of the number of steps during the learning process, a step aggregation function 1343 that aggregates the success or failure of episodes and the number of steps during the learning process, and a learning pattern configuration function 1344 that configures learning patterns.

[0150] The episode data drawing function 1342 is expressed as a GUI (Graphical User Interface), but it means that the relationship between failure or success in episode order and the number of steps can be obtained during the learning process, and it becomes the unit of aggregation.

[0151] The above is a description of an example of a system configuration according to the present invention.

[0152] <Screening model> Next, a model (failure risk modeling 153) for screening that determines the risk of failure in tracking behavior based on the positional relationship between the target moving object and the own moving object will be described.

[0153] Fig. 13 is a diagram showing an example of the positional relationship between a target moving body and a self moving body. The positional relationship between the target moving body and the self moving body will be summarized using Fig. 13. In the figure, the self moving bodies are indicated by black circles 1401, 1402, and 1403, and the target moving bodies are indicated by white circles 1411 and 1412. The symbol p indicates a position, the upper right subscript vcl indicates the self moving body, and tgt indicates the target moving body.

[0154] The subscripts at the bottom right indicate time. For example, black circles 1401, 1402, and 1403 represent the time series of three times t-1, t, and t+1, and the arrows between the black circles 1401, 1402, and 1403 indicate the direction of travel of the moving object. vcl t indicates the velocity at each time.

[0155] The time t corresponds to the step data in the episode. Figure 13 is drawn assuming a two-dimensional space with x and y axes. The relationship between the target moving object and the moving object is isomorphic with the distance D and the direction θ, and is further expressed by adding time information.

[0156] The trajectory may involve actions of a dozen to several dozen steps that require distinguishing between moving straight, zigzagging, oscillating, or curving over a long period of time.

[0157] However, even if the episode proceeds in a straight line, it may or may not succeed. In other words, the characteristics of the trajectory are not directly related to the failure of "capture" or "loss." Therefore, distance and direction are sufficient as features for evaluating the positional relationship of the trajectory.

[0158] The feature quantities are defined below. Distance D and direction θ are expressed by the following equations 3.1 and 3.2. The function atan2 maps a two-dimensional direction vector to radians, and the function degree converts radians to angles in the 360-degree system.

[0159]

number

[0160] Next, relative velocity can be mentioned as a quantity related to the passage of time. Since it is a relative relationship, velocity is treated as a scalar. In relativizing velocity, the velocity vector v at time t is calculated from the position information at time t and t-1. tgt t Let the velocity vector v at time t-1 be tgt t-1 =p tgt t -p tgt t-1 The relative velocity |v rel t | is expressed as the following equation 3.3.

[0161]

number

[0162] The distance change between times is also introduced as a feature. The difference between the distance at time t and the distance at time t-1 of the previous step, that is, the distance change DD t is defined as follows:

[0163]

number

[0164] Both relative velocity and distance change are velocities, but they are separated because the way time is measured is different: velocity until the next time and velocity from the previous time.

[0165] Since the magnitude of each feature value is different, they may be normalized to compare and adjust the coefficients. Normalization of data di with index i of sample data is performed, for example, using the following formula 3.5. Here, D is the set of data di, and minD and maxD are operations to take the minimum and maximum values ​​of set D, respectively.

[0166]

number

[0167] Depending on the purpose of data collection, the feature amount may be normalized, for example, using the standard deviation σ as a standard within a 3σ range of normal distribution, or may be normalized within a range of 0-1.

[0168] The relationship between the feature amount of the trajectory and the failure of the episode will be examined using Fig. 14. Fig. 14 is a diagram showing an example of changes in distance, direction, relative speed, and distance change amount corresponding to the trajectories of successful and failed episodes.

[0169] The feature list 1500 in FIG. 14 is a graph of distances 1511, 1512, 1513, 1514, 1515, directions 1521, 1522, 1523, 1524, 1525, relative velocities 1531, 1532, 5133, 1534, 5135, and distance changes 1541, 1542, 1543, 1544, 1545 for trajectories 1501, 1502, 1503 of successful episodes and trajectories 1504, 1505 of unsuccessful episodes, respectively.

[0170] The horizontal axis of the graph of the feature quantity represents the chronological steps of the episode, and the vertical axis represents the value of each variable. Regarding failures, trajectory 1504 shows an example of "losing sight," and trajectory 1505 shows an example of "capture."

[0171] In the successful episodes, the distance is maintained within a certain range, and periodic changes are observed, particularly in trajectories 1501 and 1502. In particular, in distance 1512, the mobile unit moves so as not to fall below a certain distance. In addition, in trajectory 1503, each variable remains within a certain range so as to maintain a range that does not result in failure.

[0172] In the failed episodes, neither trajectories 1504 nor 1505 show any behavior to correct the distance, and no regular relationship between the features is observed. In the successful episodes, for example, in trajectory 1501, the distance and direction to the other person correspond, and the relative speed decreases as the distance decreases, and increases as the distance increases.

[0173] From these findings, it can be assumed that a good state is maintained by maintaining a certain combination of feature quantities that does not fail. In the failed episodes, the combination of feature quantities deviates from a certain state. Even when looking at a single state quantity, for example, there are distance states that are common to all episodes, and the orientation also changes from -180 degrees to 180 degrees as the moving object rotates, and the relative speed also fluctuates between a minimum and maximum range, making it difficult to identify the cause of failure individually. For this reason, it is necessary to make a judgment based on all of the state quantities together.

[0174] In this embodiment, an index is introduced to determine the risk of failure based on feature quantities. That is, the index value of the action failure with respect to the target moving object is modeled and formulated using the distance, direction, relative speed, and distance change amount.

[0175] The judgment index takes a value other than 0 if the episode is in a failed state. However, the range in which the distance of the moving object approaches or moves away from the threshold of "lost sight" or "capture" is necessary for risk assessment.

[0176] As shown by distances 1514 and 1515 in FIG. 14, the distance tends to change monotonically before failure occurs, so attention is focused on the number of steps of continuous (monotonically increasing, monotonically decreasing) distance change before failure.

[0177] 15A and 15B, in the case of "capture", the range 1602 for that number of steps is set to "-1", and in the case of "lost sight", the range 1604 is set to "1". FIG. 15A is a diagram showing an example of the relationship between the change in distance and the judgment index relative to the number of steps in the episode in the case of "capture". FIG. 15B is a diagram showing an example of the relationship between the change in distance and the judgment index relative to the number of steps in the episode in the case of "lost sight".

[0178] As an example, in FIG. 15A, the range of the number of steps is 20. The number of steps is related to the time per step, the speed, and the distance to the target. As an example of modeling the judgment index, the explanatory variables are distance D, direction θ, relative speed |v rel |, and the distance change DD is expressed by the following linear formula.

[0179]

number

[0180] In the above formula 3.6, parameters a0, a1, a2, a3, and a4 are calculated using multiple regression analysis. Calculating the parameters is also called fitting. Figure 16 shows the results of fitting for 17 episodes of "losing sight" failure, covering a total of approximately 2,000 steps. Figure 16 shows an example of a graph of the training data and predicted values ​​of the judgment index model.

[0181] In the figure, the solid line is the value predicted (also called estimated) by multiple regression analysis of a linear model, and the dashed line is the teaching data, which is set to "1" within the range for determining "loss of sight." When the determination index value exceeds approximately "0.4," it can be determined that there is a high risk of failure due to "loss of sight."

[0182] Figures 17A to 17C show the results of predicting the judgment index values ​​for individual episodes. Figures 17A and 17B show the prediction results for a successful episode, and when the threshold for the judgment index is set to "0.4," there is no deviation from the threshold. On the other hand, Figure 17C shows a failed episode due to "losing sight," and the successive steps leading up to the failure deviate from the threshold of "0.4." In this way, the judgment index can be modeled.

[0183] In the above modeling, we have used a linear model and multiple regression analysis as an example of the fitting analysis, but other models and analysis methods can also be used. As an example, we will use a method called random forest.

[0184] The teaching data are the output judgment index values ​​shown above and the input distance, direction, relative speed, and distance change amount. In the case of Random Forest, the number of trees (graph structure) that make up the model and the tree depth (levels) are set as hyperparameters, but the fitting and prediction processes are obtained using the same procedure, i.e., setting the input and output data, performing the fitting process, providing the desired input data, and outputting the predicted value. The results are compared with those of multiple regression analysis and are shown in Figure 18.

[0185] 18 is a diagram showing an example of the frequency of output for input data in the decision index model using random forest ((b) in the figure) and multiple regression analysis ((a) in the figure). The horizontal axis represents the decision index value, and the vertical axis represents the frequency of data for each step.

[0186] As shown in the legend 1901, the solid lines represent samples (lost_good) 1902 with index value set to "0" in episode data for "lost," the dotted lines represent samples (lost_bad) 1903 with index value set to "1" in episode data for "lost," the dashed lines represent samples (captured_good) 1904 with index value set to "0" in episode data for "capture," and the dashed lines represent samples (captured_bad) 1905 with index value set to "-1" in episode data for "capture."

[0187] The random forest (b) in Figure 18 has more data for the value "0," which indicates low risk, and for the values ​​"1" and "-1," which indicate a risk of failure, making risk identification clearer.

[0188] Even if the risk of failure is small, multiple regression analysis (a) shows a wide distribution in the range of "-0.4" to "0.6," which can be used as a reference for situations where failure is likely. The judgment index value varies depending on the model formula and analysis method, so the decision on which to adopt should depend on the purpose of the judgment. Alternatively, a mixture of methods can be used. There are also various analysis methods, such as support vector regression, elastic nets, and neural network learning, which can be used to evaluate the risk of failure.

[0189] This concludes the explanation of the model for screening to determine the risk of failure for episode data of pursuit behavior (failure risk modeling 153).

[0190] <Orbital Classification> Next, we will explain a model for episode screening (global trajectory classification model 154) that classifies the characteristics of global trajectories across episodes.

[0191] A global trajectory is a trajectory observed over an episode of, for example, 400 steps, and if the condition is that the subject moving object maintains an appropriate distance from the target moving object, a wide variety of trajectories can be obtained that satisfy the goal of episode success.

[0192] For each type of trajectory, there are cases where the trajectory itself reflects characteristics such as whether it is likely to succeed or fail, and in particular, characteristics of good or bad based on factors that are not set as reward conditions when designing reinforcement learning.

[0193] Furthermore, the global trajectory is the average distance between the target moving body and the own moving body and the way in which the positional relationship with the target moving body changes, and it is difficult to directly associate it with the risk of tracking failure due to the characteristics of the positional relationship between the target moving body and the own moving body.

[0194] In this embodiment, features (trajectory features) that can classify the characteristics of a trajectory are provided to determine whether the trajectory is good or bad, and episode data is screened using the trajectory features to correspond to the classification. Note that, as described above, the features of the positional relationship between the target moving object and the own moving object are position and orientation, and in the case of a global trajectory, the trajectory itself is the result of the passage of time, and variables such as speed are not used as features.

[0195] As a result of observing Figures 6 and 7, the characteristics of the global trajectory were spiral, one rotation, parallel running, and parallel running with rotation. The positional relationship between the target and the moving body in these trajectories is shown in Figure 19. Figure 19 is a diagram showing an example of the positional relationship between the target moving body and the moving body itself in the classification 2000 of the moving body trajectory. In addition, there are trajectories that do not fit these characteristics or are a combination of these, making them difficult to classify.

[0196] If we divide the motion of a spiral ((a) in the figure) into translational and rotational components, the rotation is the revolution of the moving body around the target, so we use the word revolution here. An orbit that goes around the target only once is called one revolution, but this rotational component is also revolution, and it is a type of spiral.

[0197] Rotation is rotation without revolving around the target moving body. In other words, the characteristic orbits (a) spiral is called revolution, (b) parallel movement of the target is called parallel movement, and (c) parallel movement with rotation is called rotation.

[0198] Regarding the characteristics of the trajectory, we first consider the change in distance. The average can be used for the overall distance, and since the judgment index itself evaluates the success or failure of an episode based on distance, there is no need to evaluate it as a trajectory feature.

[0199] With revolution, for example, the position fluctuates greatly during one revolution, either closer to or further away from the target moving body. With parallel movement, there is little fluctuation. With rotation, there is a difference in relative speed during rotation, and the distance fluctuates, but the fluctuation is smaller than with revolution. Since the magnitude of the distance fluctuation differs for each orbit, the distance fluctuation (variation) is treated as the standard deviation and is used as an orbit feature.

[0200] Regarding orientation, parallel orbit is an orbit that maintains a constant orientation without fluctuation. In the case of revolution and rotation, the orientation fluctuates within the range of -180 degrees to 180 degrees. Figure 20 explains the difference between rotation and revolution in terms of orientation.

[0201] Ignoring relative translational movement, the position of the target moving object is shown by a black dot, and the moving object itself is shown by a circular orbit. In the figure, (1) is the position where the heading is +90 degrees, (2) is ±180 degrees, (3) is -90 degrees, and (4) is the position where the heading is 0 degrees.

[0202] In the revolution shown in Figure 20(a), the angle gradually changes from +90 degrees to +180 degrees in the range from (1) to (2), and from 0 degrees to +90 degrees in the range from (4) to (1). However, it suddenly changes in a short interval, from -180 degrees to -90 degrees in the range from (2) to (2'), and from -90 degrees to 0 degrees in the range from (3') to (4).

[0203] From (2') to (3') in the figure, there is a small fluctuation around -90 degrees. Overall, there is a lot of data in the direction around -90 degrees. On the other hand, in the rotation of Figure 20 (b), when moving from positions (1), (2), (3), (4), to (1), there is no sudden change to +90 degrees, ±180 degrees, -90 degrees, 0 degrees, and +90 degrees, but rather a gradual change.

[0204] In other words, the orientation data is evenly distributed from 0 degrees to ±180 degrees, resulting in a large variation. In other words, the orientation variation reflects the difference between (a) revolution and (b) rotation, so the orientation variation (variation) is treated as an orbit feature quantity as the standard deviation.

[0205] Regarding the average direction, based on the relationship between distance and direction in Figure 20, the longer the average distance, the larger the average absolute value of the direction, and so it is not useful for comparison between trajectory features. Therefore, it is not necessary to consider it as a trajectory feature.

[0206] From the above, the trajectory patterns are evaluated and classified using the standard deviations of distance and direction as feature quantities.

[0207] In the calculation, both distance and direction are normalized within the range of "-1" to "1". The calculation method is as shown in formulas 3.7 and 3.8 below. The direction unit is written in 360 degrees. Dlower and Dupper are the lower and upper limits of distance, and in tracking, they are the thresholds for "capture" and "lost".

[0208]

number

[0209] For screening classification, the magnitude of distance variation and orientation variation is divided into three levels: large, medium, and small. According to the definition of orbital features, when distance variation and orientation variation are small, it is a parallel orbit without rotation, a parallel orbit with rotation, or a spiral orbit with little revolution. When distance variation is large, it is a spiral, and when orientation variation is large, it is a parallel orbit with rotation.

[0210] Figure 21 shows a list 2200 of orbit classifications based on orbit feature values. Representative orbits are shown in bold. Note that orbits that are difficult to classify often have large variations in orientation.

[0211] By quantifying the distance variation and azimuth variation using the above formulas (3.7) and (3.8) and aggregating them according to the characteristics of the orbit, thresholds for dividing the azimuth variation into large, medium, and small can be obtained. Since both distance and azimuth are normalized and range from "-1" to "1," if the standard deviation is expected to be around one decimal place, two thresholds for large, medium, and small can be prepared, such as "0.1" and "0.3."

[0212] By using these thresholds, episode screening can be used to obtain episodes with global trajectory characteristics that should be the subject of transfer learning.

[0213] This concludes the description of the model for episode screening (global trajectory classification model 154) that classifies the characteristics of global trajectories across episodes.

[0214] <Tracking by multiple moving objects> Below are examples of reinforcement learning, transfer learning, and episode screening for the case where there are multiple mobile objects tracking a target.

[0215] In the following, we will focus on two moving bodies tracking a target moving body. As shown in Figure 3B, in the case of two bodies, a communication limit 312 and a proximity limit 313 are set between the wingmen. This constraint also applies to more than two bodies. Similarly, the target moving body and each moving body may experience episode failures due to "losing sight" and "capture."

[0216] If two wingmen exceed the communication limit 312, they are considered to be too far apart (Too Far), and if they exceed the approach limit 313, they are considered to be too close (Too Close).

[0217] FIG. 22 shows the classification of episodes into failure and success according to the relationship with another mobile object, a wingman M (Mate), when the mobile object fails or succeeds in tracking a target mobile object (Target).

[0218] Moving objects are classified as successfully tracked (OK), lost (LOST), or captured (CAPTURED) when it comes to target moving objects (Target), and as no problem with wingmen (OK), too far (TooFar), or too close (TooClose), resulting in nine different classifications.

[0219] "OO" in Figure 22 is an abbreviation for the combination (OK, OK) where the target moving object is "OK" and the wingman is "OK." If the target moving object is "LOST" and the wingman is "OK," then (LOST, OK) is written as "LO."

[0220] Similarly, the combination (CAPTURED, OK) is abbreviated as "CO", (OK, TooFar) as "OF", (LOST, TooFar) as "LF", (CAPTURED, TooFar) as "CF", (OK, TooClose) as "OC", (LOST, TooClose) as "LC", and (CAPTURED, TooClose) as "CC".

[0221] The self-moving object may fail in relation to the target moving object or in relation to its wingman. It is considered unlikely that the relationship between the target moving object and the wingman will simultaneously exceed the failure threshold at a certain step during an episode.

[0222] The classification of factors that cause the success or failure of an episode as a result of reinforcement learning processing with two moving objects is shown in Figure 23. Figure 23 shows the occurrence status of success or failure of an episode according to the factors in tracking two moving objects.

[0223] In FIG. 23, the frequency of episode failures is high, but failures due to relationships with only the target moving object "LO" and "CO" and failures due to relationships with only the area "OF" and "OC" all occur.

[0224] Although the frequency of "OO" when an episode is successful is low, there are also "LF," "LC," "CF," and "CC" cases where the target vehicle and wingman simultaneously fail. In other words, episode screening must be able to screen for all cases.

[0225] Even if the positional relationship between the wingman and the own moving body is good, if failure occurs due to the positional relationship with the target moving body, the result is "LO" or "CO." In this case, the failure risk judgment index and a global orbit classification model can be obtained using the method described above.

[0226] In the case of "OF" and "OC," where the positional relationship between the target moving body and the own moving body is good but the failure occurs due to the positional relationship with the wingman, the model parameters are determined by the positional relationship between the own moving body and the wingman in the case of the failure risk judgment index.

[0227] In other words, the feature quantities are the distance, direction, relative speed, and change in distance between two moving bodies. The definition of the formula, how to obtain the index value for teaching (handling the range of the number of steps leading to failure), and the calculation method can be modeled using the screening method described above.

[0228] Regarding the classification of global orbital characteristics, the orbits of each mobile unit still have characteristics such as spiral, translation, and translation with rotation. However, if there are two mobile units, each unit has three types: spiral, translation, and translation with rotation, so there are 3 x 3 = 9 types of classifications.

[0229] Trajectory features are divided into three levels for distance and orientation variance, so there are 9 x 9 = 81 possible classifications from the perspective of trajectory features, but screening is possible. If you want to increase the number of screening results, you can adjust the classification threshold for trajectory features or perform reinforcement learning more often to accumulate episode data. In other words, a global trajectory classification model can be obtained by the trajectory classification described above, and the number of classification combinations will increase.

[0230] If the positional relationships of the target moving body, wingman, and own moving body fail simultaneously, such episodes can be identified by the results of the episodes themselves and samples can be acquired. The relationship between the own moving body and the target moving body and the relationship between the own moving body and wingman can be used as data for the model of failure risk judgment index and the model of global trajectory feature classification, respectively.

[0231] Since simultaneous failures in the relationships between the target moving object, wingman, and own moving object are semantically undesirable situations, a decision is made as to whether the data should be avoided for transfer learning or whether it should be used as a bad example for learning.

[0232] The above is an explanation of the modeling of screening when there are two mobile objects. The modeling method is similar when there are three or more mobile objects. Next, the configuration of the learning pattern will be explained.

[0233] The learning process of reinforcement learning for tracking by two mobile units is shown in Figure 24. The difference from the learning process of one mobile unit shown in Figure 9 is that there are more types of failures and a larger number of episodes.

[0234] The tendency of the increase in the number of steps in the learning process of tracking by two mobile objects is similar to that in Figure 9, and therefore a learning pattern can be configured based on the explanation using Figures 10 and 11 above. In this case, the number of episodes is set for each cause of failure, and the number of episodes is set according to the cause. The number of episodes can be set by tallying up the learning processes that led to success. Even if there are three or more mobile objects, the number of episodes can be set according to the tendency of the learning process.

[0235] The above is an explanation of reinforcement learning, transfer learning, and episode screening when there are multiple mobile objects tracking a target.

[0236] As described above, the behavior control planning device 100 of this embodiment accumulates the results of reinforcement learning and performs transfer learning by utilizing episode data, thereby improving the quality of behavioral capabilities and behavior control methods. <Conclusion>

[0237] According to the present invention, it is possible to learn a method of controlling the behavior of a moving object to achieve a goal such as tracking, even in various environments, particularly when the positional relationship between the moving object and the movement and behavior of the target varies.

[0238] By using reinforcement learning for a simple target moving object, we accumulate successful results in achieving the goal, and by using the accumulated episode data for transfer learning (episode data learning), it becomes possible to learn multiple patterns of movement for the target moving object.

[0239] By screening the trajectory data from episodes obtained during the reinforcement learning process for trajectories with a low risk of failure and the global trajectory characteristics of an entire episode, and then using the episode data from the screening results for transfer learning, it is possible to improve learning ability and the quality of the learning results.

[0240] A model for determining the risk of failure (failure risk modeling 153) can be modeled using statistical machine learning, for example, based on the success or failure of the trajectory and action selection in a reinforcement learning episode, allowing for automatic generation of a screening model. In addition, a model of trajectory characteristics is modeled based on the relative positional relationship between the target moving object and the self-moving object (global trajectory classification model 154), allowing for quantitative classification of global trajectory characteristics to be modeled.

[0241] Even if there are two or more self-moving bodies, learning is possible by modeling the failure risk for each failure factor between the target moving body and the self-moving body, and between the self-moving bodies, and also by modeling the classification of trajectories.

[0242] As described above, the behavior control planning device 100 accumulates the results of reinforcement learning (episode data) and utilizes this episode data to improve the learning ability through transfer learning and the quality of the resulting behavior control method. In particular, since the data of the learning results can be used as is without any additional design or settings for learning such as rewards in reinforcement learning, and since items that are difficult to set as rewards, such as trajectory characteristics, can also be used as data for learning, this method can be applied to many different environments.

[0243] The behavior control planning device 100 of the above embodiment can be configured as follows.

[0244] (1) A behavior control planning device (100) having a processor (11) and a memory (12) for planning the behavior of a subject moving body with respect to the behavior of a target moving body, the behavior control planning device (100) including a simulator (102) that receives simulation conditions (111) under which the behaviors of the subject moving body and the target moving body are preset, and outputs the positional relationship between the subject moving body and the target moving body as step data at predetermined time intervals based on the simulation conditions (111) and a preset machine learning model (deep neural network 10); and a data storage unit (104) that acquires the step data, provides the step data to the machine learning model (104) for learning, and stores episode data including a state quantity indicating the positional relationship between the subject moving body and the target moving body, a reward, and a state as a learning result. 4), a screening unit (episode screening unit 152) that selects the episode data stored in the data storage unit based on predetermined conditions, a learning pattern generation unit (learning pattern design unit 161) that aggregates the episode data in the data storage unit (104) and generates a learning pattern (1100) including an order in which the episode data will be used based on the aggregation result, and a transfer learning unit (163) (transfer learning execution unit 163) that provides the episode data selected by the screening unit (152) to a machine learning model (10) as the learning pattern (1100) generated by the learning pattern generation unit (161) to perform learning for each of the episode data.

[0245] With the above configuration, the behavior control planning device 100 can cause a machine learning model (deep neural network) to learn behavior control for achieving a goal such as tracking or pursuit, even when the positional relationship between the target moving object and the target moving object varies. The results of reinforcement learning are then accumulated as episode data, and the accumulated episode data is utilized in transfer learning, thereby improving the quality of the machine learning model. The quality of the behavior control plan can then be improved by utilizing the machine learning model obtained from transfer learning.

[0246] (2) The behavior control planning device according to (1) above, wherein the screening unit (152) outputs first learning episode data selected based on the risk of failure of the episode data and second learning episode data selected based on the characteristics of the trajectories of the own moving body and the target moving body, and the transfer learning unit (163) performs learning using the first learning episode data and the second learning episode data.

[0247] With the above configuration, transfer learning is performed using the first learning episode data selected based on the failure risk assessment result output by the episode screening unit 152 and the second learning episode data set based on the trajectory pattern. This allows the episode screening unit 152 to quantify the failure risk and select the first learning episode data suitable for transfer learning, and also classify the global trajectory features and select the second learning episode data suitable for transfer learning. The transfer learning execution unit 163 can improve the quality of the machine learning model by using the selected first and second learning episode data.

[0248] (3) The behavior control planning device according to (2) above, wherein the screening unit (152) has a first model (failure risk modeling 153) that selects the first learning episode data from the episode data in the data accumulation unit (104) based on a failure risk, and a second model (global trajectory classification model 154) that selects the second learning episode data from the episode data in the data accumulation unit (104) based on trajectory characteristics.

[0249] With the above configuration, the failure risk modeling 153 (first model) quantifies the risk of failure and selects first learning episode data from the episode data, and the global trajectory classification model 154 can select second learning episode data from the episode data using trajectory features.

[0250] (4) The behavior control planning device according to (3) above, wherein the first model (153) includes a judgment index for determining the risk that the behavior of the own moving body will fail with respect to the movement of the target moving body.

[0251] With the above configuration, the failure risk modeling 153 (first model) can select episode data using a judgment index that quantifies the risk that the action of the own moving body in relation to the movement of the target moving body will fail using feature quantities (distance, direction, relative speed, and distance change).

[0252] (5) The behavior control planning device according to (4) above, wherein the first model (153) uses the distance, direction, relative speed, and distance change amount between the own moving body and the target moving body as the determination index for determining whether the behavior toward the target moving body will be a failure.

[0253] With the above configuration, failure risk modeling 153 (first model) can evaluate the behavior of the own moving body in response to the movement of the target moving body using features calculated from the distance, direction, relative speed, and distance change amount between the own moving body and the target moving body.

[0254] (6) A behavior control planning device according to (3) above, characterized in that the second model (global trajectory classification model 154) quantifies and classifies the characteristics of the trajectories of the target moving body and the own moving body.

[0255] With the above configuration, the global trajectory classification model 154 (second model) can classify trajectory patterns based on the feature quantities of the trajectories, which are the trajectories of the target moving object and the own moving object, and select episode data.

[0256] (7) The behavior control planning device according to (6) above, wherein the second model (154) classifies the characteristics of the trajectory into translational motion and rotational motion of the own moving body relative to the target moving body.

[0257] With the above configuration, the global trajectory classification model 154 (second model) can classify episode data by classifying the trajectories of the target moving object and the own moving object into translational motion and rotational motion.

[0258] (8) The behavior control planning device according to (7), wherein the second model (154) classifies the trajectory using the variation or standard deviation of the distance between the target moving body and the own moving body and the variation or standard deviation of the orientation between the target moving body and the own moving body as the feature of the trajectory.

[0259] With the above configuration, the global trajectory classification model 154 can classify episode data by setting thresholds using distance variation and azimuth variation or standard deviation as feature quantities.

[0260] (9) The behavior control planning apparatus according to (1) above, characterized in that the self moving body is plural.

[0261] The above configuration makes it possible to provide a machine learning model in which multiple own moving bodies act toward a target moving body.

[0262] (10) The behavior control planning device according to (1) above, wherein the behavior of the own moving object with respect to the target moving object is pursuit.

[0263] The above configuration makes it possible to provide a machine learning model in which multiple own moving bodies track a target moving body.

[0264] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, with respect to part of the configuration of each embodiment, addition, deletion, or substitution of other configurations can be applied alone or in combination.

[0265] Furthermore, the above-described configurations, functions, processing units, and processing means may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations and functions may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0266] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0267] 100 Action control planning device 102 Simulator 103 Reinforcement Learning 104 episode database 105 Episode Screening 106 Transfer Learning 111 Condition Settings 121 Action Selection and Update Section 131 DNN Training Department 152 Screening Department 153 Failure Risk Modeling 154 Global Orbital Feature Classification 161 Learning Pattern Design Department 162 Episode Settings and Selection 163 Transfer Learning Execution Unit

Claims

1. A behavior control planning device having a processor and a memory for planning a behavior of a moving object with respect to a behavior of a target moving object, a simulator that receives simulation conditions in which the actions of the subject moving body and the target moving body are set in advance, and outputs the positional relationship between the subject moving body and the target moving body as step data at predetermined time intervals based on the simulation conditions and a preset machine learning model; a reinforcement learning unit that acquires the step data, provides the step data to the machine learning model to learn the step data, and accumulates episode data, including a state quantity indicating the positional relationship between the own moving body and a target moving body, an action, and a reward, in a data accumulation unit as a learning result; a screening unit that selects the episode data stored in the data storage unit based on predetermined conditions; a learning pattern generation unit that aggregates the episode data in the data accumulation unit and generates a learning pattern including an order in which the episode data is used based on the aggregation result; a transfer learning unit that provides the episode data selected by the screening unit to a machine learning model using the learning patterns generated by the learning pattern generation unit, and performs learning for each of the episode data; A behavior control planning device comprising:

2. The behavior control planning device according to claim 1, The screening unit outputting first learning episode data selected based on a risk of failure of the episode data and second learning episode data selected based on characteristics of the trajectories of the own moving body and the target moving body; The transfer learning unit A behavior control planning device, characterized in that learning is carried out using the first learning episode data and the second learning episode data.

3. The behavior control planning device according to claim 2, The screening unit a first model that selects the first learning episode data from the episode data in the data storage unit based on a failure risk, and a second model that selects the second learning episode data from the episode data in the data storage unit based on trajectory characteristics.

4. The behavior control planning device according to claim 3, The first model is The behavior control planning device includes a judgment index for determining the risk that the behavior of the own moving body will fail with respect to the movement of the target moving body.

5. The behavior control planning device according to claim 4, The first model is A behavior control planning device characterized in that the distance, direction, relative speed, and amount of change in distance between the own moving body and the target moving body are used as the determination indices for determining whether the behavior of the own moving body relative to the target moving body will be a failure.

6. The behavior control planning device according to claim 3, The second model is A behavior control planning device characterized by quantifying and classifying characteristics of the trajectories of the target moving body and the own moving body.

7. The behavior control planning device according to claim 6, The second model is A behavior control planning device characterized in that the characteristics of the trajectory are classified into translational movement and rotational movement of the own moving body relative to a target moving body.

8. The behavior control planning device according to claim 7, The second model is A behavior control planning device characterized in that the trajectory is classified using the variation or standard deviation of the distance between the target moving body and the own moving body and the variation or standard deviation of the orientation between the target moving body and the own moving body as feature quantities of the trajectory.

9. The behavior control planning device according to claim 1, The behavior control planning device is characterized in that the self moving body is a plurality of moving bodies.

10. The behavior control planning device according to claim 1, A behavior control planning device, wherein the behavior of the own moving body with respect to the target moving body is pursuit.

11. A behavior control planning method in which a computer having a processor and a memory plans a behavior of a moving object with respect to a behavior of a target moving object, comprising: a simulation step in which the computer receives simulation conditions in which the actions of the subject moving body and the target moving body are set in advance, and outputs the positional relationship between the subject moving body and the target moving body as step data at predetermined time intervals based on the simulation conditions and a preset machine learning model; a reinforcement learning step in which the computer acquires the step data, provides the step data to the machine learning model to learn, and accumulates episode data, including a state quantity indicating the positional relationship between the own moving body and a target moving body, an action, and a reward, in a data accumulation unit as a learning result; a screening step in which the computer selects the episode data stored in the data storage unit based on predetermined conditions; a learning pattern generation step in which the computer aggregates the episode data in the data storage unit and generates a learning pattern including an order in which the episode data is used based on the aggregation result; a transfer learning step in which the computer provides the selected episode data to a machine learning model using the generated learning pattern to perform learning for each episode data; A behavior control planning method comprising:

Images