Behavior control planning device, behavior control planning method, and behavior control planning system

Abstract

To plan behavior of an own mobile body in response to behavior of other mobile body against the own mobile body.SOLUTION: A behavior control planning apparatus according to the present invention for planning behavior of an own mobile body with respect to behavior of other mobile body in configured to: set behaviors of the own mobile body and the other mobile body as simulation conditions; simulate, based on the simulation conditions and a predetermined machine learning model, a positional relationship between the own mobile body and the other mobile body to output step data; give the step data to the machine learning model and cause episode data indicating a positional relationship with the other mobile body and including a state quantity, reward, and state to be output as a learnt result and be accumulated in an episode data accumulation unit; analyze the episode data in the episode data accumulation unit to generate a learning pattern; acquire episode data corresponding to the learning pattern to generate learning episode data; and give the learning episode data to the machine learning model for learning.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a behavior control planning device that automatically plans a behavior control method for a moving body such as a ship or vehicle against other moving bodies, particularly in actions such as tracking and patrol that take into account the positional relationship with other moving bodies. [Background technology]

[0002] In order to determine the movement method of a moving object according to the environment, i.e., the content of behavior control, it is necessary to take into consideration various objectives and requirements. In addition to fixed limitations such as a movement range limit and obstacles, the environment also has limitations due to the positional relationship with other moving objects. In particular, regarding the positional relationship with other moving objects, in addition to the objective of avoiding obstacles, there are also objectives such as keeping a constant distance from other moving objects, tracking or following them, or searching for or detecting other moving objects.

[0003] Conventionally, to determine the behavioral control content of an owned vehicle, which is the object of control, a control method is designed based on control theory, but by utilizing reinforcement learning, the behavioral control content can be planned. In recent years, learning capabilities have been improved by methods such as deep reinforcement learning, which is defined as the relationship between state quantities, actions (policies), and action values ​​or rewards, and reinforcement learning is performed by training deep neural networks, and the application of reinforcement learning technology is being considered.

[0004] Patent Document 1 proposes a technology for determining the behavior of a moving object by deep reinforcement learning in order to determine the behavior of the moving object in an environment where obstacles are set.

[0005] Patent Document 2, particularly with regard to deep reinforcement learning technology, proposes a technology in which, when training a deep neural network, the expected learning progress, which is an index of the effectiveness of learning, is calculated and associated with the state quantity, action, and reward data for each step, and then prioritized from the replay memory (accumulation of step data for training) for use in training. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-198012 A [Patent Document 2] JP 2020-47285 A Summary of the Invention [Problem to be solved by the invention]

[0007] When using reinforcement learning technology to determine the behavioral control of a mobile object according to the environment, the problem is how to realize the learning. If the prerequisite conditions for achieving the objective of the mobile object's behavior, such as the environment, are simple, learning is easy, but if the conditions are complex, many training sessions are required for learning, and furthermore, learning itself becomes impossible to achieve.

[0008] In particular, the behavior control plan for a moving body must reflect the positional relationship with other moving bodies (hereafter, "other moving bodies") for the purpose of tracking and searching. In this case, the positions and directions of the other moving bodies may vary with respect to the position and direction of the own moving body, such as approaching, moving away, or crossing, and it is important to be able to take appropriate action in each case.

[0009] In Patent Document 1, the environment is set to include stationary obstacles or paths, and no learning method is shown that can adapt to other diverse environments based on the learning results of one moving object.

[0010] Patent Document 2 describes a technology for making training of a deep neural network more efficient and for making learning successful. In order to learn according to the environment, the setting of the episode itself, which indicates the point in time when success or failure is reached, must be changed for learning, but no ingenuity is shown regarding the setting method.

[0011] The object of the present invention is to plan actions so that a moving body can act in response to various environments, particularly in response to various actions taken by other moving bodies toward the moving body. For this purpose, a technology is provided that utilizes episode data resulting from reinforcement learning for different actions of other moving bodies, and learns an action plan for the moving body using the episode data. [Means for solving the problem]

[0012] The present invention is a behavior control planning device having a processor and a memory, which plans the behavior of a self moving body in response to the behavior of another moving body, and includes a simulation condition setting unit which sets the behavior of the self moving body and the other moving body as simulation conditions, a simulator which outputs the positional relationship between the self moving body and the other moving body as step data at predetermined time intervals based on the simulation conditions and a predetermined machine learning model, a reinforcement learning unit which acquires the step data and supplies the step data to the machine learning model to output episode data including a state quantity, a reward, and a state indicating the positional relationship between the self moving body and the other moving body as a learning result, and accumulates the episode data in an episode data accumulation unit, an episode analysis unit which analyzes the episode data accumulated in the episode data accumulation unit to generate a learning pattern, and an episode data learning unit which acquires the episode data corresponding to the learning pattern from the episode data accumulation unit to generate learning episode data, and supplies the learning episode data to the machine learning model for learning. Effect of the Invention

[0013] According to the present invention, even in various environments, particularly in cases where the behaviors of other moving objects have diverse relationships with respect to the moving object, it is possible to learn the behavior of the moving object to achieve a purpose such as tracking or detection.

[0014] The details of at least one implementation of the subject matter disclosed herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosed subject matter will become apparent from the following disclosure, drawings, and claims. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram illustrating an example of a process performed by the action control planning device according to the embodiment of the present invention. [Diagram 2] FIG. 1 illustrates an embodiment of the present invention, showing an example of processing of a behavior control plan by reinforcement learning, episode analysis, and episode data learning. [Diagram 3] FIG. 2 is a diagram illustrating an example of a flowchart of a behavior control plan for a moving object according to the embodiment of the present invention. [Figure 4] 1 is a diagram illustrating an example of a configuration of a behavior control planning device for executing a behavior control plan for a moving object according to an embodiment of the present invention. [Diagram 5] FIG. 2 illustrates an embodiment of the present invention and shows an example of the behavioral environment of a moving object and another moving object during tracking. [Figure 6] FIG. 2 is a diagram illustrating an example of the behavioral environment of a mobile unit and other mobile units on patrol according to an embodiment of the present invention. [Figure 7A] FIG. 2 is a diagram illustrating an example of a configuration of a patrol area according to an embodiment of the present invention. [Figure 7B] FIG. 11 illustrates an example of a behavior pattern of another moving object according to the embodiment of the present invention. [Figure 8] FIG. 13 illustrates an embodiment of the present invention, showing an example of configuration of a policy function and a state value function using a deep neural network. [Figure 9A] FIG. 11 illustrates an example of an embodiment of the present invention, showing an example of a failure in constructing episode data. [Figure 9B] FIG. 11 illustrates an embodiment of the present invention and shows a successful example of the configuration of episode data. [Figure 10] FIG. 1 is a diagram illustrating an example of a flowchart of a series of reinforcement learning procedures according to an embodiment of the present invention. [Figure 11A]FIG. 2 illustrates an embodiment of the present invention and shows an example of episode data for one reinforcement learning process. [Figure 11B] FIG. 2 illustrates an embodiment of the present invention and shows an example of episode data for one reinforcement learning process. [Figure 12] FIG. 1 illustrates an example of a plot of step numbers for a number of successful reinforcement learning training runs, illustrating an embodiment of the present invention. [Figure 13] FIG. 13 is a diagram illustrating an example of the relationship between the number of steps for each failure and success episode in a reinforcement learning process, according to an embodiment of the present invention. [Figure 14] FIG. 13 illustrates an example of the range of occurrence of failure and success episodes in a reinforcement learning process according to an embodiment of the present invention. [Figure 15] FIG. 11 illustrates an example of a learning pattern according to the embodiment of the present invention. [Figure 16] FIG. 1 illustrates an example of a graph of a learning pattern as a plot of step number versus reinforcement learning process, according to an embodiment of the present invention. [Figure 17] FIG. 11 illustrates an example of a sampling result of episode data corresponding to a learning pattern according to the embodiment of the present invention. [Figure 18A] FIG. 13 illustrates an embodiment of the present invention, showing an example in which the sampling results of episode data are replaced. [Figure 18B] FIG. 13 illustrates an embodiment of the present invention and shows another example in which the sampling results of episode data are replaced. [Figure 19] FIG. 1 is a diagram illustrating an example of a flowchart of a processing procedure for learning episode data according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0017] To plan the behavior control of an autonomously moving body (self-moving body), reinforcement learning technology is used by simulating the movement of the self-moving body and other moving bodies. In reinforcement learning, the body acts one step at a time from the initial positional relationship between the self-moving body and other moving bodies until it succeeds or fails in achieving a specific goal. Note that a step indicates a simulation (or learning or recognition) process that is executed at a specified time interval.

[0018] An episode is a period from the start of a simulation of moving the self moving object and other moving objects to the point at which the action of the self moving object succeeds or fails. The action control planning device uses a machine learning model to learn action control that achieves a goal by repeating episodes. For reinforcement learning, the relationship between state quantities, actions (policies), and state values ​​is defined in a deep neural network (machine learning model), and in the action control planning device, the deep neural network is trained by reinforcement learning and episode data learning using data for each step of the state, action, and reward for each episode.

[0019] The steps constituting an episode include, for example, a state quantity, an action, and a state value for each predetermined time interval. The state quantity can include the position and speed of the own moving body and other moving bodies, the action can include the course and speed of the own moving body, and the state value can include a reward for each step. The other moving bodies can be one or more moving bodies, and multiple moving bodies can also be included.

[0020] The relationship between state quantities, actions (policies), and state values ​​is trained by optimization that minimizes the loss function, so step data that makes optimization more efficient is prioritized and selected. Also, episodes, which are the chronological sequence of steps, are the sequence until the completion of learning. The loss function is a function that evaluates the prediction accuracy of a deep neural network, and for example, a well-known or publicly known technique such as a regression model can be applied.

[0021] If the behavior pattern of other moving objects relative to the self moving object is simple or uniform, learning can be achieved by applying reinforcement learning. However, the behavior patterns of other moving objects are expected to be diverse, and it cannot be assumed that there is only one uniform pattern.

[0022] Therefore, the behavior control planning device accumulates episode data when reinforcement learning is performed for each of various behavior patterns of the other moving bodies and the own moving body. The episode data is data consisting of an arrangement of state quantities, actions (measures), and state values ​​for each step. The episode data can also include multiple step data. In this embodiment, the learning process for each episode until learning is successful is analyzed from the episode data accumulated in advance.

[0023] As a result of the analysis, the behavior control planning device sets a learning pattern, which is the order of episodes from the start of learning, including failure episodes and success episodes according to the number of steps. Note that the behavior control planning device can set a learning pattern according to the number of steps included in one episode.

[0024] Deep neural networks, which use state quantities as input and actions (policies) and state values ​​(rewards) as outputs, and in which the relationships between them are defined, are trained using pre-stored episode data. Even using pre-stored episode data, deep neural networks can be trained for each step and episode. During this training, random action selection, as used in reinforcement learning, is not performed. Learning by training deep neural networks using this episode data is called episode data learning.

[0025] In episode data learning, the deep neural network can be trained efficiently according to the learning pattern settings, and the efficiency can be improved in terms of, for example, completing learning in a smaller number of episodes, and the simulation results by the deep neural network 10 of the learning results are more likely to be successful.

[0026] The behavior control planning device utilizes episode data obtained as a result of reinforcement learning of individual simulation results, sets learning patterns in advance for multiple simulation results (e.g., capture, loss, good position, etc.), and extracts episode data that matches the learning patterns from the pre-accumulated episode data to perform episode data learning.

[0027] The simulation results of the other moving body are not limited to the above three types, and any type of simulation result may be set in advance. The simulation results can be determined, as described below, from the positions of the other moving bodies at the time when the positional relationship between the moving body itself and the other moving bodies is changed from the initial value and a predetermined end condition is satisfied (the end time of the episode).

[0028] As a result, the behavior control planning device learns the selection of behavior (measure) for state quantities for multiple behavior patterns of other moving objects using a deep neural network. That is, the deep neural network learns a behavior control plan for the own moving object according to the behavior patterns of other moving objects. Completion of learning is determined by attempting to process episodes for all multiple simulation results, and each attempt being successful.

[0029] As described above, by utilizing episode data, which is the result of reinforcement learning for each simulation result, the number of successful and unsuccessful steps for each episode leading to successful reinforcement learning is analyzed to set a learning pattern.

[0030] The behavior control planning device can plan a behavior control method for the own moving body for a plurality of behavior patterns by learning episode data about the results of a plurality of simulations. The behavior control planning device of this embodiment makes it possible for a deep neural network to efficiently learn behavior control in the complex relationship between the behaviors of the own moving body and other moving bodies, which is difficult to achieve by conventional reinforcement learning alone.

[0031] <Processing Overview> An overview of processing of a behavior control plan by reinforcement learning, episode analysis, and episode data learning according to an embodiment of the present invention will be shown with reference to an example of processing of the behavior control planning device in Fig. 1. First, the overall flow of processing performed by the behavior control planning device 105 will be explained with reference to the flowchart in Fig. 2.

[0032] In order to plan behavior control of the own moving body for various environments and behavior patterns of other moving bodies, the behavior control planning device 105 first performs reinforcement learning of behavior control of the own moving body for individual simulation conditions (201). In this reinforcement learning, the deep neural network 10 advances action selection by one step at a time according to the state of each moving body, and trains the deep neural network 10, which is a function that determines the relationship between state quantities, measures, and state values, in the processing of one episode until the purpose of the own moving body is achieved and the result is success or failure.

[0033] The behavior control planning device 105 repeats the processing of the episodes and completes the learning by obtaining a function that always or sufficiently achieves the objective. The behavior control planning device 105 accumulates (202) the state quantity, action, and reward data for each step of the episode until the learning is completed as the learning result by all episodes for the sequence of the reinforcement learning process in the episode database 122.

[0034] The behavior control planning device 105 (see FIG. 1) accumulates learning results for various behavior patterns. In addition, for the same learning pattern, not just one learning result but multiple learning results are accumulated. Data for one episode of these learning results is called episode data.

[0035] The behavior control planning device 105 collects episode data by carrying out reinforcement learning for each individual condition on the episode data stored in advance in the episode database 122, and analyzes the episodes of the learning process by tallying up the types of episodes (success and failure) and the number of steps of the episodes in the order of the learning process (203).

[0036] In this way, the behavior control planning device 105 collects the successes and failures and the number of steps thereof in the order of episodes in the learning process, and sets a learning pattern.

[0037] Then, the behavior control planning device 105 collects episode data from the episode database 122 according to the settings of the learning patterns for multiple conditions. Then, the behavior control planning device 105 performs episode data learning (204) by training the state quantity, the policy, and the state value function (deep neural network) using the episode data in the order of the learning process. Upon completion of each episode in the order of the learning process, the completion of learning is determined by determining whether the mobile object's purpose has been achieved (simulation result).

[0038] The behavior control planning device 105 accumulates the learning results of the episode data learning (205). The episode data in the episode data learning is already accumulated in the episode database 122, and it is not necessary to accumulate the episode data itself. It is sufficient to accumulate a list of the episode data used in the learning process and the determination result of the goal achievement of the moving object for each episode (type of episode) in the episode data learning.

[0039] The above is an explanation of the overall flow of processing of the behavior control plan by reinforcement learning, episode analysis, and episode data learning, using the flowchart in Figure 2. Through the above processing, episode data output by reinforcement learning is accumulated in the episode database 122, the deep neural network 10 is trained by the episode data according to the learning pattern set in the episode analysis, and the training results are accumulated as a list of episode data used in the episode data learning and the judgment result of the goal achievement of the moving object for each episode.

[0040] The diagram showing an example of processing in Fig. 1 shows the necessary processing functions, the processing order, and data transfer with arrows in the outline of the processing of this embodiment. The vertical direction in the diagram is classified into a reinforcement learning unit 101, an episode analysis unit 102, and an episode data learning unit 103, and the horizontal direction in the diagram is classified into an operator 104 and a behavior control planning device 105 as the subject of processing. The processing by the operator 104 means a processing performed manually by a user or the like.

[0041] The reinforcement learning unit 101 processes condition settings (simulation condition setting unit) 111 for simulating the spatial environment in which the own moving body moves, the behavior patterns of other moving bodies, and the behavior settings such as the moving direction and speed of the own moving body. The reinforcement learning unit 101 simulates the behavior of the other moving bodies and the own moving body using a simulator 112 under conditions (simulation conditions) specified by an operator 104. The simulator 112 performs the simulation using a deep neural network (machine learning model) that has already completed learning. The simulator 112 may use any known or publicly known technology, and a simulator 112 according to the type of moving object may be used. The types of moving objects do not need to be the same; for example, the own moving object may be a ship and the other moving object may be a submarine, or the own moving object may be an aircraft and the other moving object may be a ship.

[0042] In the simulator 112, state update 113 and action selection 114 are performed for each step (predetermined time interval). In particular, in the action selection 114, not only is an action selected from a function of state quantity and policy, but also a process of randomly selecting an action is performed for reinforcement learning.

[0043] An episode of the self-moving body's goal success or failure includes step-by-step processing, and the learning device 115 processes the training of the deep neural network 10, which is a function of the state quantity, policy, and action value, using data on the state quantity, action, and reward of the step.

[0044] Data on state quantities, actions, and rewards for each step of the simulation results are stored in the memory (storage device) of the action control planning device 105, and training data is acquired from the simulator 112 by a training data acquisition function 116 and processed in deep neural network training 117. Training of the deep neural network 10 progresses so that the goal of the own moving body is achieved, and when reinforcement learning is completed, episode data on the processing process in the reinforcement learning unit 101 is accumulated in an episode database (episode data accumulation unit) 122.

[0045] In the episode analysis unit 102, episode data of the reinforcement learning process is collected from the episode database 122 for each type of episode and each simulation result of another moving object, and a learning pattern is generated by the episode analysis function 121.

[0046] The episode data learning unit 103 performs a process of setting a learning pattern 131 for learning episode data, using a learning pattern generated for the simulation results of a plurality of other target moving objects.

[0047] Then, episode selection 132 is performed based on the learning pattern set in learning pattern setting 131, and episode data learning 133 is performed. In episode data learning as well, learning device 115 is used to train deep neural network 10.

[0048] Incidentally, the condition setting 111 and the learning pattern setting 131 are shown in FIG. 1 on the assumption that they are manually set by the operator 104. However, when the conditions to be set are regularly defined or the method for setting the learning pattern is determined in advance, these may be processed by the behavior control planning device 105.

[0049] The above is an explanation of the processing performed by the action control planning device 105 in FIG.

[0050] <Behavior control plan> 1 shows an example of a behavior control plan for a moving object according to the present invention.

[0051] An example of a method for planning action control of a moving object is shown in the flowchart of Fig. 3. The process of the flowchart is divided into a reinforcement learning unit 101, an episode analysis unit 102, and an episode data learning unit 103. Each process is described as being executed by the action control planning device 105.

[0052] In the reinforcement learning unit 101, first, simulation conditions such as the environment, the behavior patterns of other moving objects, and the behavior of the own moving object are accepted, and simulation conditions for parameters for simulation and reinforcement learning are acquired (301). The simulation conditions can be set by the operator 104 as shown in FIG.

[0053] In the process of training the deep neural network 10, which is a function of the policy for the state quantity and the action value, the process is repeated from step 302 to step 304 of the episode repetition for reinforcement learning. One repetition corresponds to the processing of one episode.

[0054] The behavior control planning device 105 makes the deep neural network 10 learn the behavior selection and state change for each step of one episode for each repetition (303). The processing of one episode is a process for each step from the initial state of the moving object to the success or failure of whether the purpose of the moving object is achieved.

[0055] The behavior control planning device 105 trains the deep neural network 10 using the state quantity, behavior, and reward data at each step and at the end of each episode. In the behavior selection during training, not only the behavior selection by the deep neural network 10 but also random selection (random behavior selection) is performed.

[0056] Furthermore, at the end of processing one episode, the behavior control planning device 105 tests (attempts) the processing of the episode by behavior selection using the trained deep neural network 10, and if a predetermined purpose is achieved, the type of the episode is determined to be successful and learning is completed. In this case, the completion of learning may be determined, for example, when the episode passes the test five times in a row.

[0057] When the learning of the deep neural network is completed, the behavior control planning device 105 performs an episode trial (305) using the deep neural network of the learning result to confirm that the learning is completed and the behavior of the moving object. Alternatively, a record such as a log can be left so that it can be confirmed that the learning is completed.

[0058] Then, in database accumulation (306) of reinforcement learning results (episodes), the reinforcement learning unit 101 accumulates data on state quantities, actions, and rewards for each step of all episodes in the learning process in reinforcement learning as episode data for the learning process in the episode database 122. When accumulating episode data, the reinforcement learning unit 101 accumulates the data in the episode database 122 together with the simulation results of other moving objects and the type of episode.

[0059] In this flowchart, the reinforcement learning unit 101 shows a single flow, but actually executes reinforcement learning on the behavior patterns of other moving objects, including the same behavior pattern, multiple times.

[0060] In the episode analysis unit 102, first, for each simulation result of another moving object, episodes (for analysis) (307) are collected from the episode database 122. At this time, information on the order of the learning process in reinforcement learning is also acquired.

[0061] In collecting episode failure / success processes of successful learning (308), the episode analysis unit 102 collects and lists the successes and failures of episodes and the number of steps of episode data for the learning processes in which the deep neural network 10 that performed reinforcement learning was successful in the simulation. Then, the number of failures and successes and the number of steps of each episode data are tallied in the order of the episode data in the learning process for each interval in the episode order.

[0062] The behavior control planning device 105 sets the number of successful and unsuccessful samples (episode data) in the episode order and the range of the number of steps based on the compilation result of the episode data, and configures a learning pattern (309).

[0063] Although this flowchart shows the episode analysis unit 102 performing one flow, the episode analysis unit 102 also performs episode analysis on different behavior patterns of other moving objects in the same manner.

[0064] In the episode data learning unit 103, first, conditions (episode conditions) are acquired (including the learning pattern) (310) including the learning pattern in addition to the same conditions as in step 301. Note that the conditions (episode conditions) may also include the designation of simulation results of multiple other moving objects.

[0065] Then, for the purpose of episode learning, episode collection (for learning) is performed (311), in which episode data corresponding to the episode type (success or failure) and the number of steps in the order of the learning process specified in the learning pattern is searched and acquired from episode database 122.

[0066] In the episode data learning unit 103, the process of training the deep neural network, which is a policy for state quantities and a function of action values, is performed for the collected episodes by repeating the process for each episode from step 312 to step 314. The process for each episode can be repeated until a preset termination condition (e.g., the number of episodes that have been processed) is met.

[0067] The behavior control planning device 105 learns the behavior selection and state data of one episode for each repetition (313). The learning here is not so-called reinforcement learning, which may randomly select actions, but learning that directly uses episode data. One episode data is a process for each step from the initial state of the moving object to the success or failure of the episode.

[0068] Episode data learning unit 103 proceeds with the processing for each step of the episode data, but does not use simulator 112 to select the action of the moving object, update the state, or calculate the state quantity and reward.

[0069] The episode data learning unit 103 sets the state quantity, action, and reward of the episode data for each moving object for each step. This data is stored in a memory, and the deep neural network is trained using the state quantity, action, and reward data for each step and at the end of the episode.

[0070] At the end of processing one episode, the episode data learning unit 103 tests the processing of the episode by action selection using the trained deep neural network. At this time, the episode processing for the simulation results of all other moving objects is tested, and if all are successful, the learning is completed. In addition, it is also possible to determine the completion of learning according to a termination condition, such as if the episode is successful five times in a row for each simulation result.

[0071] When learning is complete, the episode data learning unit 103 performs an episode trial (315) using the learning results to confirm that learning is complete and the behavior of each moving object. Alternatively, it outputs a record such as a log so that the results can be confirmed. The episode data learning unit 103 performs a trial on all of the simulation results of other moving objects specified in the learning.

[0072] The episode data learning unit 103 accumulates (316) the results of the episode data learning in a database. Note that the episode data used in the learning is not registered. The learning process of the episode data learning, such as the number of episodes required to complete the learning and the episode data used, is accumulated in the episode database 122.

[0073] The above is an explanation of an example of a method for planning behavior control of a moving object, using the flowchart in Fig. 3. Through the above process, the episode data output by the deep neural network 10 is stored in the episode database 122 as a result of reinforcement learning, and the episode analysis unit 102 collects the successes and failures of the episodes and the number of steps in the episode data, sets the number of successes and failures of the episodes and the range of the number of steps, and generates a learning pattern as described below.

[0074] The episode data learning unit 103 selects episode data corresponding to the learning pattern from the episode database 123, and performs episode data learning 133 using the learning device 115 to train the deep neural network 10. Then, the episode data learning 133 performs trials on the trained deep neural network 10, and then accumulates data on the learning process, such as the number of episodes, in the episode database 122.

[0075] <Configuration of the behavior control planning device> Next, an example of the function of the behavior control planning device 105 in Fig. 4 will be described. Fig. 4 shows the function of the behavior control planning device 105 that realizes the contents described in the flowchart in Fig. 3.

[0076] The behavior control planning device 105 is composed of a computer including a processor 11, a memory 12, a storage device 13, and an input / output device 14. The input / output device 14 includes input devices such as a keyboard, a mouse, or a touch panel, and an output device such as a display.

[0077] Functional units of a simulator 400, a reinforcement learning / episode data learning unit 410, and an episode analysis unit 430 are loaded as programs into the memory 12 of the behavior control planning device 105 and executed by the processor 11. Note that data used by each program (behavioral environment database 418, episode database 419, etc.) may be stored in the storage device 13.

[0078] In the reinforcement learning / episode data learning unit 410, the processing related to training of the deep neural network (DNN) 10 is common to the reinforcement learning unit (corresponding to 101 in FIG. 1) and the episode data learning unit (corresponding to 103 in FIG. 1), so the reinforcement learning unit and the episode data learning unit are combined into one functional block (410). However, since the episode data learning unit uses a learning method different from reinforcement learning, a functional block of the episode data learning unit 420 is placed inside the functional block (410).

[0079] The function of the simulator 400 (corresponding to 112 in FIG. 1) is used by the reinforcement learning / episode data learning unit 410. The simulator 400 is a function for proceeding with the processing for each step in the processing of learning (303) of action selection and state change for one episode in FIG.

[0080] The simulator 400 has a behavior environment setting function 401 , a mobile object behavior processing function 402 , a mobile object state quantity calculation function 403 , and a mobile object state determination function 404 .

[0081] The behavior environment setting function 401 is a function for setting the initial positions of the own moving body and other moving bodies, obstacles, and other environments, and determines the initial state by this function.

[0082] The mobile object behavior processing function 402 selects the behavior of the mobile object for each step and performs movement processing. The mobile object behavior processing function 402 also performs the movement of other mobile objects, thereby updating the state for each step. The mobile object behavior processing function 402 corresponds to the state update 113 and the action selection 114 in FIG. 1.

[0083] When the state is updated for each step, the state quantity is calculated by the moving object state quantity calculation function 403. The state quantity is information required for the moving object to achieve a preset objective, such as the distance to another moving object, the distance to an obstacle, the direction of the other moving object, the speed of the other moving object, and the like.

[0084] The moving body state determination function 404 determines the state of the moving body at that step from the calculated state quantity. For example, the state is determined based on the distance between the moving body and the other moving body, and symbolic state names such as a state where the moving body is in a good position where it can detect the other moving body (good position), a state where the other moving body is too far away to be detected and lost, and a state where the moving body can be detected but is too close and is found and captured by the other moving body. Note that a ship is shown as an example of the moving body in this embodiment.

[0085] The moving object state quantity calculation function 403 uses this judgment result to determine whether the episode has failed if the moving object has lost sight of another moving object or been captured, or whether the episode has succeeded if the predetermined number of steps have been met and the moving object is in a good position (good position), and ends the episode. The reward value is set based on this state.

[0086] The above functions realize the process of step 303 in Fig. 3 (learning behavior selection and state changes). Note that executing a simulation is necessary for reinforcement learning, but is an unnecessary function for episode data learning itself. However, the simulator 400 generally holds data related to the own moving object, other moving objects, and the environment, and has a function to access episode data as a program, and the function of the program can also be used for episode data learning.

[0087] The frame of the reinforcement learning / episode data learning unit 410 shows each function for realizing learning. Among these, functions related to reinforcement learning are particularly shown. The reinforcement learning / episode data learning unit 410 has components of a policy / state value function configuration function 411, an episode processing function 412, a reward setting function 413, a DNN training function 414, a DNN training data sampling function 415, a policy / state value calculation function 416, an episode data accumulation function 417, a behavioral environment database 418, and an episode database 419.

[0088] A policy / state value function configuration function 411 configures a deep neural network 10 that inputs a state quantity and outputs a selected action, i.e., a policy, and a state value, before the start of reinforcement learning.

[0089] The episode processing function 412 processes one episode step by step from the initial state, and proceeds with the processing until the end of the episode depending on whether the moving object achieves the goal. The episode processing function 412 is a function that performs the processing of step 303 in Fig. 3, and is a function that performs various controls such as calling each function of the simulator 400, the reward setting function 413, the DNN training function 414, the DNN training data sampling function 415, and the policy / state value calculation function 416.

[0090] The reward setting function 413 is a function that determines a reward depending on the state obtained as a result of the mobile object state determination function 404 .

[0091] The DNN training function 414 trains the deep neural network 10 by optimizing the parameters of the deep neural network 10 using data on state, action, and reward for each step.

[0092] The DNN training data sampling function 415 samples data for training the deep neural network 10 from data for each step in an episode, particularly for efficient optimization. The sampling of episodes may be performed not only for one episode, but also over multiple episodes.

[0093] The policy / state value calculation function 416 is a function that obtains a policy and a state value, which are the output of the deep neural network 10, by providing a state quantity, which is the input of the deep neural network 10. Obtaining a policy means selecting an action, and the mobile object action processing function 402 determines the action by calling this policy / state value calculation function 416.

[0094] An episode data storage function 417 stores the episode data from step 306 in FIG.

[0095] The behavioral environment database 418 is a database in which data on the behavioral environment of the own moving object is stored.

[0096] Episode database 419 is a database in which episode data is stored, and corresponds to episode database 122 in FIG.

[0097] Before describing each function of episode data learning unit 420, each function of episode analysis unit 430 will be described in the order of the flowchart in FIG.

[0098] The episode analysis unit 430 is composed of an episode data collection function 431, an episode data drawing function 432, a step-by-step episode tallying function 433, and a learning pattern configuration function 434. The episode analysis unit 430 corresponds to the episode analysis unit 102 in FIG.

[0099] The episode data collection function 431 collects episode data in the learning process of reinforcement learning for each simulation result of another moving object.

[0100] The episode data drawing function 432 draws, for example, the layout or a map of the environment, the own moving body, and other moving bodies in an episode for each step. It also includes drawing of changes in state quantities for each step. This function allows the operator 104 to manually check the contents of the episode. In this embodiment, the generation of image data is regarded as drawing.

[0101] The step-by-step episode tallying function 433 lists the successes and failures of episodes and the number of steps in the learning process, and tallies the number of successes and failures and the number of steps in the episode data in episode order.

[0102] The learning pattern configuration function 434 configures a learning pattern by tallying up the number of episode types (success and failure) and the range of the number of steps in the episode data for each interval in the episode order.

[0103] A description will be given of each function constituting episode data learning unit 420 in reinforcement learning / episode data learning unit 410. Episode data learning unit 420 is composed of a learning pattern acquisition function 421, an episode data collection and listing function 422, and an episode data learning function 423.

[0104] A learning pattern regarding the simulation results of other moving objects is acquired by the learning pattern acquisition function 421. This function is used in step 310 in Fig. 3, and at this time, various conditions are also acquired by the behavioral environment setting function 401. The learning pattern acquired by the learning pattern acquisition function 421 is the learning pattern output by the learning pattern configuration function 434 of the episode analysis unit 430.

[0105] The episode data collection and listing function 422 collects episode data that meets the episode conditions (episode conditions) specified in the learning pattern from the episode database 419. The episode data collection and listing function 422 lists the collected results as data to be used in episode data learning.

[0106] The episode data learning function 423 processes the episode data learning. The episode data learning function 423 corresponds to the episode processing function 412 in reinforcement learning, and has a function of controlling each function. Before the start of episode data learning, the deep neural network 10 is configured by the policy / state value function configuration function 411.

[0107] The reinforcement learning / episode data learning unit 410 stores data for each step of the episode data in the memory 12, and uses a DNN training function 414 and a DNN training data sampling function 415 to train the deep neural network 10. When the episode data learning is completed, the episode data accumulation function 417 stores the learning result in the episode database 419.

[0108] Each of the above functions is loaded into memory 12 as a program and executed by processor 11. Processor 11 operates as a functional unit that provides a specified function by executing processing according to the program of each functional unit. For example, processor 11 provides behavioral environment setting function 401 by executing processing according to a behavioral environment setting program. The same applies to other programs. Furthermore, processor 11 also operates as a functional unit that provides each function of the multiple processes executed by each program. Computers and computer systems are devices and systems that include these functional units.

[0109] The above is an explanation of an example of the functions of the moving object action control planning device 105 with the configuration in FIG.

[0110] <Moving object behavior> 2 shows examples of the purpose and environment of the action of a mobile object in the present invention.

[0111] The purpose of the action of the own mobile unit is to track (pursue) other mobile units, or to patrol to detect the intrusion of other mobile units into a set area.

[0112] FIG. 5 shows an example of the behavioral environment of a moving object and other moving objects during tracking.

[0113] An X coordinate axis 502 and a Y coordinate axis 503 for taking coordinates of a plane that is a two-dimensional space are set in a frame 501. Note that although the action area is shown as a plane, it may be a three-dimensional space including a depth direction.

[0114] The mobile object 504 can move freely in space by moving straight, turning, and changing speed. The initial position and initial speed are set in advance as simulation conditions, but the direction and speed change depending on the action selected for each step.

[0115] Another moving object 505 moves from an initial position at a constant speed in one direction 506. The initial position, speed, and direction are set in advance, and this combination forms a behavior pattern of one other moving object 505.

[0116] A detection distance 507 is set for the own moving body 504, and another moving body 505 is detected inside a circle 508 having a radius of the detection distance 507. At the start of tracking, that is, at the initial position, the other moving body 505 is set so as to be detectable by the own moving body 504.

[0117] Outside the circle 508, the own moving body 504 cannot detect the other moving body 505, and this becomes a range 509 in which it will be lost. If the own moving body 504 enters the range 509 in which it will lose sight of the other moving body 505, the simulation results in the own moving body 504 being lost, the objective of tracking the other moving body 505 is not achieved, and the episode type is a failure.

[0118] Also, if the own moving body 504 gets too close to the other moving body 505, it is detected and captured by the other moving body 505. This state is considered to be a capture, and the purpose of continuing the pursuit is not achieved, so the episode type is a failure. For this reason, a capture range 510 is set for the own moving body 504, and when the other moving body 505 enters inside the circular capture range 510, it is determined that the other moving body 505 has been captured.

[0119] If the vehicle 504 does not lose sight of or is captured within a specified or set number of steps from the start of the simulation, it is determined that the goal of tracking the vehicle 504 has been achieved, and the episode type is determined to be successful.

[0120] This concludes the explanation of the example of the behavioral environment in tracking shown in Figure 5.

[0121] FIG. 6 shows an example of the behavioral environment of the own mobile unit and other mobile units during patrol.

[0122] A patrol area 601 is set in advance. A host moving object 602 can move in a free direction 605 from an initial state (position, speed) within the patrol area 601. The moving direction and speed of the host moving object 602 change depending on the action selected for each step.

[0123] A self moving object 602 has the ability to detect other moving objects, and if the other moving object is within the range of a circle 604 having a radius equal to a detection distance 603, it is determined that the other moving object has been detected.

[0124] In the example of Fig. 6, the other moving bodies have a variation in behavior, such as another moving body 606 that invades the patrol area 601 in the vertical direction, and another moving body 607 that invades in the horizontal direction. The invading positions are various or random. When the own moving body 602 enters the detection distance 603 of the invading other moving bodies 606 and 607, it is determined that the detection has been successful.

[0125] If another moving object passes through the patrol area 601 without being detected, it is considered that the detection has failed. Since the object of the own moving object 602 is to detect another moving object, if the detection is successful, a positive reward is given, and if the detection is unsuccessful, a negative reward is given.

[0126] As for an episode, the type of the episode can be classified as a success or failure depending on whether the detection of the behavior of one other moving object is successful or unsuccessful. Also, the number of steps of an episode may be determined in advance, and reinforcement learning may be performed with a large number of other moving objects. In this case, the behavior of the own moving object 602 that can obtain a large reward can be obtained.

[0127] Regarding the behavior of other moving bodies, if the occurrence location is random, general patrol behavior of the own moving body 602 will be obtained, but if the occurrence location is specified in advance, patrol behavior of the own moving body 602 in response to specific intrusion tendencies of other moving bodies will be obtained.

[0128] The patrol behavior depends on the size of the patrol area 601 and also on the intrusion behavior patterns of other mobile objects into the patrol area 601, and is a condition of the environment.

[0129] Figure 7A shows an example of a patrol area and a coordinate system. Figure 7B shows an example of the configuration of a patrol area and the behavior patterns of other moving objects. In Figure 7A, a patrol area 701 is a rectangular area, with the center (centre of gravity) as the origin, and X-axis 702 and Y-axis 703 are taken. The size of the area is set by the range Lx 704 in the X direction and the range Ly 705 in the Y direction. If half of the ranges Lx 704 and Ly 705 are taken as hLx 706 and hLy 707, the coordinate values ​​are ±hLx and ±hLy.

[0130] An example of the behavior pattern of another moving object is shown in Fig. 7B. The area expanded by adding an offset to the outside of the patrol area 701 is set as another moving object occurrence area 710. Another moving object occurs outside the patrol area 701 and inside the other moving object occurrence area 710.

[0131] The patrol area 701 is a rectangle, and other moving objects enter from the top 711, bottom 712, right 713, and left 714 of each of the four sides of the area in movement directions 715, 716, 717, and 718.

[0132] The number of other moving bodies entering the patrol area 701 can be determined by, for example, allowing one moving body to enter for each preset step, or by keeping the number of other moving bodies in the patrol area 701 constant.

[0133] The above is an explanation of the operating environment of the own mobile unit and other mobile units during patrol, as shown in Figures 6, 7A, and 7B.

[0134] The above is a description of the purpose and environment of the action of the mobile object in the present invention.

[0135] <Reinforcement Learning> The details of the reinforcement learning in the present invention will be described below. The reinforcement learning corresponds to the processing of the reinforcement learning unit 101 in Fig. 1 and Fig. 3 and the reinforcement learning / episode data learning unit 410 in Fig. 4.

[0136] Reinforcement learning is generally based on the Markov Decision Process (MDP) model, which selects an action in a certain state to update the state, and this action or state is accompanied by a value (or state value) that is expressed as a reward.

[0137] Reinforcement learning methods include Q-learning, which selects actions based on a table value of action value, DQN (Deep Q-learning Network), which uses a deep neural network to function as an action value and selects actions from the action value, and A3C (Asynchronous Advantage Actor-Critic), which uses a deep neural network for the policy (action selection) function and state value function.

[0138] There are other algorithms that use deep neural networks, but all of them use reinforcement learning to train models using states, actions, and rewards. This invention will be described with the focus on A3C.

[0139] Reinforcement learning is a method of optimizing a policy function and a state value function so that an agent selects the optimal action for a state in a repetition of episodes. An agent is the subject of learning, and in this invention, for example, it is a self-moving body. The agent selects an action for a state at each step in an episode to obtain the next state.

[0140] In reinforcement learning, rewards are given for the results of action selection or for the state. If the episode termination condition is met, the episode ends. Reinforcement learning ends when the correct action selection is made to obtain a good state for the state by repeating the episode.

[0141] A3C stands for Asynchronous Advantage Actor-Critic, and is characterized by asynchronous processing, advantage evaluation, actor-critic strategies, and a state value function model. Asynchronous processing processes the agent's episodes in each of the individually prepared environments.

[0142] In this case, each agent trains a single policy function and state value function to make learning more efficient. Advantage is the difference between action value and state value, and is a quantity that represents the goodness of action selection. Actor-critic means that the action selection is calculated as the actor and the state value is calculated as the critic, using separate models.

[0143] An example of the configuration of the policy function and state value function of the present invention using a deep neural network is shown in Figure 8. The deep neural network receives a state quantity 801 as input, and outputs an action 802 and a state value 803.

[0144] A network from a state quantity 801 to an action 802 is a policy function, and a network from a state quantity 801 to a state value 841 is a state value function. The state quantity 801 is a distance 821 from the own moving body to another moving body, a direction 822 of the other moving body as seen from the own moving body, and a speed 823 of the own moving body.

[0145] The action 802 is a turning action of the own moving body, right 831, left 832, forward 833, and a speed change action of deceleration 834, acceleration 835. The state value 803 is itself a numerical value, and only the state value 841 is an item.

[0146] The deep neural network is configured as a multi-layer structure of an input layer 811, an intermediate layer 812, and an output layer 813, and the intermediate layer 812 in particular has multiple layers. Note that in Fig. 8, the nodes are drawn as white circles, but not all the nodes and layers are drawn, and the connections are represented by dotted lines, and the collection of nodes is represented by dotted circle enclosures.

[0147] Each node is a function, and the lines connecting the nodes represent the input / output relationships. The state quantity 801 directly corresponds to the input layer 811. Each node of the output layer 813 corresponds to each item of the action 802 and the state value 803.

[0148] The nodes in the output layer 813 corresponding to each item of the action 802 are defined as activation functions using a Softmax function so that events are divided in terms of probability so that the sum is 1. The output function corresponding to the state value 841 is defined as a Linear function. Note that the nodes in the intermediate layer 812 are defined as Leru functions, Sigmoid functions, etc.

[0149] Data for each step in an episode is stored in the memory 12, and the deep neural network is trained at a predetermined timing for each step, for example, once every 10 steps, or at the end of the episode. The data for each step is the state, action, and reward.

[0150] 9A and 9B show the configuration of a series of step data (episode data) of state quantities, actions, and rewards in an episode. Fig. 9A shows an example of step data 900A in the case of failure, and Fig. 9B shows an example of step data 900B in the case of success.

[0151] Each of Figures 9A and 9B is episode data that constitutes one episode. The items in each figure can be roughly divided into steps 901, state quantities 902, actions 903, rewards 904, and states 905. The states 905 are not used in training the deep neural network, but are required for reference and episode analysis.

[0152] The state quantity 902 is a distance 911, a direction 912, and a speed 913, and the numerical values ​​are normalized to a range of 0 or more and 1 or less. The actions 903 are forward 914, right 915, left 916, deceleration 917, and acceleration 918, and since one action is selected in one step, one item is set to 1 and the others are set to 0. Note that the distance 911 indicates the distance from the own moving body to the other moving body, the direction 912 indicates the direction of the other moving body as seen from the own moving body, and the speed 913 indicates the speed of the own moving body.

[0153] The reward 904 is -1 if the episode fails, 1 if it succeeds, and 0 otherwise. There are three types of states 905: good position, lost, and captured.

[0154] We will now explain the training process for the policy function and state value function. A deep neural network is a network that takes the state quantity s as input, outputs the policy, which is the action output, as π(s), and outputs the state value as V(s).

[0155] Let θ be the parameter of the activation function defined for the nodes in the network. The policy is also called a stochastic policy, and can also be written as π(a|s), the probability of taking action a in state s. Training the policy function and state value function means optimizing the parameter θ of the deep neural network so that a relationship that predicts the relationship between state s, action a, next state s', and reward r can be obtained from the data.

[0156] The state value function V(s) is expressed by the following equation (1) under the policy π(s).

[0157]

number

[0158] Here, E means the expected value of the policy π(s), which is the subscript. The discount rate γ is a coefficient for correcting the next state value (future value) to the present value. The value of the discount rate γ is 0.99 as an example, but it can be adjusted as a setting for reinforcement learning. The value of the policy is expressed as the expected value in the distribution ρ of state s by the following equation (2).

[0159]

number

[0160] Equation (2) is called the discounted reward function. The policy gradient theorem, expressed in equation (3), applies to the change in the discounted reward function with respect to the parameter θ of the deep neural network.

[0161]

number

[0162] Here, ∇ means the gradient (first order partial differential with respect to the basis). The expected value is the range of state s over the distribution ρ of policy π, and action a over the range of policy π for state s. The action value function Q(s,a) and advantage function A(s,a) are as follows and can be calculated using data.

[0163]

number

[0164] By optimizing the network parameter θ to maximize the reward function, we can obtain a relationship between the policy and state value that gives a good reward for the state. If the negative of the reward function is treated as the policy loss, the policy loss is minimized.

[0165] In addition to the policy loss, since it is desirable for the action value and state value to match, there is also a value loss, which means the magnitude of the absolute value of the advantage. It is also desirable for one policy to be determined for a state, and since the policy is determined probabilistically, a regularization term modeled with entropy is also used in optimization. Therefore, the loss function is the policy loss L π , Loss of value L v , regularization term L reg and minimize the loss function.

[0166]

number

[0167] where c v , c reg is a coefficient.

[0168] The policy loss is given by the definition of the discounted reward function as follows, where n is the number of data points used in training.

[0169]

number

[0170] The value loss is the square of the advantage function.

[0171]

number

[0172] The regularization term is obtained by calculating the entropy H(π(s)).

[0173]

number

[0174] where n action is the number of actions.

[0175] Gradient information is used to train neural networks through optimization calculations, and the training itself is the same as training deep neural networks in deep learning.

[0176] The above is an explanation of the processing steps for training the policy function and the state value function.

[0177] We explain how to sample data at each step to make training of deep neural networks more efficient.

[0178] In general, in optimization calculations using data, if the objective function value from a sample deviates from the optimization objective, the adjustment range of the model parameters becomes larger, and the optimization becomes more efficient.

[0179] For example, when minimizing an objective function, the parameter values ​​can be adjusted to significantly reduce the objective function value by performing optimization calculations using data that increases the objective function value. By increasing the change in the objective function value, the parameter adjustment range can be increased.

[0180] The objective function is the loss function expressed in equation (6), and minimizing this is the purpose of optimization. The policy function, value function, and regularization term, which are the terms in the objective function, are calculated using data on the state quantity s and action a, as shown in equations (7), (8), and (9). These calculations use the policy function π(s) or π(a|s) and the advantage function A(s,a).

[0181] The policy function is a function that selects an action from the state quantity in a deep neural network, and can obtain the value of each action from the state quantity s. The advantage function is defined in equation (5) and uses the action value function Q(s, a) in equation (4).

[0182] The action value function Q(s, a) is calculated from the state value function V(s') and the reward r in equation (4). The state value function is a function that calculates the state value from the state quantity using a deep neural network, and its value can be obtained from the state quantity s.

[0183] In the action value function, the next state s' is a step in the episode data, and is calculated using the state quantity s of the step in the next row. For the step in the last row, the state value function V(s') should be set to zero. From the above, the loss function value can be calculated for each step from the state quantity s, action a, and reward r.

[0184] Since a loss function value is obtained for each step, by arranging the step data in order of increasing loss function value and sampling the state, action, and reward data of the step with the largest value, training data that can make training more efficient can be obtained.

[0185] The number of data required for training can be sampled from many step data in order of the magnitude of the loss function value, or it can be sampled stochastically from limited step data, allowing overlaps. Also, since a small loss function value reflects the characteristics of the parameters of the deep neural network that has been trained up to that point, step data with a small loss function value can also be mixed in for sampling.

[0186] This concludes the explanation of the method for sampling step data to make training of a deep neural network more efficient. The main processing unit is the action control planning device 105.

[0187] An example of a series of reinforcement learning processing procedures will be described with reference to the flowchart of FIG.

[0188] Steps 1001 to 1018 are a repetition of the episode.

[0189] In processing an episode, first, the states of the environment, the own moving body, and other moving bodies are reset (1002).

[0190] Steps 1003 to 1014 are repetitions of steps in one episode.

[0191] The reinforcement learning / episode data learning unit 410 selects an action for the host moving object (1004). This process is performed by inputting the current state quantity to a policy function.

[0192] Then, the reinforcement learning / episode data learning unit 410 executes the selected action and updates the state quantity (1005). The reinforcement learning / episode data learning unit 410 uses the functions of the simulator 400 to calculate the state quantity.

[0193] The reinforcement learning / episode data learning unit 410 performs a NG judgment regarding failure of the episode based on the state quantity of the own moving object (1006). In the case of a tracking example, the judgment of losing sight of or capturing corresponds to failure.

[0194] If the NG decision 1006 is yes (NG), the reinforcement learning / episode data learning unit 410 sets a flag indicating the episode has failed (1007).

[0195] Next, the reinforcement learning / episode data learning unit 410 determines the number of steps performed in the episode (1008). If the number of steps performed exceeds a predetermined number, it is determined that the episode has ended and the process proceeds to step 1009, otherwise the process proceeds to step 1011. If the failure flag has already been set in step 1007, the determination in step 1008 is unnecessary, and the determination result will be no (not the number of end steps). In that case, the episode end flag is also set.

[0196] If the step count determination (1008) is yes (the number of processed steps is the specified end step number), the reinforcement learning / episode data learning unit 410 sets a success flag (1009). This means that the episode type is a success because the processing was able to continue for the specified number of steps, rather than a failure. In the case of a tracking example, the continuation of a good position state corresponds to success.

[0197] In the case of the patrol example, if the NG judgment (1006) is OK (no), the number of detections is counted as detection of other moving objects. If the NG judgment (yes), the number of losses is counted as a result of failed detection. In either case, processing is performed to remove other moving objects from the environment. The step number judgment (1008) is a judgment of only the number of completed steps, and there is no particular need to set a success flag (1009).

[0198] If the step number determination (1008) is yes, the reinforcement learning / episode data learning unit 410 sets an episode end flag (1010).

[0199] The reinforcement learning / episode data learning unit 410 sets training data for training the deep neural network (1011). This step 1011 is to prepare a set of data from the memory 12 for training, and also includes processing such as sampling to make training more efficient.

[0200] Then, the reinforcement learning / episode data learning unit 410 inputs the training data set in step 1011 above to the deep neural network for training (1012).

[0201] The processes in steps 1011 and 1012 do not necessarily have to be performed for every step, but may be conditionally branched so as to be performed every few steps or at the end of an episode.

[0202] Next, the reinforcement learning / episode data learning unit 410 judges the end of the episode (1013). If the episode end flag is set, the reinforcement learning / episode data learning unit 410 judges the episode end value. If the judgment is no (episode end flag is not set), the process returns to step 1003 and repeats the above process.

[0203] If the answer to the episode end determination 1013 is yes (the end flag is set), the process leaves the loop from step 1003 to step 1014 and proceeds to step 1015.

[0204] When the reinforcement learning / episode data learning unit 410 finishes repeating the steps in the episode, it outputs the episode data and stores it in the episode database 122 (1015).

[0205] Next, the reinforcement learning / episode data learning unit 410 determines the number of consecutive successes of the episode (1016). If the reinforcement learning is to end after one successful episode, the reinforcement learning / episode data learning unit 410 may determine the success of the episode as yes. For example, if the end condition is 10 consecutive successes, the determination result is no if there are fewer than 10 successes, but the unit adds 1 to the number of consecutive successes and proceeds to step 1017. If the episode is a failure, the number of consecutive successes is set to 0, the determination result is no, and the unit proceeds to step 1017.

[0206] Next, the reinforcement learning / episode data learning unit 410 determines the number of episodes that have been performed (1017). Reinforcement learning is repeated for each episode, and steps 1001 to 1018 are repeated indefinitely unless the episode success count determination (1016) is yes. Therefore, the reinforcement learning / episode data learning unit 410 sets an upper limit on the number of episodes, and if the number of times an episode has been performed exceeds the upper limit, it determines yes and exits the episode repetition loop. In this case, reinforcement learning fails.

[0207] Finally, to end the reinforcement learning, a learning end process is performed (1019). The reinforcement learning / episode data learning unit 410 outputs a log of information on episodes in the reinforcement learning process and the like.

[0208] This concludes the description of an example of a series of processing steps of reinforcement learning using the flowchart in Fig. 10. This concludes the description of the contents of reinforcement learning in the present invention.

[0209] <Episode Analysis> An example of the contents of episode analysis in the present invention will be described below.

[0210] The episode analysis corresponds to the contents of the episode analysis unit 102 in FIG. 1 and FIG. 3 and the episode analysis unit 430 in FIG.

[0211] Episode analysis is a series of processes in which the episode data accumulated in the episode database 419 is analyzed individually for the order of success and failure of episodes in the reinforcement learning process, with respect to the behavior patterns of other moving objects, to construct a learning pattern.

[0212] 9A and 9B, each episode includes data on state quantity 902, action 903, reward 904, and state 905 for each step 901 constituting one episode. As a result of one reinforcement learning, one or more pieces of episode data, in fact a large number of pieces of episode data, are stored in the episode database 419 for the reinforcement learning process.

[0213] The episode data in the following explanation is an example of data about reinforcement learning of ship tracking.

[0214] Episode data for one reinforcement learning process is shown in Fig. 11A and Fig. 11B. Fig. 11A shows an example where an episode was successful due to reinforcement learning, and Fig. 11B shows an example where an episode was unsuccessful due to reinforcement learning. In the illustrated graph, the horizontal axis is episode (number) 1101, and the vertical axis is the number of steps at the end of the episode 1102. As shown in legend 1103, the plot shapes are black squares (filled squares) for loss of sight, white triangles (open triangles) for capture, and white circles (open circles) for good position. In addition, if the simulation result is loss of sight or capture, the episode type is failure, and if the simulation result is good position, the episode type is failure.

[0215] In the graph of Figure 11A, up to about 150 episodes in the reinforcement learning process, the episodes fail without exceeding 100 steps and are lost. There are also some cases of capture. After 150 episodes, the episodes fail, but the number of steps starts to increase.

[0216] After 200 episodes, we can see successful episodes at a good position where the number of steps reaches 400. After 250 episodes, the type of episode is almost always successful, and successful episodes are repeated until the reinforcement learning is successful at 270 episodes.

[0217] As an example of failure, in the graph of Figure 11B, the number of steps hardly exceeded 100 up to episode 100, but successful episodes appeared from episode 100 to episode 200. There were many successful episodes between episodes 200 and 300, but there were also many failed episodes in which the simulation results were rejected.

[0218] Around episode 300, there were only captures, and after episode 350, there were only failures due to capture and losing sight. There were no consecutive successful episodes, and at episode 550, the reinforcement learning failed.

[0219] For the reinforcement learning process of reinforcement learning in which the episode was successful, plots of the number of steps for episode failures and successes for the same simulation results of other moving objects are shown in Figure 12. Note that in the following figures, as in Figure 11A above, if the simulation result is loss of sight or capture, the episode type is failure, and if the simulation result is a good position, the episode type is failure.

[0220] In Figure 12, the plot of episode success for achieving 400 steps shows that the number of episodes ranges from 150 to 350, with learning being completed in 250 episodes on average.

[0221] The number of steps in episodes in which the target was lost or failed to be captured also gradually increases along with the number of episodes in the reinforcement learning process. However, the increase is about 150 steps up to 100 episodes, and on average, it increases by about 50 steps.

[0222] Moreover, looking at the plot, there is a large variance in the types of episodes in the process leading to success in reinforcement learning. This is thought to be because in reinforcement learning, action selection is random, which introduces a large variance into the results of training the deep neural network 10. If training goes through ideal episodes, it is thought that the number of episodes can be reduced.

[0223] In examining the learning patterns, episodes of loss and failed capture, and successful episodes of 400 consecutive steps in a good position are organized as a simple line graph of the number of episodes and the number of steps, as shown in Figure 13. In the figure, line 1301 represents loss, and dashed line 1302 represents capture. It is assumed that these two lines have the same tendency. Shaded line 1303 represents successful episodes. It is drawn using the average values ​​of the plots of success and failure in Figure 12.

[0224] Figure 14 shows the ideal range of failure and success episodes in the reinforcement learning process in terms of the number of episodes when the deep neural network 10 that has performed reinforcement learning is successful in the simulation. In the figure, solid black 1401 indicates loss of sight, diagonal shading 1402 indicates capture, and dotted shading 1403 indicates a good position for 400 consecutive steps.

[0225] The episode data learning unit 420 (122) assumes that reinforcement learning will be completed in 150 episodes, and allocates 40 episodes up to episode 150 as successful episodes. Since there are many cases of loss of sight and capture failures in a small number of steps in the early stages of the reinforcement learning process, 20 episodes each in the range of step numbers 0 to 100 are allocated as episodes of loss of sight and capture failures. Then, 20 episodes each in the range of step numbers 100 to 200 are allocated as episodes of loss of sight and capture failures. The range of 110 episodes from the 80th episode to the successful episode is allocated to episodes of loss of sight and capture from step numbers 200 to 300 and from step numbers 300 to 400.

[0226] The types of simulation results (lost, captured, good position) that the episode data learning unit 420 (122) assigns to the above-mentioned ranges of step numbers and ranges of episode numbers can be assigned at a preset ratio (or number).

[0227] The learning pattern shown in Fig. 15 is obtained from the successful and unsuccessful episodes in the reinforcement learning process shown in Fig. 14 and their ranges of step numbers. Learning pattern 1500 is a data structure defined by episode result 1501, lower limit 1502 and upper limit 1503 of the end step number range, and episode number 1504. Episode result 1501 is the state at the end of the episode. The end step number range is specified by lower limit 1502 and upper limit 1503.

[0228] The learning pattern 1500 shown in Fig. 15 is plotted with the number of episodes on the horizontal axis and the number of steps on the vertical axis as shown in the graph of Fig. 16. In Fig. 16, the step number range is set to the center value of the range.

[0229] The above shows an example of constructing a learning pattern for the behavior patterns (capture, loss, good position) of other moving objects using episode data collected from the episode database 419. The episode analysis unit 430 in the behavior control planning device 105 in Fig. 4 is processed by an episode data collection function 431, a step-by-step episode aggregation function 433, and a learning pattern construction function 434.

[0230] Episode analysis is a process of determining the range of the number of steps for the number of episodes from the start of reinforcement learning, which is a component of the learning pattern, by evaluating the average and variance of all episode data in Figure 12, and can be automated by computation.

[0231] In episode data learning using a learning pattern, the behavior control planning device 105 collects episode data from the episode database 419. A method of collecting episode data will be described below. Note that episode data is collected for each simulation result of another moving object.

[0232] This processing is the content of episode collection (for learning) 311 in the flowchart of FIG. 3, and is processing by the episode data collection and listing function 422 in the configuration of the behavior control planning device 105 in FIG.

[0233] Figure 17 shows the results of sampling episode data corresponding to the learning pattern shown in Figure 15. Figure 17 shows that episode data samples within the range of step numbers are included for each range of episode numbers in the order of episode results.

[0234] For example, episodes 1 to 20 show samples of losses (squares in the figure) with 100 or fewer steps, episodes 21 to 40 show samples of captures (triangles in the figure) with 100 or fewer steps, and episodes 111 to 140 show samples of successful episodes with 400 or fewer steps.

[0235] The distribution of episodes of the reinforcement learning process in Figure 17, including loss, capture, and good positioning, is quite different from the distribution of the successful reinforcement learning examples shown in Figure 11A. This is because there is no variation in the range of episodes in the horizontal axis (number of episodes) direction for the simulation results of loss, capture, and good positioning in Figure 17.

[0236] Therefore, in order to provide variation within the episode range in sampling by a learning pattern such as that shown in Figure 17, there is a method of performing replacement between episodes (positions) in the reinforcement learning process. Replacement between episodes is, for example, an operation of exchanging the 10th episode and the 20th episode since the start of reinforcement learning. A replacement in which the selection of the episode to be replaced is decided randomly is called random replacement.

[0237] FIG. 18A shows an example of random permutation with 50 permutations, and FIG. 18B shows an example of random permutation with 100 permutations, showing the distribution of the number of episodes and the number of steps in the reinforcement learning process. In FIG. 18A and FIG. 18B, episodes after the 130th episode are not subject to permutation. In this way, it is possible to give variation to the sampling by the learning pattern within the episode range. Note that although the word permutation is used in the explanation, it may also be called shuffling.

[0238] The above is an explanation of the method for collecting episode data using learning patterns.

[0239] The above is an explanation of an example of the contents of the episode analysis in the present invention.

[0240] <Episode data learning> An example of the contents of episode data learning in the present invention will be described below.

[0241] The episode data learning corresponds to the contents of episode data learning unit 103 in FIGS. 1 and 3 and episode data learning unit 410 in FIG.

[0242] Episode data learning is a method of learning to select actions according to the situation by collecting episode data according to learning patterns based on the simulation results of other moving objects, and training a deep neural network by directly using the step data of each episode (episode data).

[0243] In episode data learning, the first step is to obtain the conditions, similar to reinforcement learning. However, in order to learn episode data, there are multiple simulation results for other moving objects, and the corresponding conditions for the other moving objects and learning patterns are obtained.

[0244] The collection of episode data is as described above for the learning patterns of this embodiment.

[0245] In episodic data learning, episodic data is directly used to train a deep neural network using data from steps in an episode. The purpose of episodic data learning is to select an action according to the state quantity, and the learning model is based on the Markov decision process (MDP) model, similar to reinforcement learning.

[0246] In other words, as described above in the reinforcement learning section, the policy function and state value function are defined in a deep neural network, and episodes are repeated, and in each episode, the deep neural network is trained using data for each step.

[0247] For the construction and training of the deep neural network, the calculation method expressed by the above formulas (1) to (10) is used. Also, the sampling of the step data is the same as that of reinforcement learning.

[0248] The difference between episodic data learning and reinforcement learning is that reinforcement learning mixes random selection and selection using a policy function when selecting the agent's action at each step, and obtains state, action, and reward data by updating the state based on the action.

[0249] In contrast, episodic data learning involves directly acquiring state, action, and reward data, for example by reading them from a file. Episode data learning does not have the exploration and exploitation features of reinforcement learning.

[0250] In this respect, episode data learning itself does not require a simulator, or in some cases equipment or experiments in a real environment. However, meaningless episode data that does not lead to successful learning is not used, and episode data obtained in advance during reinforcement learning is used.

[0251] This method makes learning more efficient by selecting the episode data required for learning as a learning pattern, and also enables learning of different environments or simulation results of other moving objects, where learning would be more difficult.

[0252] An example of a series of processing steps for learning episode data will be described with reference to the flowchart in Fig. 19. The main subject of the processing in Fig. 19 is the action control planning device 105.

[0253] First, the episode data learning unit 410 collects (1901) episode data from the episode database 419 (122).

[0254] The range from step 1902 to step 1918 is a repetition for each episode. The repetition of an episode is a repetition of episode data collected from the episode database 122 in the order of the reinforcement learning process based on the designation of the learning pattern.

[0255] In addition, this episode data includes a mixture of episode data from multiple simulation results of other moving objects. For example, if there are two simulation results and each has the same number of episodes, one episode is arranged for each simulation result. If the number of episodes in one simulation result is twice as many as the other, the two episodes of the first simulation result are arranged together with one episode of the other simulation result.

[0256] In repeating an episode, the episode data learning unit 410 first resets the states of the environment, the own moving body, and other moving bodies (1903).

[0257] The range from step 1904 to step 1909 is repeated for each step in one episode.

[0258] The episode data learning unit 410 acquires the state quantity, action, and reward of the current step (1905). In reinforcement learning, as shown in Fig. 10, in the repetition of a step, there is an action selection, an action execution, and a failure or success judgment based on the state, but in episode data learning, the data of the step is simply read.

[0259] Next, the episode data learning unit 410 uses the DNN training function 414 to train the deep neural network with the data from the step (1906).

[0260] The episode data learning unit 410 judges whether the episode has ended (1907). Since the number of steps is determined by the episode data, if it is the last step, the episode has ended. On the other hand, if it has not ended, the step is updated (1908) and the above process is repeated for the next step.

[0261] Episode data learning unit 410 performs a test for completion of learning in steps 1910 to 1915 every time the processing of an episode is completed.

[0262] The process from step 1910 to step 1915 is repeated for all the simulation results.

[0263] The range from step 1911 to step 1914 is a repetition of the specified number of trials. If the number of trials is, for example, five, five trials are performed for one simulation result, and if five consecutive successes are obtained, it means that the learning for that simulation result was successful.

[0264] First, the episode data learning unit 410 executes a specified episode using the deep neural network of the training result (learning result) (1912). This is the same process as step 305 in Fig. 3, in which an action is selected using a policy function for each step from the initial state, and the state is updated.

[0265] Next, episode data learning unit 410 determines whether the episode is a failure or success (1913). If it is a failure, the answer is yes, and the repetition from step 1910 to step 1915 is immediately terminated. If it is not a failure (no), the repetition continues. If all repetitions are successful, the repetition from step 1913 to step 1910 is not terminated.

[0266] Next, episode data learning unit 410 determines whether all the simulation results were successful (1916). If they were successful, the answer is yes, in which case learning is successful and the repetition of the episode is terminated.

[0267] Next, the number of episodes is judged (1917). This process is performed when the episode is not successful within a predetermined number of episode repetitions, and if the judgment is yes, the learning is considered to have failed and the episode repetition is terminated. If learning is performed by repeating all episodes, the judgment in step 1917 is not necessary.

[0268] Finally, episode data learning unit 410 performs a learning end process (1919). This process is an output process of logs and the like.

[0269] The above is an explanation of an example of a series of processing steps for learning episode data, using the flowchart in Fig. 19. Through the above processing, episode data that matches a preset learning pattern for multiple simulation results of other moving objects (for example, capture, loss, good position, etc.) is extracted, episode data learning is performed, and the learning results can be stored in episode database 122.

[0270] As a result, instead of using all of the episode database 122 calculated by reinforcement learning, episode data extracted from the episode database 122 using a preset learning pattern is extracted, and the deep neural network 10 is directly trained with episode data from multiple simulation results.

[0271] In episode data learning, it is possible to efficiently train the deep neural network according to the learning pattern settings, and for example, it is possible to complete the training with a small number of episodes while making it easier to succeed in learning. This makes it possible to improve the accuracy of the behavior control of the deep neural network 10 and facilitate the operation of autonomously acting mobile objects.

[0272] The above is an explanation of an example of the contents of episode data learning in the present invention.

[0273] The mobile object behavior control planning device of the present invention uses calculation processing as its main technology and software as its main implementation means. Therefore, it can be implemented in the form of package software or cloud service. The policy function and state value function resulting from learning can be incorporated into the machine as control software. Therefore, it can be incorporated into mobile objects such as ships and vehicles that actually operate, and can also be used in other mobile objects, devices, and facilities that remotely control mobile objects. The present invention does not limit the scope of such applications.

[0274] The above has explained the technology for planning behavior control of a moving body in response to various behaviors of other moving bodies by performing reinforcement learning on the behaviors of other moving bodies and then using the resulting episode data of the various behaviors to learn the behavior of the moving body.

[0275] <Conclusion> As described above, the behavior control planning device 105 of the present invention is capable of learning behavior control of its own moving body using a deep neural network to achieve goals such as tracking and detection, even when the behaviors of other moving bodies have diverse relationships with respect to its own moving body, and applying the deep neural network obtained as a result of the learning to the control of moving bodies capable of acting autonomously.

[0276] In addition, by prioritizing episode data based on the loss function value, which is the objective function, in order to optimize parameters in training a deep neural network, reinforcement learning can be performed efficiently, and a deep neural network that has undergone reinforcement learning can improve the chances of an episode being successful.

[0277] The deep neural network 10 that has implemented reinforcement learning for individual environments, particularly for the simulation results of other moving objects, is successful in the simulation. Successful reinforcement learning episodes are accumulated as episode data. Using episode data of the simulation results of multiple other moving objects, the deep neural network can be trained directly with the step data of the episodes through episode data learning, allowing the deep neural network to learn behavioral control corresponding to multiple simulation results.

[0278] For successful learning of episode data, the accumulated episode data is collected and analyzed. For successful reinforcement learning by type of simulation results of individual environments and other moving objects, the successful and unsuccessful episodes and their number of steps are listed in relation to the order of episodes, and the range of failures, successes and number of steps for the order of episodes is defined as a learning pattern.

[0279] By performing episode data learning by utilizing episode data for the episode order defined above, the likelihood of an episode being successful can be improved even when the behavior of other moving objects has diverse relationships with respect to the own moving object.

[0280] By planning the behavior control of a moving object through reinforcement learning and episode data learning, it becomes possible to deal with a variety of environments and behavior patterns of other moving objects that are difficult to plan manually. Furthermore, the behavior control planning device of this embodiment has the potential to obtain behavior control content that would be difficult for a human being to come up with.

[0281] In the above embodiment, an example has been shown in which the present invention is applied to the behavior control of a ship on patrol or in pursuit as a moving body to which a deep neural network learned by the behavior control planning device 105 is applied. However, the present invention is not limited to this, and can be applied to moving bodies that act autonomously, such as aircraft, vehicles, submarines, or drones.

[0282] In the above embodiment, the action control planning device 105 includes the simulator 400 (or 112), but the present invention is not limited to this, and the simulator 400 may be implemented by an external computer. In this case, the action control planning device 105 transmits the simulation conditions received in the condition setting 111 in Fig. 1 to the external simulator to execute the simulation, and obtains the results of the simulation. In this example, a action control planning system can be configured with the action control planning device 105 and the external simulator. Then, the action control planning device 105 performs reinforcement learning of the deep neural network 10 based on the simulation results from the outside.

[0283] As described above, the behavior control planning device 105 of the above embodiment can be configured as follows.

[0284] (1) A behavior control planning device (105) having a processor (11) and a memory (12) for planning behavior of a moving body with respect to behavior of another moving body, the behavior control planning device (105) including a simulation condition setting unit (111) for setting behaviors of the moving body and the other moving body as simulation conditions, a simulator (112) for outputting a positional relationship between the moving body (504) and the other moving body (505) as step data (900A) at predetermined time intervals based on the simulation conditions and a machine learning model (deep neural network 10) set in advance, and a state in which the positional relationship between the moving body (504) and the other moving body is indicated as a learning result by acquiring the step data (900A) and providing the step data (900A) to the machine learning model (10). 1. A behavior control planning device comprising: a reinforcement learning unit (101) that outputs episode data including a quantity (902), a reward (904), and a state (905) and accumulates the episode data in an episode data accumulation unit (episode database 122); an episode analysis unit (102) that analyzes the episode data accumulated in the episode data accumulation unit (122) to generate a learning pattern; and an episode data learning unit (103) that acquires the episode data corresponding to the learning pattern from the episode data accumulation unit (122) to generate learning episode data (episode data collection and listing function 422), and provides the learning episode data to the machine learning model (10) for learning.

[0285] With the above configuration, the behavior control planning device 105 can learn the behavior of its own moving body to achieve a purpose such as tracking or detection, even in various environments, especially when the behaviors of other moving bodies have diverse relationships with respect to the own moving body.

[0286] (2) The behavior control planning device (105) described in (1) above, wherein the reinforcement learning unit (101) sets the step data (900A) from the start of the step data (900A) until a predetermined end condition is reached as one episode, sets a step corresponding to the step data (900A) in the episode data, sets a simulation result for each of the step data (900A) in the state (905) of the episode data, and sets an episode type (1007, 1009) indicating whether a predetermined purpose has been achieved or not according to the simulation result in the episode, and the episode analysis unit (102) generates the learning pattern based on the episode type, the simulation result, the number of episodes, and the number of steps in the episode data.

[0287] With the above configuration, the episode analysis unit 102 evaluates the average and variance of all episode data, thereby narrowing down the range of the number of steps relative to the number of episodes from the start of reinforcement learning, which is a component of the learning pattern, making it possible to efficiently learn (train) the machine learning model while suppressing the number of learning subjects.

[0288] (3) The behavior control planning device (105) described in (2) above, wherein the episode analysis unit (102) sets a range of the number of episodes (1504) and a range of the number of steps of the episode data (1502, 1503) for each of the simulation results, and the episode data learning unit (103) acquires episodes that fall within the range of the number of episodes (1504) and the range of the number of steps of the episode data (1502, 1503) for each of the simulation results from the episode data storage unit (122), and causes the machine learning model (10) to learn the behavior control planning device.

[0289] With the above configuration, the episode data learning unit 103 can efficiently train the deep neural network 10 according to the learning pattern settings, and for example, learning is likely to be successful even while completing the training with a small number of episodes. This improves the accuracy of the behavior control of the deep neural network 10, making it easier to operate a mobile object that behaves autonomously.

[0290] (4) The behavior control planning device (105) described in (3) above, wherein the episode data learning unit (103) performs a predetermined replacement in the order of the episodes extracted within the number of episodes (1504) and the range of step numbers (1502, 1503) for each simulation result. With the above configuration, the episode analysis unit 102 can suppress variation in the training results of the deep neural network 10 by performing a predetermined replacement on the order of episodes.

[0291] (5) The behavior control planning device (105) described in (3) above, wherein the episode data learning unit (103) directly inputs the episode data corresponding to the acquired episode into the machine learning model (10) to have it learn (1905).

[0292] With the above configuration, the episode data learning unit 103 can learn to select an action according to a situation by directly using episode data to train the deep neural network 10.

[0293] (6) The behavior control planning device (105) according to (1) above, wherein the predetermined purpose is for the own moving body (504) to track the other moving body (505).

[0294] With the above configuration, it is possible to generate a deep neural network 10 that can track the other moving body 505 without losing sight of the own moving body 504 and without being captured by the other moving body 505.

[0295] (7) The behavior control planning device (105) described in (2) above, wherein the episode analysis unit (102) allocates the episodes in a predetermined ratio according to the type of the simulation result within the range of the number of episodes (1504) and the range of the number of steps of the episode data (1502, 1503).

[0296] The episode analysis unit 102 sets the ratio to be allocated to each type of simulation result within the range of the number of episodes and the range of the number of steps of the episode data, making it possible to apply trends in the type of simulation result according to the number of episodes and the number of steps.

[0297] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment is described in detail to easily explain the present invention, and is not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, the addition, deletion, or replacement of other configurations can be applied to part of the configuration of each embodiment, either alone or in combination.

[0298] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations and functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0299] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0300] 11 Processors 12. Memory 13 Storage Devices 101 Reinforcement Learning Department 102 Episode Analysis Section 103 Episode Data Learning Unit 104 Operator 105 Action Control Planning Device 111 Condition Setting 112 Simulator 113 Status Update 114 Action Selection 115 Learning Machine 116 Acquiring training data 117 Deep Neural Network Training 121 Episode Analysis 122 Episode Database 131 Learning Pattern Settings 132 Episode Selection 133 Episode Data Learning 400 Simulator 401 Behavioral environment setting function 402 Mobile behavior processing function 403 Moving object state quantity calculation function 404 Moving object status determination function 410 Reinforcement Learning / Episode Data Learning Department 411 Policy-state value function composition function 412 Episode processing function 413 Reward setting function 414 DNN training function 415 Data sampling function for training DNN 416 Policy and state value calculation function 417 Episode data accumulation function 418 Behavioral Environment Database 419 Episode Database 420 Episode Data Learning Department 421 Learning pattern acquisition function 422 Episode data collection and listing function 423 Episode Data Learning Function 430 Episode Analysis Section 431 Episode data collection function 432 Episode data drawing function 433 Step-by-step episode aggregation function 434 Learning pattern configuration function

Claims

1. A behavior control planning device having a processor and a memory for planning a behavior of a moving object with respect to a behavior of another moving object, a simulation condition setting unit that sets the behavior of the own moving body and other moving bodies as simulation conditions; a simulator that outputs a positional relationship between the own moving body and another moving body as step data at predetermined time intervals based on the simulation conditions and a preset machine learning model; a reinforcement learning unit that acquires the step data, provides the step data to the machine learning model, and outputs episode data including a state quantity, a reward, and a state indicating the positional relationship between the own moving body and another moving body as a learning result, and accumulates the episode data in an episode data accumulation unit; an episode analysis unit that analyzes the episode data stored in the episode data storage unit to generate a learning pattern; an episode data learning unit that acquires the episode data corresponding to the learning pattern from the episode data storage unit to generate learning episode data, and provides the learning episode data to the machine learning model to learn the model; having The reinforcement learning unit is the step data from the start of the step data until a predetermined end condition is reached is regarded as one episode, a step corresponding to the step data of the episode is set as the episode data, a simulation result for each of the step data is set as the state of the episode data, and an episode type indicating whether or not a predetermined purpose has been achieved according to the simulation result is set as the episode; The episode analysis unit includes: generating the learning pattern based on the episode type, the simulation result, the number of episodes, and the number of steps of the episode data; The episode analysis unit includes: setting a range of the number of episodes and a range of the number of steps of the episode data for each of the simulation results; The episode data learning unit includes: acquiring episodes corresponding to the range of the number of episodes and the range of the number of steps of the episode data for each of the simulation results from the episode data storage unit, and causing the machine learning model to learn the acquired episodes; The episode data learning unit includes: a behavior control planning device which performs a predetermined replacement in the order of the episodes extracted within the range of the number of episodes and the number of steps for each of the simulation results.

2. The behavior control planning device according to claim 1, The episode data learning unit includes: The behavior control planning device is characterized in that the episode data corresponding to the acquired episode is directly input to the machine learning model and learned.

3. The behavior control planning device according to claim 1, The predetermined objective is to 2. A behavior control planning device, comprising: a moving body tracking the other moving body.

4. A behavior control planning device as claimed in claim 1, The episode analysis unit includes: A behavior control planning device, characterized in that the episodes are allocated at a predetermined ratio according to the type of the simulation result within the range of the number of episodes and the range of the number of steps of the episode data.

5. A behavior control planning method in which a computer having a processor and a memory plans a behavior of a moving object with respect to a behavior of another moving object, comprising: a simulation condition setting step in which the computer sets actions of the own moving body and other moving bodies as simulation conditions; a simulation step in which the computer outputs a positional relationship between the own moving body and another moving body as step data at predetermined time intervals based on the simulation conditions and a preset machine learning model; a reinforcement learning step in which the computer acquires the step data, provides the step data to the machine learning model, and outputs episode data including a state quantity, a reward, and a state indicating the positional relationship between the own moving body and another moving body as a learning result, and accumulates the episode data in an episode data accumulation unit; an episode analysis step in which the computer analyzes the episode data stored in the episode data storage unit to generate a learning pattern; an episode data learning step in which the computer acquires the episode data corresponding to the learning pattern from the episode data storage unit to generate learning episode data, and provides the learning episode data to the machine learning model to learn the learning episode data; Including, The reinforcement learning step includes: the step data from the start of the step data until a predetermined end condition is reached is regarded as one episode, a step corresponding to the step data of the episode is set as the episode data, a simulation result for each of the step data is set as the state of the episode data, and an episode type indicating whether or not a predetermined purpose has been achieved according to the simulation result is set as the episode; The episode analysis step includes: generating the learning pattern based on the episode type, the simulation result, the number of episodes, and the number of steps of the episode data; The episode analysis step includes: setting a range of the number of episodes and a range of the number of steps of the episode data for each of the simulation results; The episode data learning step includes: acquiring episodes corresponding to the range of the number of episodes and the range of the number of steps of the episode data for each of the simulation results from the episode data storage unit, and causing the machine learning model to learn the acquired episodes; The episode data learning step includes: a step of performing a predetermined replacement of the episodes extracted within the range of the number of episodes and the number of steps for each of the simulation results in the order of the episodes.

6. A behavior control planning device having a processor and a memory; a simulator connected to the behavior control planning device and performing a predetermined simulation, the behavior control planning system planning a behavior of a moving object with respect to a behavior of another moving object, The simulator comprises: carrying out the simulation under the simulation conditions received from the action control planning device; The behavior control planning device includes: a simulation condition setting unit that sets the behaviors of the own moving body and the other moving bodies as simulation conditions, transmits the simulation conditions and a preset machine learning model to the simulator, and outputs the positional relationship between the own moving body and the other moving bodies as step data at predetermined time intervals; a reinforcement learning unit that acquires the step data, provides the step data to the machine learning model, and outputs episode data including a state quantity, a reward, and a state indicating the positional relationship between the own moving body and another moving body as a learning result, and accumulates the episode data in an episode data accumulation unit; an episode analysis unit that analyzes the episode data stored in the episode data storage unit to generate a learning pattern; an episode data learning unit that acquires the episode data corresponding to the learning pattern from the episode data storage unit to generate learning episode data, and provides the learning episode data to the machine learning model to learn the model; having The reinforcement learning unit is the step data from the start of the step data until a predetermined end condition is reached is regarded as one episode, a step corresponding to the step data of the episode is set as the episode data, a simulation result for each of the step data is set as the state of the episode data, and an episode type indicating whether or not a predetermined purpose has been achieved according to the simulation result is set as the episode; The episode analysis unit includes: generating the learning pattern based on the episode type, the simulation result, the number of episodes, and the number of steps of the episode data; The episode analysis unit includes: setting a range of the number of episodes and a range of the number of steps of the episode data for each of the simulation results; The episode data learning unit includes: acquiring episodes corresponding to the range of the number of episodes and the range of the number of steps of the episode data for each of the simulation results from the episode data storage unit, and causing the machine learning model to learn the acquired episodes; The episode data learning unit includes: a predetermined replacement is performed in the order of the episodes extracted within the range of the number of episodes and the number of steps for each of the simulation results.

Images