Estimation device, system and method
The estimation device uses separate encoders for states and dynamics parameters to efficiently adapt simulation-trained models to real-world scenarios, addressing the inefficiencies of reinforcement learning in real environments.
Patent Information
- Application Number
- JP2019209036
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2019-11-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-11-19
AI Technical Summary
Existing reinforcement learning methods require a large number of trials in real environments, making them time-consuming and less secure when starting from random initial strategies.
An estimation device that estimates dynamics and behavior using separate encoders for states and dynamics parameters, trained in simulation environments, allowing for efficient adaptation to real-world scenarios without extensive real-time training.
Enables quick acquisition of strategies for process automation in real environments, improving accuracy and safety by leveraging simulation-trained models.
Smart Images

Figure 0007579632000015 
Figure 0007579632000016 
Figure 0007579632000017
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an estimation device, a training device, an estimation method, a training method, a program, and a non-transitory computer-readable medium. [Background technology]
[0002] Conventionally, attempts to optimize control for automatically executing a specific task by a control device or the like through machine learning, particularly reinforcement learning, and to optimize the method of allocating tasks have been widely made today. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Bernd Waschneck, et. al., "Optimization of global production scheduling with deep reinforcement learning," Procedia CIRP, pp. 1264-1269, Vol. 72, 2018. Summary of the Invention [Problem to be solved by the invention]
[0004] Therefore, the present disclosure provides an estimation device capable of appropriate estimation, and a training device for easily training such an estimation device. [Means for solving the problem]
[0005] An estimation device according to one embodiment includes one or more memories and one or more processors. The memory stores a trained model capable of estimating information of dynamics parameters. The one or more processors: estimating behavior information based on at least information on a state of a control object in a real environment and information on dynamics parameters estimated based on a time series of information on the state of the control object in the real environment and a time series of behavior using at least the trained model; The trained model is trained in a plurality of simulated environments simulating environments having different dynamics; The information of the dynamics parameters estimated by the trained model is an encoding result of the dynamics parameters; the information on the state of the control object is an encoding result of the state of the control object, The one or more processors: The encoding of the state and the encoding of the dynamics parameters are performed using different trained models. [Brief description of the drawings]
[0006] [Figure 1] FIG. 1 is a block diagram of an estimation device according to an embodiment. [Diagram 2] 4 is a flowchart showing a process of an estimation device according to an embodiment. [Diagram 3] FIG. 1 is a block diagram of a training device according to one embodiment. [Figure 4] 11 is a flowchart showing the processing of a training device according to an embodiment. [Diagram 5] 11 is a flowchart showing the processing of a training device according to an embodiment. [Figure 6] FIG. 13 shows the results according to one embodiment. [Figure 7] FIG. 13 shows the results according to one embodiment. [Figure 8] FIG. 13 shows the results according to one embodiment. [Figure 9] FIG. 1 illustrates an example of a hardware implementation according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The drawings and the description of the embodiment are shown as an example and do not limit the present invention. Hereinafter, the model (trained model) provided in each configuration may be a neural network such as MLP (Multi-Layer Perceptron) or CNN (Convolutional Neural Network), but is not limited to these, and may be any model that can appropriately obtain output data for input data. Also, a model other than a neural network model may be used.
[0008] In this embodiment, an estimator that estimates dynamics in a simulation environment and an estimator that estimates behavior in the simulation environment according to the dynamics are used. As a result, reinforcement learning, which requires a large number of trials for learning, is intended to be more applicable to a real environment. For example, in reality, trying reinforcement learning, which requires a considerable number of trials for training, in a real environment takes too much time, and there is a problem that selecting behavior with a randomly initialized policy reduces safety. In this embodiment, training and estimation are performed to solve this problem.
[0009] (estimation device) 1 is a block diagram showing functions of an estimation device according to an embodiment. The estimation device 1 in the present disclosure includes a first estimation unit 10 and a second estimation unit 12. The first estimation unit 10 estimates a behavior based on a dynamic condition. The second estimation unit 12 estimates a dynamic situation.
[0010] The first estimator 10 includes a first encoder 100, a second encoder 102, and a decoder 104. In addition, the training device 2 may include an interface for inputting and outputting data, etc., and a storage unit for storing data, parameters, etc.
[0011] The first estimator 10 inputs, for example, a state o and a dynamics parameter η for a certain object, and outputs an action a. These quantities are expressed, for example, as vectors, but are not limited to this, and may be expressed as tensors having any dimension.
[0012] The state indicates, for example, the observed state of an object and a controlled object in a certain environment. This state is, for example, information acquired by various sensors, and may include information such as the state of the controlled object and the state of an object on which the controlled object exerts some action.
[0013] Dynamics refers to, for example, the dynamics of the environment of the space in which the controlled object and the object exist, and parameters that characterize this dynamics are described as dynamics parameters. These dynamics parameters may include, for example, information such as the coefficient of friction between the object and the area in which the object is placed, the inertial system in which the object is placed, or the situation in which the sensor senses the object and the controlled object. In addition, the dynamics parameters may include parameters that indicate the influence of the environment on the controlled object and the object.
[0014] The behavior indicates, for example, a desired behavior to be taken by the controlled object or a desired control to be given to the controlled object in the case of an input state in a space described by the input dynamics parameters. That is, the behavior indicates, for example, a behavior conditioned on the dynamics parameters.
[0015] A policy is, for example, a function that outputs a desired action to be taken by a controlled object or a desired control to be given to the controlled object in an input state. A policy is generally expressed in the form of a conditional probability distribution regarding the action or control, conditioned by the state. Here, for example, a policy is expressed as a conditional probability distribution conditioned not only by the state but also by dynamics parameters.
[0016] The first encoder 100 is a state o at time t in an environment e. t (e) It takes as input the state, encodes it, and outputs its features.
[0017] The second encoder 102 detects the dynamics parameters η e The dynamics parameters are encoded and the features are output.
[0018] The transformation of the first encoder 100 is f φ () and the conversion of the second encoder 102 is M ζ (), the latent state Z in environment e at time tt =[f φ (o t (e) ), M ζ (η e )]. As described above, the first encoder 100 and the second encoder 102 encode the state and dynamics parameters, respectively.
[0019] By encoding the state separately from the dynamics parameters, it is possible to distinguish between a state that changes from moment to moment and dynamics parameters that change only slowly, even if they change, in comparison with the case where the state and dynamics parameters are encoded together, and therefore it is possible to efficiently handle them, for example by reusing the encoding of the same dynamics parameters even if the lighting or camera changes in the same environment where the dynamics parameters do not change. Also, while the state changes from moment to moment based on data sensed per unit time, for example, the dynamics parameters change less than the state, and may not change per unit time but may be virtually fixed within a certain period of time, so the state and dynamics parameters are encoded separately in this way.
[0020] That is, the re-encoding of the state and the dynamics parameters does not have to be performed at the same timing. For example, the same encoded value may continue to be used for a while. The frequency of re-encoding may differ between the state and the dynamics. When an encoded value exists, for example, the encoding of the state may be performed in a span different from that of the encoding of the dynamics parameters, for example, in a shorter span. For this reason, in this embodiment, the state and the dynamics parameters are encoded using the first encoder 100 and the second encoder 102.
[0021] The decoder 104 generates the latent state Z t Take action a as input t (e) The behavior is output, for example, as a distribution of predicted behaviors or controls. In one example, the decoder 104 outputs the latent state Zt Given a, we choose action a from the conditioned probability distribution. t (e) The decoder 104 outputs probabilistically. This probability distribution may be a distribution that generates a behavior sample deterministically, such as a δ function, or may be a normal distribution. When the decoder 104 outputs the behavior as a normal distribution, the average value m t and the variance of the action,Σ t and the latent state Z t Using these values, the action may be determined based on, for example, t (e) ~N(m t , Σ t ) where N(m, Σ) denotes a normal distribution with mean m and variance vector Σ. The decoder 104, for example, t For example, we output the distribution and choose the action a t (e) is generated as a sample.
[0022] That is, the decoder 104 outputs information on, for example, what behavior the controlled object should take. More specifically, the decoder 104 outputs a probabilistic behavior of the controlled object. The conversion of the decoder 104 is g θ If we express it as (), for example, the distribution of behavior in the case of normal distribution is (m t , Σ t )=g θ ([f φ (o t (e) ), M ζ (η e )]).
[0023] The distribution of the behavior is not limited to a normal distribution. It may be any distribution that can appropriately express the probability distribution of the behavior, and in the case of another distribution, the decoder 104 outputs the behavior probabilistically based on parameters suitable for expressing the distribution.
[0024] In the above description, the first encoder 100, the second encoder 102, and the decoder 104 are described as having separate models, but this is not limited thereto, and all or at least two of the configurations may be included in the same model. For example, in a case where it is desired to estimate the next action when the state is updated, the first encoder 100 and the decoder 104 may be one continuous model. In this case, the configuration may be such that the output of the second encoder 102 can be accepted in the middle of the processing. As another example, all of the above configurations may be included in one model. In this case, the input of the dynamics parameters and the input of the state may be executed at different timings.
[0025] The first encoder 100, the second encoder 102, and the decoder 104 may be pre-trained models having any configuration, for example trained with any machine learning method.
[0026] In this manner, the first estimator 10 estimates the behavior based on the state and the dynamics parameters.
[0027] The second estimation unit 12 of this embodiment includes an estimator 120 and an aggregator 122. The second estimation unit 12 estimates dynamics parameters corresponding to various environments to be input to the second encoder 102. When a state time series and an action time series are input, the second estimation unit 12 outputs dynamics parameters. The output dynamics parameters may be expressed as probability distributions.
[0028] State o at a certain time t and Action a t The relationship holds that if you take action on a situation, the next situation will occur. That is, o at time t t and a t Based on the dynamics parameters, t+1 This means that the set of these three values (o t , a t , o t+1The processing of the second estimation unit 12 is executed by a function having the following input:
[0029] Here, the input from t=1 to kT is {(o t , a t , o t+1 ) n} n=1 kT If there is a time series of input data, split the input into chunks of, say, T pieces. That is, divide the time series of input data into chunks of, say, {(o t , a t , o t+1 ) t} t=1 T ,{(o t , a t , o t+1 ) t} t=T+1 2T ,···,{(o t , a t , o t+1 ) t} t=(k-1)T+1 kT The data is divided into the following. Processing is performed for each divided data. If the time series does not have a number of pieces that is a multiple of T, for example, the last chunk may be up to the last data, or may be supplemented with appropriate data without being limited to this.
[0030] The estimator 120 obtains an estimate of the dynamics parameters for the partitioned chunks based on the input for each partitioned chunk. The estimator may be expressed in the form of a probability distribution, e.g., with mean μ and variance σ 2 In other words, it is possible to obtain an estimate of T (o t , a t , o t+1 ) to obtain an estimate of the dynamics parameter. This is performed for k chunks to obtain an estimate for each chunk. The estimator 120 obtains, for example, the dynamics parameter estimate μ (1) ,σ (1) ,μ (2) ,σ (2) ,μ(k) ,σ (k) Get the.
[0031] This calculation may be performed in parallel, and for example, multiple estimators 120 may be provided for each chunk. In addition, in this embodiment, the average value and the variance are obtained as an example of the estimator, but the present invention is not limited to this, and any suitable estimator may be obtained.
[0032] The aggregator 122 synthesizes the estimated quantities output by the estimator 120 using, for example, a trained model, and estimates and outputs dynamics parameters in the environment in which the time-series data is observed.
[0033] The estimator 120, aggregator 122 may be, for example, a trained model having any configuration, trained with any machine learning technique.
[0034] In this manner, the second estimator 12 estimates the dynamics parameters based on the state and the action.
[0035] The estimation device 1 estimates dynamics parameters based on already acquired state and action data, for example, by the second estimation unit 12. Then, the first estimation unit 10 estimates an action based on the state at the current time and the estimated dynamics parameters.
[0036] FIG. 2 is a flowchart showing the process of the estimation device 1 according to this embodiment.
[0037] First, the acquired state and action are input to the second estimation unit 12 to estimate dynamics parameters (S100).
[0038] Next, the estimated dynamics parameters are encoded by the second encoder 102 of the first estimation unit 10 (S102).
[0039] Next, the observed state is encoded by the first encoder 100 of the first estimator 10 (S100).
[0040] Next, the decoder 104 of the first estimation unit 10 estimates the behavior based on the encoded estimated dynamics parameters and the encoded state (S106).
[0041] Next, it is determined whether or not to end the process (S108).
[0042] If the process does not end (S108: NO), the process is repeated from S 104. In this case, for example, the state may be a new state to which a transition has occurred due to the estimated action.
[0043] If the process is to be ended (S108: YES), the necessary end process is executed and the process is ended.
[0044] Although the above describes the case where the dynamics parameters are not updated, the dynamics parameters may be updated at a predetermined timing. For example, when the process is not to be ended (S108: NO), it is determined whether or not to update the dynamics parameters (S110).
[0045] If the dynamics parameters are not to be updated (S110: NO), the process repeats from S104.
[0046] When the dynamics parameters are updated (S110: YES), the process from S100 is repeated. In this case, the state and action used to estimate the dynamics parameters may be the state observed up to that time and the action executed up to that time.
[0047] In this way, the dynamics parameters may be updated at appropriate times.
[0048] As described above, by estimating dynamics parameters based on already acquired state and behavior data, and further estimating behavior based on the acquired state and the estimated dynamics parameters, it is possible to quickly acquire a measure for acquiring a desired behavior without interacting with the actual environment. As a result, it is possible to improve the accuracy of automating processes in, for example, production sites. Note that, although the first estimation unit 10 and the second estimation unit 12 are configured separately in the above, this is not limiting. These estimation units may be implemented as a single estimation unit.
[0049] The estimation device 1 thus implemented can ensure safety more than when a random initial value is used for trial and training in a real environment. Furthermore, according to the above-described embodiment, it is possible to obtain a policy that can adapt to differences that arise between a policy trained in a simulation environment and a real environment. This result can be used, for example, for automation using a robot.
[0050] For example, dynamics parameters may change depending on the environment, and a second encoder 102 capable of outputting appropriate results for various dynamics parameters may be required. Therefore, the first estimation unit 10 uses the second encoder 102 trained in a simulator environment, for example. By using the second encoder 102 trained in a simulator environment, online behavior estimation for various environments becomes possible.
[0051] (Training device) The training device according to this embodiment trains the first encoder 100, the second encoder 102, the decoder 104, the estimator 120, and the aggregator 122 of the estimation device 1. The training is performed, for example, by a machine learning technique. When a simulator is used, the training according to this embodiment is based on the fact that dynamics parameters can be acquired in advance. That is, training is performed so that actions can be acquired from states using known dynamics parameters in a simulator environment. Then, training is performed to improve the estimation accuracy of actions in the real space by adapting this simulation environment to a real space.
[0052] 3 is a block diagram of the training device 2 according to this embodiment. The training device 2 includes a forward propagation unit 20, an error calculation unit 22, and an update unit 24. In addition, the training device 2 may include an interface for inputting and outputting data, etc., and a storage unit for storing data, parameters, etc. The training device 2 trains models in the components of the estimation device 1 described above.
[0053] The training device 2 may execute training of the first encoder 100, the second encoder 102, the decoder 104, the estimator 120, and the aggregator 122 shown on the right side of the figure in a single device. As another example, one training device 2 may be provided for each of the first estimator 10 and the second estimator 12. As yet another example, a training device 2 may be provided for training some of all models. When one training device 2 trains multiple models, each process such as a loss calculation method and an update method, i.e., a training method, may be switched for each model.
[0054] As described above, it is preferable that the model of each component of the first estimation unit 10 is trained in a simulation environment. It is generally difficult to obtain accurate information about dynamics parameters in a space in which a device or the like to be controlled is located. By preparing a plurality of simulation environments in which each element of the dynamics parameters is given and performing training in the simulation environments, it is possible to easily obtain a model capable of estimating behavior toward a device or the like to be actually controlled.
[0055] The second estimator 12 estimates dynamics parameters from states and actions, and the dynamics parameters estimated by the second estimator 12 can be used as dynamics parameters for constructing a simulation environment for executing training of the first estimator 10. The second estimator 12 is also used not only for training the first estimator 10, but also for estimating dynamics parameters for estimating actions in an actual environment, as described above. In this way, the training device 2 trains a model for estimating dynamics parameters for executing training of the first estimator 10 or estimation.
[0056] The forward propagation unit 20 executes a forward propagation process for a model to be trained. For example, it inputs predetermined data to each model and obtains output data.
[0057] The error calculation unit 22 calculates an error based on the data acquired by the forward propagation unit 20 and teacher data, or a loss described by an accumulated cost acquired from the system by executing the policy. The method of calculating the loss may differ depending on the model. As will be described later, the error calculation unit 22 may calculate a loss described by the cost of a reward (accumulated reward) in reinforcement learning, and the like, without necessarily being used for error calculation in supervised learning. In this way, the error calculation unit 22 calculates, in a broad sense, the loss, reward, and other values required for parameter update based on the objective function.
[0058] The update unit 24 updates the network based on the loss calculated by the error calculation unit 22. The network is updated, for example, by backpropagating the error. When backpropagating the error, the processes of the error calculation unit 22 and the update unit 24 may be executed for each layer of the neural network, as shown by the dotted arrows in the figure.
[0059] The model training provided for each department is explained in detail below.
[0060] First, a description will be given of the model provided in the first estimator 10. The models provided in the first encoder 100, the second encoder 102, and the decoder 104 of the first estimator 10 may be trained at the same time.
[0061] 4 is a flowchart showing the training process of the first estimator 10 according to this embodiment. The training process of the first estimator 10 of the training device 2 will be described with reference to FIG.
[0062] First, the training device 2 initializes parameters (S200). The parameters are the model f φ , M η , g θ , and a model f for training the first encoder 100 and the decoder 104. rec , g inv These are parameters related to
[0063] Next, n environments are randomly generated (S202). These environments are formed on a simulator, for example. Based on the state and action acquired from the simulator, the second estimation unit 12 can estimate and acquire n dynamics parameters corresponding to each environment, which will be described later.
[0064] Next, parameter updates are performed by loop processing. First, one environment is randomly selected from the n environments generated (S204). The process from the selection of this environment is regarded as an episode, and parameter updates may be repeated within the episode so as to satisfy a predetermined condition. In addition, any algorithm may be used as the training algorithm for each model.
[0065] Next, the dynamics parameters corresponding to the selected environment are acquired (S206). The dynamics parameters may be, for example, those estimated by the second estimation unit 12 in S202.
[0066] Next, an initial state is sampled randomly (S208). For example, an initial state is randomly obtained from various data given as test data.
[0067] Next, the parameters are updated (S210). More specifically, the initial state obtained in S208 is input to the first encoder 100, and the dynamics parameters obtained in S206 are input to the second encoder 102, and the latent state Z t Then, we obtain this Z t is input to the decoder 104, the action a t (e) This may be performed, for example, by the forward propagation unit 20 forward propagating input data to each model. Then, based on the obtained results, the error calculation unit 22 calculates appropriate errors and the like for each model, performs backpropagation, and the update unit 24 updates the parameters. Training of each component will be described later.
[0068] Next, it is determined whether or not to end the process (S212). The end of the process may be determined, for example, when a predetermined number of episodes have been repeatedly calculated for a predetermined number of epochs, when the accuracy has reached a predetermined value or more, when a predetermined condition has been satisfied in validation, etc. Here, the predetermined numbers for episodes and epochs do not have to be the same, and may be set separately for each.
[0069] When it is determined not to end the process (S212: NO), the process from S204 is repeated. In this case, if the number of repetitions of the episode is within a predetermined number, the episode may be repeated. When the repetition of the episode is ended, it may be determined whether the number of epochs has reached a predetermined number, and if the number of epochs has not reached the predetermined number, the process of the next epoch may be executed.
[0070] On the other hand, if it is determined that the processing should be ended (S212: YES), the training device 2 executes necessary processing, such as outputting and storing the optimized parameters, and then ends the processing.
[0071] The first estimator 10 may thus be trained in a simulation environment. This simulation environment may be optimized by imparting randomness and learning a strategy that can adapt to this. Since a simulation environment is used, the training device 2 can obtain, for example, dynamics parameters for various environments. The training device 2 uses the dynamics parameters to train the first estimator 10.
[0072] The training of the first estimator 10 can be performed by reinforcement learning. Since reinforcement learning generally requires a very large number of trials, it is practically difficult to perform reinforcement learning in a plurality of different environments so as to be able to perform appropriate estimation in different environments. In addition, training in a real environment may be dangerous. However, in this embodiment, by training the first estimator in a simulation environment, reinforcement learning can be easily performed, and therefore a first estimator capable of performing appropriate estimation in different environments can be obtained.
[0073] In reinforcement learning, for example, the probability distribution for generating an action a is π(a|o, η) and the reward r k You can also set it and use the following objective function for reinforcement learning:
number
[0074] When performing reinforcement learning, parameters in the simulation environment may be acquired in advance in the simulation environment as described above. When reinforcement learning is performed as in this embodiment, it becomes possible to apply the results of training in the simulation environment to the real environment while suppressing a decrease in accuracy.
[0075] The training of the first encoder 100 may further include the following loss L rec (t) may be additionally implemented.
number
[0076] The objective function is not limited to this, and may be any objective function that allows a comparison between an appropriately encoded state and the original state.
[0077] In addition, the training of the second encoder 102 and the decoder 104 may be further performed using the following loss L inv (t) may be additionally implemented.
number
[0078] By defining Lrec(t) and Linv(t) in this way and training the model, it is possible to perform training so as to reduce the influence of information that is unnecessary for estimating behavior, such as changes in light intensity, lighting direction, and shadow position, from the state and environment and perform estimation.
[0079] L inv (t) is the first action actually given t The state that occurs due to the first action, and what second action a will be taken when that state occurs t Using this loss function, the model M ζ , model g of the decoder 104 θ and a model g that obtains actions from latent states inv Update the parameters of
[0080] Next, training of the second estimator 12 that estimates the dynamics parameters to be input to the first estimator 10 will be described.
[0081] In training the first estimator 10, it is necessary to obtain dynamics parameters in n random environments. First, actions may be randomly sampled within the range of permitted actions, and a policy may be set that allows several episodes or the like to be performed in each environment. The permitted actions here refer to, for example, actions permitted in the environment, or any actions in a safe policy previously trained.
[0082] Therefore, the state transition data is, as described above, t (e) , a t (e) , o t+1 (e) )} and training of the second estimator 12 is performed. Here, F() is defined as a dynamics parameter model forward propagated by the simulator. The true next state and the next state by the simulator are defined by the dynamics parameters η e When the true values of can be obtained, they are expressed as follows:
number
number
[0083] As mentioned above, D e Let T be a tuple (o t (e) , a t (e) , o t+1 (e) ), into k chunks, and for each chunk i∈{1, ... , k}, (i) and σ (i) The model of the estimator 120 is trained so that it can estimate the time series of actions in chunk i in environment e as x i (e) , the time series of states is y i (e) Then, this may be performed by estimating the following posterior distribution p:
number
[0084] k η calculated for each chunk i e To aggregate the estimates of e The posterior distribution of p(η e |x i (e) , y i (e) ) and the dynamics parameters η conditioned on the entire data set e We can use the relationship that should hold between the posterior distribution of η e The posterior distribution of is expressed as follows:
number
number
number
number
number
[0085] As a method for approximating the posterior distribution, KL (Kullback-Leibler) divergence may be used. For example, the true posterior distribution p true (η e |D e ) and the posterior distribution p θ (η e |D e ) the KL divergence KL[p θ (η e |D e )|p true (η e |D e )] can be approximated by minimizing it with respect to the parameter θ. Following lower bound optimization, the true posterior distribution p true (η e |D e ) can be used to approximate the posterior distribution for each environment e without explicitly evaluating it. For example, the optimization can be performed using the following formula:
number
[0086] Also, the true posterior distribution p true (η e |D e ) to obtain a normal distribution whose KL divergence is close to that of the normal distribution, a reparameterization trick often used in VAE (Variational Auto Encoder) can be used. That is, ε j ~N(ε j |0, 1) and the obtained ε j Using η e By expressing it according to the following formula, we can obtain the parameterized posterior distribution p θ (η e |D e ) may be replaced by the expectation of the standard normal distribution ε~N(ε|0, 1).
number
[0087] As shown above, a standalone tuple (o t (e) , a t (e) , o t+1 (e) ), by considering chunks of the time series, it becomes possible to effectively approximate the posterior distributions of complex dynamics parameters such as friction, gravity, etc. In general, the posterior distributions of the dynamics parameters can be complex multimodal distributions, but as the number of samples in a chunk of the time series increases, e.g., following the central limit theorem, the posterior distribution of η conditioned on the samples can be approximated to a normal distribution.
[0088] FIG. 5 is a flowchart showing the training process of the second estimator 12 by the training device 2. The details have been described above, so a brief description will be given. First, the parameters of the estimator 120 and the aggregator 122 are initialized (S300). Next, the dynamics parameters of the environment e are set randomly, and state and action samples are extracted to generate an entire time series (S302). Next, the time series is divided to generate a plurality of chunks to be used for parameter generation (S304). According to the above [Equation 4] to [Equation 13], the parameters of the estimator 120 and the aggregator 122 are updated (S308). If the process is completed (S308: YES), the training device 2 completes the training process of the second estimator 12. The completion of the training may be in accordance with the conditions described in the training of the first estimator 10, or may be different. If the process is not completed (S308: NO), the process may be repeated from the extraction process of state and behavior samples in a different environment (S302). Also, for example, when changing the training environment, the process may be repeated from the extraction process of state and behavior samples (S302) as necessary.
[0089] In this way, by providing state transition data, it is possible to estimate dynamics parameters of a predetermined environment, for example, an environment including a real environment such as a test environment, using the trained model. According to the above, estimation of dynamics parameters may be performed using only data outside the policy. This data outside the policy can be collected by simply executing a random policy or an existing policy created using a rule base or the like that is known to be safe to execute. Therefore, data for training a model to obtain dynamics parameters in various environments can be collected at low cost.
[0090] As described above, according to this embodiment, it is possible to train a model of the second estimator 12 that acquires dynamics parameters, and generate a simulator using the dynamics parameters acquired by this model. Then, by performing training in various environments using the simulator for an observed state, it is possible to apply the model in a real environment with high accuracy using the model trained in the simulator environment, and to train the model.
[0091] The second estimator 12 can be trained by supervised learning as described above by utilizing the fact that the dynamics parameters are known in the simulation environment. Furthermore, even if the dynamics parameters are unknown, it is also possible to learn the dynamics parameters and state transition parameters representing the dynamics so that future states can be appropriately predicted.
[0092] By using the second estimator 12, which has been trained in advance in such a simulation environment, it is possible to estimate the dynamics parameters of the environment by simply observing the state transitions and the sequence of actions by existing methods in the real environment, without actually selecting an action and realizing an interaction with the environment that affects the environment. By using the dynamics parameters estimated by the second estimator 12 to set the first estimator 10 as a policy, it is possible to obtain a good policy in the real environment.
[0093] The first estimator 10 is trained using the estimated values of the dynamics parameters in n environments estimated by the second estimator 12 trained in this way, and the unknown dynamics parameters η n+1 The behavior in the n+1th environment having the n+1th environment, for example, the environment of a real machine, may be estimated according to the flowchart shown in FIG.
[0094] For example, first, f φ , M η , g θ , and the respective models (and f rec , ginv Let us assume that the environment is the n+1th environment, e.g., the test environment. Here, we generate a time series of states and actions outside the policy (D n+1 This acquisition may be performed, for example, by operating the behavior acquired by the first estimation unit 10 in the real environment. n+1 The second estimation unit 12 estimates the dynamics parameter η n+1 Then, the model f φ , M η , g θ Then, a policy based on the dynamics parameters is executed to estimate an action (S106). In this way, according to this embodiment, it is possible to estimate an appropriate action from a state based on a policy trained in a simulator environment without performing trials on the environment.
[0095] In this embodiment, fine tuning may be further performed. For example, the dynamics parameters generated in the real environment are used to operate the real machine according to the estimated behavior. Then, when a mistake occurs, for example, a time series of states and behaviors is obtained by having a human behave in a more desirable manner. Fine tuning of the first estimation unit 10 may be performed using this time series. In this way, the first estimation unit 10 can be tuned to a higher accuracy.
[0096] For example, the method may include a step of evaluating whether the obtained policy can obtain good performance before actually applying the policy in a real environment. This evaluation may include whether the estimated simulation environment closely mimics the real environment and whether the policy performs well in the simulated simulation environment. In other words, when it is determined that a good policy that exceeds existing methods has been obtained, the policy may be applied to the real environment.
[0097] Then, an A / B test may be performed using result A, which is obtained by using the actually estimated policy, and result B, which is obtained by using a policy such as an existing rule base. As a result of the A / B test, the policy may be switched to the one with the better estimation. Furthermore, if result A is inferior to result B, the result may be directly fed back to improve the policy. Furthermore, if necessary, the simulated environment may be improved, and the policy may be further improved in the improved simulated environment. In this way, fine-tuning makes it possible to improve the performance of the policy in the real environment.
[0098] There are roughly two possible reasons why the estimation by the estimation device 1 is inferior. One is when the simulation environment is different from the real environment, and the other is when the simulation environment reproduces the real environment with high accuracy, but the performance of the policy itself in the simulation environment is not high.
[0099] Whether or not the simulation environment deviates from the real environment can be determined by whether or not state transitions are reproduced well. In this case, the policy may be trained to reduce this deviation. The performance of the policy in the simulation environment can be actually tested in the simulation environment. In this case, further training of the policy in the simulation environment may be performed. Then, after the performance has been improved, the estimation device 1 may be tuned by performing A / B testing again.
[0100] For example, when training in a real environment (fine tuning), if the estimation of the estimation device is inappropriate and a problem occurs, for example, in an automated factory line, the measures may be inappropriate and an error may occur in the control of the device, and in that case, a human or other device may back up. In other words, a human worker or other device may correct or respond to an error in the device. For example, when a human and a robot (control device) under training work together, if the robot's task (e.g., picking) does not go well, the human can cover for it, and training can be performed to increase the success rate of the picking task.
[0101] When learning on a factory line or the like, the speed of the line may be set so that in the early stages of training the robot performs tasks at a speed that allows humans to cover up for the robot's mistakes.
[0102] Also, the line speed may be increased as training progresses.
[0103] In the above case, the speed adjustment may be based on the success rate, may be performed by a human, or may be performed based on a success rate calculated based on information obtained by a sensor such as an imaging device installed in the environment.
[0104] This allows models to be trained while maintaining line productivity.
[0105] In addition to changing the line speed, other ways to balance model training with line productivity include changing the ratio of the current policies known to be good to the new policies to be tested, which could be the best known current policy, such as one that worked in a simulation environment, or a random policy, for example to test whether a better policy exists.
[0106] To change this ratio, for example, the ε-greedy method may be used. In reinforcement learning, the ε-greedy search is a method of probabilistically trying both a random policy and the policy that is currently considered to be the best with probabilities ε and 1-ε.
[0107] (Modification) The estimation device 1 trained by the training device 2 as described above can be made even more robust. The training device 2 executes the training in an environment having different dynamics. The dynamics determine state transition probabilities, which indicate what state a transition will occur to from a current state and a selected action. In this case, not only can different dynamics be generated by changing parameters included in the predetermined state transition probabilities, but also more diverse dynamics can be expressed by varying the actions in the state transition probabilities.
[0108] The training device 2 may apply other information added to the environment or state, such as changes in motor noise. For example, the behavior may be interpreted as a domain randomization, where deviations are added to the behavior that depend on the particular state. In this case, the deviations added to the behavior may be implemented by adding disturbances to the output of the policy model's behavior.
[0109] Disturbance ε t To estimate using a model, for example, the disturbance ε t Let us denote by the environment-dependent parameter vector ω e The current state o weighted by t (e) For example, let Φ be a nonlinear mapping, specifically, a feedforward neural network with parameters τ known or randomly assigned, and the disturbance ε t =ω e Φ τ (o t (e) ) For example, ω e is expressed as a horizontal vector, and Φ τ (o t (e)) is expressed as a vertical vector, and ω e Φ τ (o t (e) ) represents the dot product of vectors. In this case, ω e This is done by identifying the perturbations due to motor noise in the environment e through the estimation of
[0110] At time step t, the original predicted behavior is a^ t (e) The behavior that is input to the simulator due to the variability of the behavior is expressed as follows, for example:
number
[0111] As described above, the dynamics parameter η for the environment e is e By expanding the above equation, it becomes possible for the estimation device 1 to estimate information added to the environment and state, for example, the influence of motor noise. This estimation device 1 can of course estimate the dynamics parameters η e This can be realized by modifying the following:
[0112] Below, we describe some results using the techniques in this disclosure.
[0113] FIG. 6 is a graph showing the reward of the estimation device 1 trained by the above-mentioned method. In the figure, (1) is f rec , g invThe result of training without using the model of (2) is g inv As a result of training using the model, (3) is f rec As a result of training using the model, (4) is f rec , g inv The results are from training using both models. (5) is a comparative example, and is the result of training using the correct dynamics parameters in the environment. It is theoretically possible to obtain a better reward than this. (6) is a comparative example, and is the result of training using MAML (Model Agnostic Meta-Learning), a method of meta-learning.
[0114] From this graph, f rec , g inv We can see that training with both of them gives better rewards than training with either one or neither of them. We also see that the results are better than MAML.
[0115] FIG. 7 is a graph showing the transition of rewards during training in the above-mentioned method. The horizontal axis is proportional to the number of episodes in training, and the vertical axis shows rewards. (4) and (5) are the same as those in FIG. 6. As a comparative example, the results using only domain randomization are shown. As shown in this graph, it can be seen that according to this embodiment, better results can be obtained from an early stage of training than other training methods.
[0116] FIG. 8 is a graph showing the effect of the disturbances described above. The horizontal axis shows the coefficient K related to the disturbances, and the vertical axis shows the reward. τ was selected randomly. The results with disturbances shown by the solid line are obtained by setting disturbances during training and adjusting ω by the above-mentioned method. e The results of estimating the behavior by estimating ω e Without encoding ω eAs shown in this graph, according to the present embodiment, better results can be obtained by taking disturbances into account and reflecting them in the latent state Z.
[0117] A part or all of each device (estimation device 1 or training device 2) in the above-mentioned embodiment may be configured with hardware, or may be configured with information processing of software (program) executed by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) or the like. In the case of being configured with information processing of software, software for realizing at least a part of the functions of each device in the above-mentioned embodiment may be stored in a non-transient storage medium (non-transient computer-readable medium) such as a flexible disk, a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by reading the software into a computer. The software may also be downloaded via a communication network. Furthermore, the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), so that the information processing is executed by hardware.
[0118] There is no limitation on the type of storage medium that stores the software. The storage medium is not limited to removable media such as magnetic disks or optical disks, but may be fixed storage media such as hard disks or memory. The storage medium may be provided inside the computer or outside the computer.
[0119] 9 is a block diagram showing an example of a hardware configuration of each device (the estimation device 1 or the training device 2) in the above-mentioned embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76.
[0120] Although the computer 7 in FIG. 9 includes one of each component, it may include a plurality of the same components. Although one computer 7 is shown in FIG. 9, the software may be installed in a plurality of computers, and each of the plurality of computers may execute the same or different parts of the software. In this case, it may be in the form of distributed computing in which each computer executes the process by communicating via a network interface 74 or the like. That is, each device (estimation device 1 or training device 2) in the above-mentioned embodiment may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize the function. Also, it may be configured such that information transmitted from a terminal is processed by one or more computers provided on a cloud, and the processing results are transmitted to the terminal.
[0121] Various calculations of each device (estimation device 1 or training device 2) in the above-mentioned embodiment may be executed in parallel using one or more processors, or using multiple computers via a network. Moreover, various calculations may be distributed to multiple computing cores in a processor and executed in parallel. Moreover, a part or all of the processes, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on a cloud that can communicate with a computer 7 via a network. Thus, each device in the above-mentioned embodiment may be in the form of parallel computing using one or more computers.
[0122] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) including a computer control device and an arithmetic device. The processor 71 may also be a semiconductor device including a dedicated processing circuit. The processor 71 is not limited to an electronic circuit using electronic logic elements, and may be realized by an optical circuit using optical logic elements. The processor 71 may also include an arithmetic function based on quantum computing.
[0123] The processor 71 performs arithmetic processing based on data and software (programs) input from each device, etc., configured internally of the computer 7, and can output arithmetic results and control signals to each device, etc. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.
[0124] Each device (estimation device 1 and / or training device 2) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other by wire or wirelessly.
[0125] The main memory 72 is a memory device that stores instructions executed by the processor 71 and various data, and information stored in the main memory 72 is read by the processor 71. The auxiliary memory 73 is a memory device other than the main memory 72. These memory devices refer to any electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memories may be either volatile or non-volatile memories. The memory devices for saving various data in each device (estimation device 1 or training device 2) in the above-mentioned embodiments may be realized by the main memory 72 or the auxiliary memory 73, or may be realized by an internal memory built into the processor 71. For example, the memory unit in the above-mentioned embodiments may be realized by the main memory 72 or the auxiliary memory 73.
[0126] A plurality of processors may be connected (coupled) to one storage device (memory), or a single processor may be connected. A plurality of storage devices (memories) may be connected (coupled) to one processor. When each device (estimation device 1 or training device 2) in the above-mentioned embodiment is composed of at least one storage device (memory) and a plurality of processors connected (coupled) to this at least one storage device (memory), a configuration may be included in which at least one processor among the plurality of processors is connected (coupled) to at least one storage device (memory). This configuration may also be realized by storage devices (memories) and processors included in a plurality of computers. Furthermore, a configuration in which a storage device (memory) is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.
[0127] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. The network interface 74 may be an appropriate interface, such as one conforming to an existing communication standard. The network interface 74 may exchange information with an external device 9A connected via the communication network 8. The communication network 8 may be any one of a wide area network (WAN), a local area network (LAN), a personal area network (PAN), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.
[0128] The device interface 75 is an interface such as a USB that directly connects to the external device 9B.
[0129] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device connected directly to the computer 7.
[0130] The external device 9A or the external device 9B may be, for example, an input device. The input device is, for example, a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the input device may be a device equipped with an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.
[0131] Moreover, the external device 9A or the external device 9B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs sound or the like. Alternatively, the output device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.
[0132] Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage device such as an HDD.
[0133] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (the estimation device 1 or the training device 2) in the above-mentioned embodiment. In other words, the computer 7 may transmit or receive some or all of the processing results of the external device 9A or the external device 9B.
[0134] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.
[0135] In this specification (including claims), when expressions such as "data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, it includes cases where various data itself is used as input, and cases where various data that have been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) are used as input. In addition, when it is stated that a result is obtained "based on / according to / in response to data," it includes cases where the result is obtained based only on the data, and may also include cases where the result is obtained influenced by other data, factors, conditions, and / or states other than the data. In addition, when it is stated that "data is output," it includes cases where various data itself is used as output, and cases where various data that have been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) are output, unless otherwise specified.
[0136] In this specification (including the claims), the terms "connected" and "coupled" are intended as open-ended terms including any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, physically connection / coupling, etc. The terms should be interpreted appropriately depending on the context in which the terms are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in the terms without any restriction.
[0137] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured / set to actually perform operation B. For example, when element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, when element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.
[0138] In this specification (including the claims), terms implying containing or possessing (e.g., "comprising / including" and "having") are intended to be open-ended terms that include the inclusion or possession of things other than the object indicated by the object of the term. When the object of such terms implies no quantity or a singular number (such as an article "a" or "an"), the expression should be construed as not being limited to a specific number.
[0139] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places and expressions that do not specify a quantity or suggest a singular number (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles a or an) should be interpreted as not necessarily being limited to a specific number.
[0140] In this specification, when a particular advantage / result is described as being obtained from a particular configuration of an embodiment, it should be understood that the same effect can also be obtained from one or more other embodiments having the same configuration, unless there is a reason to the contrary. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, and that the effect is not necessarily obtained by the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states are satisfied, and the effect is not necessarily obtained in the invention according to the claim that specifies the configuration or a similar configuration.
[0141] In this specification (including the claims), when a term such as "maximize" is used, it includes finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these maxima probabilistically or heuristically. Similarly, when a term such as "minimize" is used, it includes finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these minima probabilistically or heuristically. Similarly, when a term such as "optimize" is used, it includes finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these optimums probabilistically or heuristically.
[0142] In this specification (including claims), when a plurality of pieces of hardware perform a predetermined process, each piece of hardware may cooperate to perform the predetermined process, or a portion of the hardware may perform all of the predetermined process. Also, a portion of the hardware may perform a portion of the predetermined process, and another piece of hardware may perform the remainder of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. The hardware may include an electronic circuit, or a device including an electronic circuit.
[0143] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, replacements, and partial deletions are possible within the scope of the conceptual idea and intent of the present invention derived from the contents defined in the claims and their equivalents. For example, in all the above-mentioned embodiments, when numerical values or formulas are used in the explanation, they are shown as examples and are not limited to these. In addition, the order of each operation in the embodiments is shown as an example and is not limited to these. [Explanation of symbols]
[0144] 1: Estimation device, 10: 1st estimation part, 100: first encoder, 102: second encoder; 104: decoder, 12:Second estimation part, 120: Estimator, 122: Aggregator, 2: Training equipment, 20: forward propagation section, 22: Error calculation section, 24: Update section
Claims
1. one or more memories; one or more processors; The memory stores a trained model capable of estimating information of dynamics parameters; The one or more processors: estimating behavior information based on at least information on a state of a control object in a real environment and information on dynamics parameters estimated based on a time series of information on the state of the control object in the real environment and a time series of behavior using at least the trained model; The trained model is trained in a plurality of simulated environments simulating environments having different dynamics; The information of the dynamics parameters estimated by the trained model is an encoding result of the dynamics parameters; the information on the state of the control object is an encoding result of the state of the control object, The one or more processors: performing the encoding of the state and the encoding of the dynamics parameters using different trained models; Estimation device.
2. The trained model is trained by simulating the control target in a simulation environment in which dynamics parameters are known. The estimation device according to claim 1 .
3. The trained model is trained by supervised learning. The estimation device according to claim 1 or 2.
4. The one or more processors estimate information of dynamics parameters based on at least information of an observed state of the control object and information of an action performed by the control object using the trained model. The estimation device according to any one of claims 1 to 3.
5. The one or more processors estimate information of dynamics parameters based on at least a time series of information of an observed state of the control object in a real environment and information of an action performed by the control object in the real environment using the trained model. The estimation device according to any one of claims 1 to 3.
6. The one or more processors: Using the trained model, estimate information of dynamics parameters corresponding to at least a first time based on the time series corresponding to the first time, and estimate information of dynamics parameters corresponding to at least a second time based on the time series corresponding to the second time; estimating behavior information corresponding to the first time based on at least the state information corresponding to the first time and dynamics parameter information corresponding to the first time, and estimating behavior information corresponding to the second time based on the state information corresponding to the second time and dynamics parameter information corresponding to the second time; The estimation device according to claim 5 .
7. The information on the estimated behavior is information on the behavior to be performed. The estimation device according to any one of claims 1 to 6.
8. The one or more processors estimate action information based on at least state information of a control object and information of dynamics parameters estimated by the trained model using a strategy learned by reinforcement learning. The estimation device according to any one of claims 1 to 7.
9. the policy is learned by reinforcement learning performed in a simulation environment where dynamics parameters are known; The estimation device according to claim 8.
10. The control target is controlled based on the information of the estimated behavior. The estimation device according to any one of claims 1 to 9.
11. The dynamics parameters include the friction coefficient information, The estimation device according to any one of claims 1 to 10.
12. by the one or more processors, The frequency of encoding the state is different from the frequency of encoding the dynamics parameters. The estimation device according to any one of claims 1 to 11.
13. The information on the behavior is a probability distribution regarding the behavior. The estimation device according to any one of claims 1 to 12.
14. The probability distribution over the actions is output using a trained model. The estimation device according to claim 13.
15. one or more memories; one or more processors; The memory stores a trained model capable of estimating information of dynamics parameters; The one or more processors: estimating behavior information based on at least information on a state of a control object in a real environment and information on dynamics parameters estimated based on a time series of information on the state of the control object in the real environment and a time series of behavior using at least the trained model; The trained model is trained in a plurality of simulated environments simulating environments having different dynamics; The information of the dynamics parameters estimated by the trained model is an encoding result of the dynamics parameters; the information on the state of the control object is an encoding result of the state of the control object, The one or more processors: performing the encoding of the state and the encoding of the dynamics parameters using different trained models; system.
16. by one or more processors, A method for estimating behavior information based on at least information on a state of a control object in a real environment and information on dynamics parameters estimated based on time series information on the state of the control object in the real environment and time series information on behavior using at least a trained model capable of estimating information on dynamics parameters, the method comprising: The trained model is trained in a plurality of simulated environments simulating environments having different dynamics; The information of the dynamics parameters estimated by the trained model is an encoding result of the dynamics parameters; the information on the state of the control object is an encoding result of the state of the control object, by the one or more processors performing the encoding of the state and the encoding of the dynamics parameters using different trained models; method.
Citation Information
Patent Citations
Device, method and program for agent learning
JP2007052589A
State transition rule acquisition device, action selection learning device, action selection device, state transition rule acquisition method, action selection method, and program
JP2019079227A
Control system, learning device, control device and control method
JP2019165602A
Motion processing apparatus, motion processing method, and program
JP2019191821A
