Processing device, processing system, processing method, and recording medium

Abstract

This processing device comprises: a determination means that determines the value of a parameter of a simulator that simulates a change in the state of an environment in which a control target has been placed, by inputting, into a model representing the relationship between a control command, which provides instructions on an operation to be performed on the environment, a change in the state caused by the operation, and a value of the parameter, a control command and a change in the state caused by an operation; and an acquisition means that acquires the result of the simulation which has operated according to the value determined by the determination means.

Description

Processing device, processing system, processing method, and recording medium

[0001] The present disclosure relates to a processing device, a processing system, a processing method, and a recording medium.

[0002] Robots are used in various fields such as logistics. Patent Literature 1 discloses a related technique relating to a device that performs machine learning on image processing.

[0003] International Publication No. 2021 / 235311

[0004] In the technical field related to Patent Document 1, there is a demand for a technology that can improve convenience.

[0005] One of the objectives of each aspect of the present disclosure is to provide a processing device, a processing system, a processing method, and a recording medium that can solve the above-mentioned problems.

[0006] According to one aspect of the present disclosure, a processing device includes a determination means for determining the parameter values ​​of a simulator that simulates changes in the state of the environment in which a controlled object is placed by inputting a control command that instructs an action to be performed on the environment and the control command and the change in state due to the action into a model that represents the relationship between the change in state due to the action and the value of the parameter, and an acquisition means for acquiring the results of a simulation by the simulator that operates in accordance with the value determined by the determination means.

[0007] According to another aspect of the present disclosure, a processing system includes the processing device described above and a robot that operates based on a control command calculated by the processing device.

[0008] According to another aspect of the present disclosure, a processing method determines the parameter values ​​of a simulator that simulates changes in the state of an environment in which a controlled object is placed by inputting a control command that instructs an operation to be performed on the environment and the change in state due to the operation into a model that represents the relationship between the control command and the change in state due to the operation, and obtains the results of a simulation by the simulator that operates in accordance with the determined values.

[0009] According to another aspect of the present disclosure, a recording medium stores a program that causes a computer to execute the following steps: determine parameter values ​​of a simulator that simulates changes in the state of an environment in which a controlled object is placed by inputting control commands that instruct an action to be performed on the environment and the changes in state due to the action into a model that represents the relationship between the control commands and the changes in state due to the action, and the values ​​of the parameters; and obtain results of a simulation by the simulator that operates in accordance with the determined values.

[0010] According to each aspect of the present disclosure, convenience can be improved.

[0011] FIG. 1 is a diagram illustrating an example of a configuration of a processing device according to some embodiments of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of a skill learning device according to some embodiments of the present disclosure. FIG. 3 is a diagram illustrating an example of a processing flow of a processing device according to some embodiments of the present disclosure. FIG. 4 is a diagram illustrating an image of a robot hand pushing an object according to some embodiments of the present disclosure. FIG. 5 is a diagram illustrating an example of a processing flow of a processing device according to some embodiments of the present disclosure. FIG. 6 is a diagram illustrating an example of a configuration of a processing system according to some embodiments of the present disclosure. FIG. 7 is a diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure. FIG. 8 is a diagram illustrating an example of a configuration of a processing device according to some embodiments of the present disclosure. FIG. 9 is a diagram illustrating an example of a processing flow of a processing device according to some embodiments of the present disclosure. FIG. 10 is a diagram illustrating an example of a configuration of a processing device according to some embodiments of the present disclosure.

[0012] Hereinafter, an embodiment will be described in detail with reference to the drawings. <Embodiment> A processing device 1 according to an embodiment of the present disclosure is a device that estimates, through simulation, how to control a control target in order to move from a first state to a second state, which is a target state. For example, the processing device 1 estimates a control method for a robot to move an object (e.g., luggage, merchandise, etc.) from a certain location (i.e., the first state) to a target location (i.e., the second state). The processing device 1 may control the robot in accordance with the estimated control. Below, the processing device 1 will be described in more detail.

[0013] 1 is a diagram illustrating an example of the configuration of a processing device 1 according to some embodiments of the present disclosure. As shown in FIG. 1, the processing device 1 includes a target environment data acquisition unit 10, a storage unit 20, a second parameter distribution estimation unit 30, a storage unit 40, a parameter distribution setting unit 50, a physical simulator setting unit 60, a physical simulator data acquisition unit 70, a storage unit 80, a system model learning unit 90, and a skill learning device 100.

[0014] The target environment data acquisition unit 10 acquires data output from the skill learning device 100. The data output from the skill learning device 100 is a function π(s(t), ξ) indicating a control command a(t) estimated for control at each predetermined time set after the first time. s(t) is data indicating the state at time t. If the first parameter is ξ1 and the second parameter is ξ2, ξ is composed of the first parameter ξ1 and the second parameter ξ2. Note that the first parameter ξ1 is a parameter used in a simulation using a physical simulator (described below) and is a parameter whose value has already been determined. For example, in a simulation using a physical simulator in which a robot moves an object to a target position, an example of the first parameter is the target position. Furthermore, the second parameter ξ2 is a parameter used in a simulation using a physical simulator (described below) and is a parameter whose value needs to be set. For example, in a simulation using a physical simulator in which a robot moves an object to a target position, an example of the second parameter is the mass of the object, the center of gravity of the object, and the coefficient of friction between the object and the contact surface.

[0015] The memory unit 20 stores transition data ot. The transition data ot is a data set that indicates the data at each time t, with s(t), a(t), and s(t+1) as a set of three data. s indicates the state of the robot and its surroundings. a is data that indicates a control command. t represents a time step. s(t) is information that indicates the environment at time step t. s(t+1) is information that indicates the environment at time step (t+1). The time steps may be equally spaced or may be unevenly spaced. a(t) represents the control command at time step t. The transition data ot is transition data that can be obtained by applying each time to the data (i.e., a function) acquired by the target environment data acquisition unit 10.

[0016] The second parameter distribution estimation unit 30 estimates a probability distribution p(ξ2) of the value of the second parameter ξ2. For example, the method for determining a probability distribution described in Patent Document "PCT / JP2023 / 033543" may be used. Specifically, the physical simulator data acquisition unit 70 performs a physical simulation (e.g., a simulation of a robot's operation) based on the state s(t) and control command a(t) indicated in the transition data ot to calculate the next state s(t+1). A physical simulation is a computer-based simulation of the movement, posture, etc. of a substance (e.g., an object) in the assumed real world. For convenience of explanation, the next state obtained by the simulation will be referred to as s^(t+1). Furthermore, data (s(t), a(t), s(t+1), s^(t+1)) obtained by adding the next state s^(t+1) obtained by the simulation to the transition data (s(t), a(t), s(t+1)) will be referred to as predicted transition data o^t. That is, it can be written as o^t = (s(t), a(t), s(t+1), s^(t+1)). The second parameter distribution estimation unit 30 then finds the probability distribution p(ξ2) of the value of the second parameter ξ2 by solving an optimization problem using the square of the evaluation function |s(t+1) - s^(t+1)|, which indicates a better evaluation the smaller the magnitude of the difference between the next state s(t+1) indicated in the transition data ot and the next state s^(t+1) obtained by simulation.

[0017] The second parameter distribution estimator 30 may calculate the probability distribution of the values ​​of the second parameter ξ2 using a method different from that described above. For example, the second parameter distribution estimator 30 may calculate the probability distribution p(ξ2) of the values ​​of the second parameter ξ2 using the method described in Non-Patent Document “Gabriele Tiboni, Karol Arndt, V. Kyrki, DROPO: Sim-to-Real Transfer with Offline Domain Randomization, https: / / arxiv.org / pdf / 2201.08434.pdf.”

[0018] The storage unit 40 stores a probability distribution p(ξ1) of the value of the first parameter ξ1. The parameter distribution setting unit 50 sets, as a parameter distribution, a joint probability distribution p(ξ) of the probability distribution p(ξ1) of the first parameter ξ1 and the probability distribution p(ξ2) of the value of the second parameter ξ2, based on the probability distribution p(ξ1) of the first parameter ξ1 stored in the storage unit 40 and the probability distribution p(ξ2) of the value of the second parameter ξ2 estimated by the second parameter distribution estimation unit 30. The joint probability distribution p(ξ) is calculated by multiplying the probability distribution p(ξ1) of the first parameter ξ1 by the probability distribution p(ξ2) of the value of the second parameter ξ2.

[0019] The physics simulator setting unit 60 sets parameters for the physics simulator. Specifically, the physics simulator setting unit 60 samples the parameter ξ from the joint probability distribution p(ξ) set by the parameter distribution setting unit 50. Then, the physics simulator setting unit 60 sets the value of the sampled parameter ξ in the physics simulator. Examples of physics simulators include MuJoCo and Isaac Sim. Note that the physics simulator is not limited to the above. Specifically, the physics simulator may be any simulator as long as it can calculate a state s(t+1) at the next time (t+1) using the state s(t), control command a(t), and model parameter ξmodel at time t, as in s(t+1) = f(s(t), a(t), ξmodel). f represents the relationship between the state s(t), control command a(t), and parameter ξmodel and the state (t+1). The model parameter ξmodel is a partial element of the parameter ξ. For example, the physics simulator may be a non-differentiable simulator.

[0020] The physical simulator data acquisition unit 70 executes a physical simulation for each combination of parameters sampled and set by the physical simulator setting unit 60. The physical simulator data acquisition unit 70 then calculates the next state s(t+1) from the executed simulation. The physical simulator data acquisition unit 70 then records (s(t+1), s(t), a(t), ξ) in the storage unit 80 as parameterized transition data pt (hereinafter referred to as "parameterized transition data pt").

[0021] The storage unit 80 stores the parameterized transition data pt calculated by the physical simulator data acquisition unit 70 .

[0022] The system model learning unit 90 uses the transition data obtained from the physical simulator to learn a machine learning model, which is a system model. Examples of machine learning models include neural networks and Gaussian process regression models. This machine learning model has a simpler configuration than the physical simulator used by the physical simulator data acquisition unit 70 for simulation.

[0023] The skill learning device 100 performs skill learning using a trained machine learning model trained by the system model learning unit 90, instead of a physical simulator. Fig. 2 is a diagram illustrating an example of the configuration of the skill learning device 100 according to some embodiments of the present disclosure. As shown in Fig. 2, the skill learning device 100 includes a system model setting unit 1001, a system model data acquisition unit 1002, a storage unit 1003, and a skill learning unit 1004.

[0024] For example, if the physical simulator is a simulator that simulates the movement of a robot, the skill here is defined as a unit of the robot's movement. The skill may be defined as a combination of primitive movements performed by the robot. Furthermore, the skill may have parameters.

[0025] The system model setting unit 1001 sets parameters in a simulator for the simulation performed by the system model data acquisition unit 1002. Specifically, for example, the skill learning unit 1004 acquires the joint probability distribution p(ξ) set by the parameter distribution setting unit 50. In other words, p(ξ) represents a distribution that represents the value of the parameter ξ and the possibility (or frequency) of the occurrence of that value. The skill learning unit 1004 samples the parameter ξ from the acquired joint probability distribution p(ξ). The system model setting unit 1001 acquires a machine learning model trained by the system model learning unit 90. The system model setting unit 1001 also acquires the value of the parameter ξ sampled by the skill learning unit 1004. The system model setting unit 1001 then sets the trained machine learning model as a simulator for the simulation performed by the system model data acquisition unit 1002, and sets the sampled value of the parameter ξ as a setting value for the simulation performed by the system model data acquisition unit 1002.

[0026] The system model data acquisition unit 1002 executes a simulation by using as a simulator the machine learning model that has been trained by the system model learning unit 90. For example, the system model data acquisition unit 1002 acquires, from the skill learning unit 1004, the skills that the skill learning unit 1004 is currently training using a predetermined method. The system model data acquisition unit 1002 calculates parameterized transition data pt using the acquired skills.

[0027] Then, the system model data acquisition unit 1002 records the parameterized transition data pt calculated by the simulation in the storage unit 1003. The storage unit 1003 stores the parameterized transition data pt calculated by the system model data acquisition unit 1002.

[0028] The skill learning unit 1004 acquires the joint probability distribution p(ξ) set by the parameter distribution setting unit 50. The skill learning unit 1004 samples the parameter ξ from the acquired joint probability distribution p(ξ).

[0029] Furthermore, the skill learning unit 1004 performs skill learning using a predetermined method using the parameterized transition data pt stored in the storage unit 1003. An example of the predetermined method is the skill learning method described in Patent Document "PCT / JP2023 / 007289." The abstract system model set by the abstract system model setting unit 14 in Patent Document "PCT / JP2023 / 007289" corresponds to the trained machine learning model trained by the system model learning unit 90 in the present disclosure and output to the skill learning device 100. The skill learning unit 15 corresponds to the skill learning device 100 in the present disclosure. However, the predetermined method is not limited to the skill learning method described in Patent Document "PCT / JP2023 / 007289." When the skill learning unit 1004 performs skill learning using the parameterized transition data pt using a predetermined method, and the skill learning progresses, the system model data acquisition unit 1002 calculates parameterized transition data pt using the learned skill. Then, the system model data acquisition unit 1002 records the calculated parameterized transition data pt in the storage unit 1003. The storage unit 1003 stores the parameterized transition data pt calculated by the system model data acquisition unit 1002.

[0030] The above-described processing performed by the processing device 1 according to the embodiment of the present disclosure is merely an example, and the processing device 1 is not limited to the above-described processing. For example, the processing device 1 may perform the processing described below.

[0031] (Processing Performed by Processing Device) Fig. 3 is a diagram illustrating an example of a processing flow of the processing device 1 according to some embodiments of the present disclosure. Next, general processing performed by the processing device 1 during learning will be described with reference to Fig. 3.

[0032] The target environment data acquisition unit 10 acquires skills output from the skill learning device 100 (step S1). The target environment data acquisition unit 10 uses the acquired skills to collect data in a target environment (step S2). The target environment refers to the environment (state) in which a task will ultimately be executed, such as an assumed real-world environment (state). The target environment data acquisition unit 10 records the collected data as transition data ot in the storage unit 20 (step S3).

[0033] The second parameter distribution estimator 30 estimates a probability distribution p(ξ2) of the value of the second parameter ξ2 (step S4). For example, the physical simulator data acquirer 70 performs a physical simulation (e.g., a simulation of the robot's operation) under the state s(t) and control command a(t) indicated in the transition data ot to calculate the next state s^(t+1). The second parameter distribution estimator 30 calculates the probability distribution p(ξ2) of the value of the second parameter ξ2 by solving an optimization problem using the square of an evaluation function |s(t+1)-s^(t+1)|, which indicates a better evaluation the smaller the difference between the next state s(t+1) indicated in the transition data ot and the next state s^(t+1) obtained by the simulation.

[0034] The parameter distribution setting unit 50 sets the joint probability distribution p(ξ) of the probability distribution p(ξ1) of the first parameter ξ1 and the probability distribution p(ξ2) of the value of the second parameter ξ2 as the parameter distribution based on the probability distribution p(ξ1) of the first parameter ξ1 stored in the memory unit 40 and the probability distribution p(ξ2) of the value of the second parameter ξ2 estimated by the second parameter distribution estimation unit 30 (step S5).

[0035] The physics simulator setting unit 60 sets parameters in the physics simulator (step S6). Specifically, the physics simulator setting unit 60 samples the parameter ξ from the joint probability distribution p(ξ) set by the parameter distribution setting unit 50. The physics simulator setting unit 60 then sets the value of the sampled parameter ξ in the physics simulator.

[0036] The physical simulator data acquisition unit 70 executes a physical simulation for each combination of parameters sampled and set by the physical simulator setting unit 60 (step S7). The physical simulator data acquisition unit 70 calculates the next state s(t+1) from the executed simulation (step S8). The physical simulator data acquisition unit 70 then records the parameterized transition data pt (s(t+1), s(t), a(t), ξ) in the storage unit 80 (step S9).

[0037] The system model learning unit 90 uses the parameterized transition data pt obtained from the physical simulator to learn a machine learning model, which is a system model (step S10).

[0038] The skill learning device 100 performs skill learning using a trained machine learning model trained by the system model learning unit 90 instead of a physical simulator.

[0039] For example, the system model setting unit 1001 sets parameters in a simulator for the simulation performed by the system model data acquisition unit 1002 (step S11). Specifically, for example, the skill learning unit 1004 acquires the joint probability distribution p(ξ) set by the parameter distribution setting unit 50. The skill learning unit 1004 samples the parameter ξ from the acquired joint probability distribution p(ξ). The system model setting unit 1001 acquires a machine learning model trained by the system model learning unit 90. The system model setting unit 1001 also acquires the value of the parameter ξ sampled by the skill learning unit 1004. The system model setting unit 1001 then sets the trained machine learning model as a simulator for the simulation performed by the system model data acquisition unit 1002, and sets the value of the sampled parameter ξ as a setting value for the simulation performed by the system model data acquisition unit 1002.

[0040] The system model data acquisition unit 1002 executes a simulation by using the machine learning model trained by the system model learning unit 90 as a simulator (step S12). For example, the system model data acquisition unit 1002 acquires the skills being learned by the skill learning unit 1004 using a predetermined method from the skill learning unit 1004. The system model data acquisition unit 1002 calculates the parameterized transition data pt using the acquired skills.

[0041] Then, the system model data acquiring unit 1002 records the parameterized transition data pt calculated by the simulation in the storage unit 1003 (step S13). The storage unit 1003 stores the parameterized transition data pt calculated by the system model data acquiring unit 1002.

[0042] The skill learning unit 1004 executes skill learning in a predetermined manner using the parameterized transition data pt stored in the storage unit 1003 (step S14).

[0043] Note that when the skill learning unit 1004 performs skill learning using the transition data in a predetermined manner, and the skill learning progresses, the system model data acquisition unit 1002 calculates parameterized transition data pt using the learned skill. Then, the system model data acquisition unit 1002 records the calculated parameterized transition data pt in the storage unit 1003. The storage unit 1003 stores the parameterized transition data pt calculated by the system model data acquisition unit 1002. The skill learning unit 1004 performs skill learning using a predetermined method using the transition data stored in the storage unit 1003. That is, for example, skill learning continues by repeating the above-described process until it is determined that the data acquired by the physical simulator data acquisition unit 70 is the desired data.

[0044] Furthermore, the skill learning unit 1004 may perform learning using a learning method that uses reinforcement learning. Examples of learning methods that use reinforcement learning include the learning method described in the non-patent document "Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine, Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, https: / / arxiv.org / pdf / 1801.01290.pdf" and the learning method described in the non-patent document "John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, Proximal Policy Optimization Algorithms, https: / / arxiv.org / pdf / 1707.06347.pdf." However, learning methods that use reinforcement learning are not limited to these methods.

[0045] In reinforcement learning, for example, a value function V(ξcond) that predicts the future cumulative reward sum in a problem setting indicated by the parameter ξ or a condition vector ξcond obtained by extracting some of its elements is learned. In this case, the skill learning unit 1004 uses the obtained value function V(ξcond) to learn a policy π(s(t), ξcond) that outputs a control command a(t) in response to the input of the condition vector ξcond and the state s(t).

[0046] Furthermore, the skill learning unit 1004 may perform additional skill learning when it is determined that the task cannot be executed. The skill learning unit 1004 evaluates the value of the sampled condition vector ξcond based on the learned skills, and determines whether the task can be executed using the learned skills. Being able to execute the task here means that the task will be successful. Being unable to execute the task means that the task will be unsuccessful. Then, when it is determined that the task will be unsuccessful, the skill learning unit 1004 may perform additional learning using a learning method that uses reinforcement learning.

[0047] Next, we will explain a specific example in which the processing device 1 moves an object (e.g., luggage, goods, etc.) from a certain location (i.e., a first state) to a target location (i.e., a second state) by the robot pushing the object with the robot hand.

[0048] (Specific example during learning) First, a specific example during learning will be described. In this case, the task is for the robot hand to push one of a plurality of types of objects, thereby moving the object to a target position in a target posture. The first parameters are the initial positions and postures of the robot hand and the object, respectively, and the target position and posture. The second parameters are the mass of the object, the center of gravity of the object, and the friction coefficient of the bottom surface of the object. FIG. 4 is a diagram illustrating an image of a robot hand pushing an object according to some embodiments of the present disclosure. FIG. 5 is a diagram illustrating an example of a processing flow of the processing device 1 according to some embodiments of the present disclosure.

[0049] The processing device 1 identifies an object for which data has not been acquired from among multiple objects (step S21). The processing device 1 collects transition data when the robot hand performs an action of pushing the object (step S22). The processing device 1 estimates a parameter distribution from the collected transition data (step S23). The processing device 1 executes learning of a system model and skills based on the estimated parameter distribution (step S24).

[0050] (Specific example of control execution) Next, a specific example of control execution will be described. Fig. 6 is a diagram showing an example of the configuration of a processing system 2 according to some embodiments of the present disclosure. As shown in Fig. 6, the processing system 2 includes a processing device 1 and a robot 3.

[0051] The processing device 1 generates a control command, and then controls the robot 3 using the generated control command.

[0052] The robot 3 operates based on the control commands generated by the processing device 1. For example, the robot 3 includes a robot hand. The robot 3 operates the robot hand in accordance with the control commands generated by the processing device 1.

[0053] 7 is a diagram illustrating an example of a processing flow of a processing system 2 according to some embodiments of the present disclosure. In the processing system 2, the processing device 1 acquires transition data of an unknown object by performing a data acquisition operation (e.g., an operation in which a robot hand pushes or grasps an object) (step S31). The processing device 1 estimates a second parameter ξ2 of the object using a parameter distribution estimation method based on the acquired transition data (step S32). For example, the processing device 1 uses the parameter distribution estimation method to obtain a second parameter distribution p(ξ2). Then, the processing device 1 obtains a statistical representative value, such as a mean or median, from the second parameter distribution p(ξ2). The processing device 1 estimates the obtained statistical representative value as the second parameter ξ2.

[0054] The processing device 1 generates a parameter estimate ξ by combining the already obtained first parameter ξ1 and the estimated second parameter ξ2 (step S33). The processing device 1 executes control by inputting the generated parameter estimate ξ into the learned skill (step S34). For example, if the skill to be learned has a hierarchical structure as in Patent Document "PCT / JP2023 / 007289" and a lower controller (e.g., the low-level control unit 36 ​​in Patent Document "PCT / JP2023 / 007289") employs a control method based on a system model such as that described in Non-Patent Document "Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou, Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Driving, https: / / arxiv.org / pdf / 1707.02342.pdf," the system model learned by the learning method described in the embodiment of the present disclosure is utilized to calculate a final control command.

[0055] The processing device 1 outputs the calculated control command to the robot 3. The robot 3 operates the robot hand in accordance with the control command. For example, when the processing device 1 outputs a control command to the robot 3 that instructs the robot 3 to push an object with its robot hand, the robot 3 operates the robot hand in accordance with the control command and pushes the object with the robot hand.

[0056] (Advantages) The processing device 1 includes a physical simulator setting unit 60 (an example of a determining means) that determines the parameter values ​​of a simulator that simulates changes in the state of the environment in which a controlled object is placed by inputting the control commands and the changes in state due to the operations into a model that represents the relationship between the changes in state due to the operations and the values ​​of the parameters, and a physical simulator data acquiring unit 70 (an example of an acquiring means) that acquires the results of a simulation by the simulator that has operated in accordance with the values ​​determined by the physical simulator setting unit 60.

[0057] This processing device 1 allows a physical simulation to be set up taking into account the second parameter distribution p(ξ2) for the variable second parameter, and results can be obtained from the physical simulation based on this setting. In other words, the processing device 1 allows for highly accurate confirmation of operation while reducing the risk of damage to the actual object due to control errors. In other words, the processing device 1 can improve convenience.

[0058] <First variant of embodiment> In a first variant of an embodiment of the present disclosure, in the processing device 1, the skill learning device 100 may learn a skill using transition data with parameters pt stored in the memory unit 20 and transition data prepared based on the transition data stored in the memory unit 80.

[0059] (Configuration of Processing Device) Fig. 8 is a diagram illustrating an example of the configuration of a processing device 1 according to some embodiments of the present disclosure. As shown in Fig. 8, the processing device 1 includes a target environment data acquisition unit 10, a storage unit 20, a second parameter distribution estimation unit 30, a storage unit 40, a parameter distribution setting unit 50, a physical simulator setting unit 60, a physical simulator data acquisition unit 70, a storage unit 80, a system model learning unit 90, and a skill learning device 100. Furthermore, as shown in Fig. 8, the processing device 1 further includes a data configuration unit 110 and a storage unit 120. The following description will focus on the differences from the processing device 1 according to an embodiment of the present disclosure.

[0060] The data constructing unit 110 acquires the transition data stored in the storage unit 20 and the parameter-attached transition data pt stored in the storage unit 80. The data constructing unit 110 aggregates the acquired transition data into one parameter-attached transition data pt. At this time, a process for creating the parameter-attached transition data pt from the transition data stored in the storage unit 20 is required. For example, the parameter-attached transition data pt may be created by simply leaving parameter elements empty and storing transition data in other elements. Alternatively, the parameter-attached transition data pt may be created by storing parameter values ​​estimated from the transition data in the parameter elements. The data constructing unit 110 records the aggregated parameter-attached transition data pt in the storage unit 120. The storage unit 120 stores the parameter-attached transition data pt aggregated by the data constructing unit 110.

[0061] The above-described processing performed by the processing device 1 according to the first modified example of the embodiment of the present disclosure is merely an example, and the processing device 1 is not limited to the above-described processing. For example, the processing device 1 may perform the processing described below.

[0062] 9 is a diagram illustrating an example of a processing flow of the processing device 1 according to some embodiments of the present disclosure. The processing device 1 performs processing of steps S1 to S13. The storage unit 1003 then stores the parameterized transition data pt calculated by the system model data acquisition unit 1002.

[0063] The data composing unit 110 acquires the transition data stored in the storage unit 20 and the parameterized transition data pt stored in the storage unit 80 (step S41). The data composing unit 110 aggregates the acquired parameterized transition data pt into one piece of transition data. The data composing unit 110 records the aggregated parameterized transition data pt in the storage unit 120 (step S42). The storage unit 120 stores the parameterized transition data pt aggregated by the data composing unit 110.

[0064] The skill learning unit 1004 executes skill learning in a predetermined manner using the transition data stored in the storage unit 1003 and the parameterized transition data pt stored in the storage unit 120 (step S43).

[0065] Note that in the early stages of learning, the model has not yet progressed, and its reliability is low. Therefore, the reliability of the skill learning data obtained using such a model can also be said to be low. Therefore, for example, when performing skill learning in a predetermined manner using the parameterized transition data pt stored in the memory unit 1003 and the parameterized transition data pt stored in the memory unit 120, the skill learning unit 1004 may use a lower proportion of the simulation model data compared to the proportions of the physical simulation environment data and the target environment data. For example, the data may be labeled to indicate which environment it was obtained from, and the skill learning unit 1004 may use 40% of the data from the physical simulation environment and 40% of the data from the target environment for skill learning, and 20% of the data from the simulation model for skill learning. Furthermore, as skill learning progresses, the skill learning unit 1004 may increase the proportion of the simulation model data.

[0066] (Advantages) The first modified example of an embodiment of the present disclosure has been described above. In a processing device 1 according to a second modified example of an embodiment of the present disclosure, the data configuration unit 110 acquires the transition data stored in the storage unit 20 and the parameterized transition data pt stored in the storage unit 80. The data configuration unit 110 aggregates the acquired parameterized transition data pt into one parameterized transition data pt. The data configuration unit 110 records the aggregated parameterized transition data pt in the storage unit 120. The storage unit 120 stores the parameterized transition data pt aggregated by the data configuration unit 110. The skill learning unit 1004 performs skill learning using a predetermined method using the parameterized transition data pt stored in the storage unit 1003 and the parameterized transition data pt stored in the storage unit 120.

[0067] The processing device 1 of the first modified example of an embodiment of the present disclosure allows the skill learning unit 1004 to perform skill learning using more data than the processing device 1 of the embodiment of the present disclosure. As a result, the skill learning unit 1004 can proceed with skill learning more appropriately than the processing device 1 of the embodiment of the present disclosure. In other words, the processing device 1 of the first modified example of an embodiment of the present disclosure can improve convenience.

[0068] Furthermore, the processing device 1 according to the first modification of the embodiment of the present disclosure can change the proportion of data used for skill learning depending on the environment in which the data was acquired. As a result, the processing device 1 according to the first modification of the embodiment of the present disclosure can more efficiently progress skill learning. In other words, the processing device 1 according to the first modification of the embodiment of the present disclosure can improve convenience.

[0069] <Second Variant of the Embodiment> In a second variant of an embodiment of the present disclosure, in the processing device 1, the skill learning device 100 may further expand the parameter-attached transition data pt used for skill learning by using the transition data stored in the memory unit 20 and the parameter-attached transition data pt stored in the memory unit 80.

[0070] (Configuration of Processing Device) FIG. 10 is a diagram illustrating an example of the configuration of a processing device 1 according to some embodiments of the present disclosure. As shown in FIG. 10 , the processing device 1 includes a target environment data acquisition unit 10, a storage unit 20, a second parameter distribution estimation unit 30, a storage unit 40, a parameter distribution setting unit 50, a physical simulator setting unit 60, a physical simulator data acquisition unit 70, a storage unit 80, a system model learning unit 90, a skill learning device 100, a data configuration unit 110, and a storage unit 120. Furthermore, as shown in FIG. 10 , the processing device 1 further includes a data extension unit 130. Below, differences from the processing device 1 according to a first modification of an embodiment of the present disclosure will be mainly described.

[0071] The data extension unit 130 generates new parameter-attached transition data pt to be used for skill learning, based on the parameter-attached transition data pt stored in the storage unit 120. For example, the data extension unit 130 generates new parameter-attached transition data pt to be used for skill learning using a method for extending transition data described in non-patent document "Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba, Hindsight Experience Replay, https: / / arxiv.org / pdf / 1707.01495.pdf" or the like.

[0072] 11 is a diagram illustrating an example of a processing flow of the processing device 1 according to some embodiments of the present disclosure. The processing device 1 performs processing of steps S1 to S13. Then, the storage unit 1003 stores the parameterized transition data pt calculated by the system model data acquisition unit 1002.

[0073] Furthermore, the processing device 1 performs the processes of steps S41 and S42. The storage unit 120 stores the parameterized transition data pt compiled by the data construction unit 110.

[0074] The data expansion unit 130 generates new parameter-attached transition data pt to be used for skill learning based on the parameter-attached transition data pt stored in the storage unit 120 (step S51). The data expansion unit 130 records the generated new parameter-attached transition data pt in the storage unit 120. The storage unit 120 stores the new parameter-attached transition data pt generated by the data expansion unit 130.

[0075] The skill learning unit 1004 performs skill learning in a predetermined manner using the parameterized transition data pt stored in the memory unit 1003, the parameterized transition data pt compiled by the data composition unit 110 stored in the memory unit 120, and new parameterized transition data pt generated by the data expansion unit 130 stored in the memory unit 120 (step S52).

[0076] (Advantages) The second modified example of an embodiment of the present disclosure has been described above. In the processing device 1 of the second modified example of an embodiment of the present disclosure, the data expansion unit 130 generates new parameter-attached transition data pt to be used for skill learning based on the parameter-attached transition data pt stored in the storage unit 120. The data expansion unit 130 records the generated new parameter-attached transition data pt in the storage unit 120. The storage unit 120 stores the new parameter-attached transition data pt generated by the data expansion unit 130.

[0077] The processing device 1 of the second modified example of an embodiment of the present disclosure allows the skill learning unit 1004 to perform skill learning using even more data than the processing device 1 of the first modified example of an embodiment of the present disclosure. As a result, the skill learning unit 1004 can proceed with skill learning more appropriately than the processing device 1 of the first modified example of an embodiment of the present disclosure. In other words, the processing device 1 of the second modified example of an embodiment of the present disclosure can improve convenience.

[0078] 12 is a diagram illustrating an example of the configuration of the processing device 1 according to some embodiments of the present disclosure. As illustrated in FIG. 12, the processing device 1 includes a determining unit 301 and an acquiring unit 302.

[0079] The determining means 301 determines the parameter values ​​of a simulator that simulates changes in the state of the environment in which the controlled object is placed by inputting the control command and the state changes due to the operation into a model that represents the relationship between the control command instructing an operation to be performed on the environment, the state changes due to the operation, and the parameter values. The acquiring means 302 acquires the results of the simulation by the simulator that operated in accordance with the values ​​determined by the determining means 301.

[0080] The determining means 301 can be realized, for example, by using the functions of the physical simulator setting unit 60 illustrated in Figures 2, 8, and 10. The acquiring means 302 can be realized, for example, by using the functions of the physical simulator data acquiring unit 70 illustrated in Figures 2, 8, and 10.

[0081] 13 is a diagram illustrating an example of a processing flow of the processing device 1 according to some embodiments of the present disclosure. Next, processing of the processing device 1 according to some embodiments of the present disclosure will be described with reference to FIG.

[0082] The determining means 301 determines the parameter values ​​of a simulator that simulates changes in the state of the environment in which the controlled object is placed by inputting the control command and the state changes caused by the operation into a model that represents the relationship between the control command instructing an operation to be performed on the environment, the state changes caused by the operation, and the parameter values ​​(step S101). The acquiring means 302 acquires the results of the simulation by the simulator that operated in accordance with the values ​​determined by the determining means 301 (step S102).

[0083] The processing device 1 according to some embodiments of the present disclosure has been described above. The processing device 1 can improve convenience.

[0084] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0085] Although the embodiments of the present disclosure have been described, the processing device 1, processing system 2, robot 3, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0086] 14 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 14 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the processing device 1, the processing system 2, the robot 3, and other control devices described above are each implemented in the computer 5. The operation of each of the processing units described above is stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-mentioned processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above in accordance with the program.

[0087] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, if the program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0088] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0089] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0090] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0091] (Supplementary Note 1) A processing device comprising: a determining means for determining parameter values ​​of a simulator that simulates changes in the state of an environment in which a controlled object is placed by inputting a control command that instructs an operation to be performed on the environment and the control command and the changes in state due to the operation into a model that represents the relationship between the changes in state due to the operation and the values ​​of the parameters; and an acquiring means for acquiring results of a simulation by the simulator that has operated in accordance with the values ​​determined by the determining means.

[0092] (Supplementary Note 2) The processing device according to Supplementary Note 1, further comprising: a generating unit configured to generate a model indicating a relationship between the parameters and the state based on the result acquired by the acquiring unit, for estimating the result faster than the simulation.

[0093] (Supplementary Note 3) The processing device according to Supplementary Note 2, further comprising: a learning unit configured to learn a skill using the model generated by the generation unit.

[0094] (Supplementary Note 4) The processing device according to any one of Supplementary Note 1 to Supplementary Note 3, further comprising: a generation means for generating learning data to be used for learning a skill based on a set state of the object and results obtained from the simulation.

[0095] (Supplementary Note 5) The processing device according to any one of Supplementary Note 1 to Supplementary Note 4, further comprising: a control unit that calculates a control command and controls the robot based on the calculated control command.

[0096] (Supplementary Note 6) A processing system comprising: the processing device according to any one of Supplementary Note 1 to Supplementary Note 5; and a robot that operates based on a control command calculated by the processing device.

[0097] (Supplementary Note 7) A processing method comprising determining parameter values ​​of a simulator that simulates changes in the state of an environment in which a controlled object is placed by inputting a control command that instructs an operation to be performed on the environment and the state changes due to the operation into a model that represents the relationship between the control command and the state changes due to the operation, and obtaining the results of a simulation by the simulator that operates in accordance with the determined values.

[0098] (Supplementary Note 8) The processing method according to Supplementary Note 7, further comprising generating a model indicating a relationship between the parameters and the state based on the obtained results, for estimating the results faster than the simulation.

[0099] (Supplementary Note 9) The processing method according to Supplementary Note 8, further comprising: learning a skill using the generated model.

[0100] (Supplementary Note 10) The processing method according to any one of Supplementary Note 7 to Supplementary Note 9, further comprising generating learning data to be used for learning a skill based on a set state of the object and results obtained from the simulation.

[0101] (Supplementary Note 11) The processing method according to any one of Supplementary Note 7 to Supplementary Note 10, further comprising: calculating a control command; and controlling the robot based on the calculated control command.

[0102] (Supplementary Note 12) A recording medium storing a program that causes a computer to execute the following steps: determining parameter values ​​of a simulator that simulates changes in the state of an environment in which a controlled object is placed by inputting a control command that instructs an operation to be performed on the environment and the change in state due to the operation into a model that represents the relationship between the control command and the change in state due to the operation, and the value of the parameter; and obtaining results of a simulation by the simulator that operates in accordance with the determined values.

[0103] (Supplementary Note 13) The recording medium according to Supplementary Note 12, storing a program for causing a computer to execute the following: generating a model indicating a relationship between the parameters and the state, based on the obtained result, for estimating the result faster than the simulation.

[0104] (Supplementary Note 14) The recording medium according to Supplementary Note 13, storing a program that causes a computer to execute the steps of: learning a skill using the generated model.

[0105] (Appendix 15) A recording medium according to any one of Appendices 12 to 14, storing a program for causing a computer to execute the following steps: generate learning data to be used for learning a skill based on a set state of the object and the results obtained from the simulation.

[0106] (Supplementary Note 16) A recording medium according to any one of Supplementary Note 12 to Supplementary Note 15, storing a program that causes a computer to execute the following steps: calculate a control command; and control a robot based on the calculated control command.

[0107] According to each aspect of the present disclosure, convenience can be improved.

[0108] DESCRIPTION OF SYMBOLS 1 Processing device 2 Processing system 3 Robot 5 Computer 6 CPU 7 Main memory 8 Storage 9 Interface 10 Target environment data acquisition unit 20, 40, 80, 120 Storage unit 30 Second parameter distribution estimation unit 50 Parameter distribution setting unit 60 Physical simulator setting unit 70 Physical simulator data acquisition unit 90 System model learning unit 100 Skill learning device 110 Data configuration unit 130 Data expansion unit

Claims

1. A processing device comprising: a determination means for determining the parameter values ​​of a simulator that simulates changes in the state of an environment in which a controlled object is placed by inputting a control command that instructs an action to be performed on the environment and the control command and the change in state due to the action into a model that represents the relationship between the change in state due to the action and the value of the parameter; and an acquisition means for acquiring the results of a simulation by the simulator that has operated in accordance with the value determined by the determination means.

2. The processing device according to claim 1, further comprising: a generating means for generating a model indicating the relationship between the parameters and the state, based on the results acquired by the acquiring means, for estimating the results faster than the simulation.

3. The processing device according to claim 2, further comprising: learning means for learning a skill using the model generated by the generation means.

4. The processing device according to claim 1, further comprising: a generating means for generating learning data to be used for learning a skill based on the set state of the object and the results obtained from the simulation.

5. The processing device according to claim 1, further comprising: control means for calculating a control command and controlling the robot based on the calculated control command.

6. A processing system comprising: the processing device according to claim 1; and a robot that operates based on a control command calculated by said processing device.

7. A processing method in which the parameter values ​​of a simulator that simulates changes in the state of the environment in which a controlled object is placed are determined by inputting control commands that instruct the operation to be performed on the environment and the state changes due to the operation into a model that represents the relationship between the control commands and the state changes due to the operation, and the parameter values, and obtaining the results of a simulation by the simulator that operates in accordance with the determined values.

8. A recording medium storing a program that causes a computer to execute the following steps: determining the parameter values ​​of a simulator that simulates changes in the state of the environment in which a controlled object is placed by inputting control commands that instruct the operation to be performed on the environment and the state changes due to the operation into a model that represents the relationship between the control commands and the state changes due to the operation; and obtaining the results of a simulation by the simulator that operates in accordance with the determined values.

Images