Operation method learning system, operation method learning method, and program

Abstract

To provide an operation method learning system, an operation method learning method, and a program which enable the execution of a target task robustly and precisely by using a machine learning model even when information inputted into the machine learning model is a partial observation.SOLUTION: An operation method learning system acquires observation data at a first time of a robot, calculates a first feature amount from the observation data by using a first encoder, calculates a second feature amount at the first time from an action of the robot at a second time, the second feature amount at the second time, and the first feature amount at the first time by using a recurrent second encoder by which the second feature amount is retained at the second time, decides the action of the robot at the first time on the basis of the second feature amount at the first time, and learns at least the first encoder and the second encoder by using comparison learning.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an operation method learning system, an operation method learning method, and a program. [Background technology]

[0002] Conventionally, to operate or recognize an object grasped by a robot hand (also called an end effector), or to recognize the object itself, control by planning based on a model designed by humans or a matching-based recognizer using an object model has been used. In recent years, there have been many approaches that use data-driven learning methods to acquire planners and recognizers.

[0003] While human-designed planning methods and matching-based recognition methods that utilize object model information are limited in their applicability and lack robustness and generalization capabilities, data-driven learning methods are known to have high robustness and generalization capabilities within the range of data included in the training data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Irmak Guzey, Ben Evans, Soumith Chintala, Lerrel Pinto, “Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play,” In CoRL 2023, 31 Aug 2023, doi:10.48550 / arXiv.2303.12076. Summary of the Invention [Problem to be solved by the invention]

[0005] However, when using information from a tactile sensor mounted on a robot as input to a learning-based planner or recognizer (hereinafter referred to as a machine learning model), only information on the position of the object in contact is obtained, which tends to result in partial observation (i.e., sparse information). Even when using a camera, occlusion of the target object by the robot itself or obstacles often occurs, resulting in only sparse information. In such cases, it can be difficult to train the machine learning model or its performance can be degraded.

[0006] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide an operation method learning system, an operation method learning method, and a program that can perform a target task robustly and with high accuracy using a machine learning model even if the information input to the machine learning model is partial observation (sparse). [Means for solving the problem]

[0007] The operation method learning system, operation method learning method, and program according to the present invention employ the following configurations. (1) A first example of the present invention is a manipulation method learning system including: an acquisition unit that acquires observation data indicating observation results of a state of a robot capable of manipulating an object at a first time; a first calculation unit that calculates a first feature amount at the first time from the observation data at the first time using a first encoder; a second calculation unit that calculates a second feature amount at the first time from a behavior of the robot at the second time, the second feature amount at the second time, and the first feature amount at the first time using a recursive second encoder that holds a second feature amount at a second time before the first time; a decision unit that determines the behavior of the robot at the first time based on the second feature amount at the first time; and a learning unit that learns at least the first encoder and the second encoder using contrastive learning.

[0008] (2) A second example of the present invention is that in the first example, the learning unit learns at least the first encoder and the second encoder as the contrastive learning so that the second feature calculated using the second encoder approaches the first target feature and the second feature moves away from the second target feature.

[0009] (3) A third example of the present invention is that, in the second example, the learning unit extracts the observation data at a first reference time and the observation data at a second reference time that is earlier than the first reference time from the observation data in a time series that includes the observation data at the first time and the second time, inputs at least the observation data at the first reference time to a third encoder, and defines a latent variable output by the third encoder in response to at least the observation data at the first reference time as the first target feature, and inputs at least the observation data at the second reference time to the third encoder, and defines a latent variable output by the third encoder in response to at least the observation data at the second reference time as the second target feature.

[0010] (4) A fourth example of the present invention is the third example, wherein the decision unit determines the behavior of the robot at the first time using reinforcement learning, and the learning unit learns the first encoder and the second encoder using a reward from the reinforcement learning.

[0011] (5) A fifth example of the present invention is that in the fourth example, the learning unit further inputs the reward at the first reference time to the third encoder in addition to the observation data at the first reference time, and sets the latent variable output by the third encoder in response to the input of the observation data and the reward at the first reference time as the first target feature, and further inputs the reward at the second reference time to the third encoder in addition to the observation data at the second reference time, and sets the latent variable output by the third encoder in response to the input of the observation data and the reward at the second reference time as the second target feature.

[0012] (6) A sixth example of the present invention is that, in the first or second example, the first calculation unit inputs the observation data at the first time to the first encoder, and calculates a latent variable output by the first encoder in response to the input of the observation data at the first time as the first feature at the first time.

[0013] (7) A seventh example of the present invention is that, in the first or second example, the second calculation unit inputs the first feature at the first time to the second encoder, and calculates a latent variable output by the second encoder in response to the input of the first feature at the first time as the second feature at the first time.

[0014] (8) An eighth example of the present invention is a method for learning an operation method, including: acquiring observation data indicating an observation result of a state of a robot capable of manipulating an object at a first time; calculating a first feature at the first time from the observation data at the first time using a first encoder; calculating a second feature at the first time from a behavior of the robot at the second time, the second feature at the second time, and the first feature at the first time using a recursive second encoder that holds a second feature at a second time before the first time; determining a behavior of the robot at the first time based on the second feature at the first time; and learning the first encoder and the second encoder using contrastive learning.

[0015] (9) A ninth example of the present invention is a program to be executed by a computer, the program including: acquiring observation data indicating an observation result of a state of a robot capable of manipulating an object at a first time; calculating, using a first encoder, a first feature at the first time from the observation data at the first time; calculating, using a recursive second encoder that holds a second feature at a second time before the first time, a second feature at the first time from a behavior of the robot at the second time, the second feature at the second time, and the first feature at the first time; determining a behavior of the robot at the first time based on the second feature at the first time; and learning the first encoder and the second encoder using contrastive learning. [Effects of the Invention]

[0016] According to the above example, even if the information input to the machine learning model is partial observation (sparse), the machine learning model can be used to perform the target task robustly and with high accuracy. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram schematically illustrating the appearance of a robot 10 included in an operation method learning system 1 according to an embodiment. [Figure 2] 1 is a configuration diagram of an operation method learning system 1 according to an embodiment. [Figure 3] 10 is a flowchart showing the flow of a series of processes by a processing unit 120 according to the embodiment. [Figure 4] FIG. 2 is a diagram schematically illustrating processing contents of a processing unit 120 according to the embodiment. [Figure 5] This diagram shows a schematic diagram of how each encoder is trained using two approaches: contrastive training and deep reinforcement learning. [Figure 6] FIG. 10 is a diagram illustrating an example of a task. [Figure 7] FIG. 10 is a diagram illustrating an example of a task. [Figure 8] FIG. 10 is a diagram illustrating an example of a task. [Figure 9] FIG. 10 is a diagram illustrating an example of a task. [Figure 10] FIG. 10 is a diagram showing the results of a comparison between the learning method of the present embodiment and other methods for each task. [Figure 11] FIG. 10 is a diagram showing the results of a comparison between the learning method of the present embodiment and other methods for each task. [Figure 12] FIG. 10 is a diagram showing the results of a comparison between the learning method of the present embodiment and other methods for each task. [Figure 13] FIG. 10 is a diagram showing the results of a comparison between the learning method of the present embodiment and other methods for each task. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of an operation learning system, an operation learning method, and a program according to the present invention will be described with reference to the drawings.

[0019] [Robot appearance] 1 is a diagram schematically illustrating the appearance of a robot 10 included in an operation method learning system 1 according to an embodiment. The robot 10 is typically a humanoid robot that can grasp or manipulate an object OB using an end effector 11, but is not limited to this and may be any type of robot that can grasp or manipulate an object OB. For example, the robot 10 may be a quadrupedal animal-type robot or any other type of robot.

[0020] The end effector 11 may also be referred to as a robot hand. The end effector 11 may be provided with, for example, several fingers (thumb, index finger, middle finger, ring finger, etc.) as grippers.

[0021] The end effector 11 is provided with a plurality of tactile sensors 13, a plurality of force sensors 14, a plurality of attitude sensors 15, and the like.

[0022] The tactile sensors 13 are, for example, distributed over the palm of the end effector 11. Specifically, a total of 224 tactile sensors 13 may be arranged on the palm of the end effector 11. In other words, the tactile sensors 13 may detect the force applied to the palm through a total of 224 channels. Each channel of the tactile sensor 13 is called a tactile pixel, also known as a taxel.

[0023] The force sensors 14 are arranged, for example, on the fingertips of the end effector 11, and detect forces (loads) on three axes (X, Y, Z) applied to each fingertip and moments (torques) around each axis. For example, if the end effector 11 is provided with a thumb, index finger, middle finger, and ring finger, one force sensor 14 is arranged on each finger, and forces and moments are detected through a total of 4 × 6 = 24 channels.

[0024] The posture sensor 15 is disposed on, for example, each finger of the end effector 11 and detects the posture of each finger. The posture detected by the posture sensor 15 is typically the joint angle of each finger, but is not limited to this and may be the angular velocity or torque of the joint angle, or a combination of these. In the following, as an example, the posture detected by the posture sensor 15 will be described as the joint angle.

[0025] For example, if the end effector 11 has a thumb, index finger, middle finger, and ring finger, and each finger has four joints, the orientation sensor 15 detects the joint angles through a total of 4×4=16 channels.

[0026] The number of tactile sensors 13 is not limited to 224, and may be any number, for example, from several tens to several hundreds. Similarly, the numbers of force sensors 14 and attitude sensors 15 may also be any numbers.

[0027] In addition to the end effector 11, the robot 10 may further include, for example, a visual sensor 12 for imaging the external environment or workspace seen from the robot 10, a control device 100 for controlling the operation of the robot 10, etc. The robot 10 performs a target task according to the actions determined by the control device 100.

[0028] The tasks include, for example, grasping an object OB with the end effector 11, transferring the object OB to the other end effector 11, or moving the object OB. However, the tasks are not limited to these, and any task can be set.

[0029] The visual sensor 12 is installed in a part of the body of the robot 10 (typically, the head). The visual sensor 12 may be, for example, a depth camera (3D camera). For example, the visual sensor 12 captures an image of a scene in which the object OB is grasped or manipulated by the end effector 11, and transmits video data of the scene to the control device 100 or to an external device (for example, a human-machine interface) via the control device 100. Note that the visual sensor 12 is not limited to a depth camera and may be, for example, a sensor that creates an image of the external environment by emitting electromagnetic waves, such as a radar or lidar.

[0030] Furthermore, the above-mentioned video data may be video data generated by an external camera (not shown) installed in the workspace of the robot 10 in addition to or instead of the video data from the visual sensor 12 installed in the robot 10. The external camera installed in the workspace of the robot 10 may be used to perform image analysis such as pattern matching. The image analysis may be image analysis that extracts an object OB from the video.

[0031] The control device 100 controls the robot 10 to perform a target task by using, for example, data indicating the detection results of various sensors (tactile sensor 13, force sensor 14, and attitude sensor 15) provided on the end effector 11 and the visual sensor 12. In this embodiment, the control device 100 compresses the dimensions of data from the various sensors provided on the robot 10 using multiple encoders, and determines the behavior of the robot 10 using the compressed data to achieve a given task. In this case, the processing unit 120 introduces contrastive learning to train the multiple encoders.

[0032] The control device 100 may typically be mounted on the robot 10. Alternatively, instead of being mounted on the robot 10, the control device 100 may be installed at a location far away from the robot 10 and control the robot 10 remotely via a network NW. The network NW includes, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).

[0033] [Robot and control device configuration] 2 is a configuration diagram of an operation method learning system 1 according to an embodiment. The operation method learning system 1 includes, for example, a robot 10 and a control device 100. The robot 10 includes an actuator 16 and a drive control unit 17 in addition to the end effector 11, visual sensor 12, tactile sensor 13, force sensor 14, and attitude sensor 15 described above.

[0034] The actuators 16 drive the various parts (arms, fingers, legs, head, torso, waist, etc.) of the robot 10 under the control of the drive control unit 17. The actuators 16 include, for example, electromagnetic motors, gears, artificial muscles, etc.

[0035] The drive control unit 17 controls the actuator 16 based on the control command generated by the control device 100 .

[0036] The control device 100 includes, for example, a communication interface 110, a processing unit 120, and a storage unit .

[0037] The communication interface 110 communicates with the robot 10 via a communication line such as a bus, and with an external device via a network NW. The communication interface 110 includes, for example, a wireless communication module including a receiver and a transmitter, and a network interface card (NIC).

[0038] The processing unit 120 includes, for example, an acquisition unit 121, a first calculation unit 122, a second calculation unit 123, a behavior determination unit 124, a command generation unit 125, a communication control unit 126, and a learning unit 127.

[0039] The components of the processing unit 120 are realized by, for example, a central processing unit (CPU) or a graphics processing unit (GPU) executing a program stored in the storage unit 130. Some or all of these components may be realized by hardware such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a system-on-chip (SOC), or may be realized by a combination of software and hardware.

[0040] The storage unit 130 is realized by, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), or a random access memory (RAM). The storage unit 130 stores model data in addition to various programs such as firmware and application programs. The model data is data (programs or algorithms) that define several machine learning models for determining the behavior of the robot 10. For example, the model data defines a query encoder MDL1, a temporal encoder MDL2, a key encoder MDL3, and a deep reinforcement learning model MDL4, which will be described later. The model data may be installed in the storage unit 130 from an external device via a network NW, or from a portable storage medium connected to a drive device of the control device 100.

[0041] [Processing flow] The processing contents of the processing unit 120 will be described below using a flowchart and a schematic diagram. FIG. 3 is a flowchart showing the flow of a series of processes of the processing unit 120 according to the embodiment. FIG. 4 is a diagram showing a schematic diagram of the processing contents of the processing unit 120 according to the embodiment. The processing of this flowchart may be repeatedly executed until the target task is achieved. More specifically, it may be repeatedly executed until the reward r of reinforcement learning described below converges.

[0042] First, the acquisition unit 121 acquires an environmental state s t Observation data O at time t is obtained from the robot 10 placed below. t (Step S100). The time t is an example of a "first time".

[0043] Observation data O t is multidimensional data that combines data indicating the forces and moments detected by the tactile sensor 13 and the force sensor 14 at time t (hereinafter referred to as tactile data) and data indicating the posture (joint angles) detected by the posture sensor 15 at the same time t (hereinafter referred to as posture data).

[0044] The tactile data is expressed as a 248-dimensional vector that combines the detection results (= 224 dimensions) of the tactile sensors 13 distributed on the palm and the detection results (= 4 fingers x 6 axes = 24 dimensions) of the 6-axis force sensors 14 on each finger.

[0045] The posture data is expressed as a 16-dimensional vector of the detection results of the posture sensor 15 for each finger (=4 fingers×4 joint count).

[0046] Therefore, the observation data O t is expressed as a multidimensional vector with 248+16=264 dimensions. Note that the number of dimensions of the observation data Ot is not limited to 264, and may vary depending on the number of sensors and the number of detected physical indices.

[0047] Next, the first calculation unit 122 calculates the observation data O acquired by the acquisition unit 121 using the query encoder MDL1.t The first feature amount at time t is calculated from MDL. MDL is simply an abbreviation for MODEL.

[0048] The query encoder MDL1 is a machine learning model that is trained to compress the dimensions of input data, and may be implemented by, for example, a multi-layer perceptron (MLP) or a convolutional neural network (CNN).

[0049] The first calculation unit 122 calculates the observed data O t The query encoder MDL1 inputs the observation data O t In response to the input, the observation data O t In other words, the query encoder MDL1 outputs a first latent variable vector having a smaller dimension than the observed data O t is embedded in the first latent space as a first latent variable vector.

[0050] The first calculation unit 122 calculates the first latent variable vector output by the query encoder MDL1 as the first feature amount at time t. The query encoder MDL1 is an example of a "first encoder."

[0051] Next, the second calculation unit 123 calculates the behavior a of the robot 10 at time t−1, which is before time t, using the temporal encoder MDL2 that performs recursive processing. t-1 The second feature amount at time t is calculated from the second feature amount at time t-1 and the first feature amount at time t calculated by the first calculation unit 122. Time t-1, which is before time t, is an example of a "second time."

[0052] The temporal encoder MDL2 is a machine learning model that is trained to compress the dimensions of input data, and may be implemented, for example, by combining two types of machine learning models. Hereinafter, the first temporal encoder MDL2 will be referred to as the first temporal encoder MDL2-1, and the second temporal encoder MDL2 will be referred to as the second temporal encoder MDL2-2. The temporal encoder MDL2 is an example of a "second encoder."

[0053] For example, the first temporal encoder MDL2-1 may be implemented by a recurrent neural network (RNN) including a long short-term memory (LSTM), and the second temporal encoder MDL2-2 may be implemented by an MLP. The LSTM temporarily stores a second feature value at time t-1. The second feature value stored in the LSTM is repeatedly updated recursively.

[0054] The second calculation unit 123 calculates, for the first temporal encoder MDL2-1, the first feature amount (i.e., the first latent variable vector) at time t and the action a of the robot 10 at time t-1. t-1 The first temporal encoder MDL2-1 receives the first latent variable vector at time t and the action a of the robot 10 at time t-1. t-1 When the input is made, the input first latent variable vector and action a t-1 and the second feature value at time t-1 stored in the LSTM, the first latent variable vector and the action a t-1 In other words, the first temporal encoder MDL2-1 outputs a second latent variable vector having a smaller number of dimensions than the first latent variable vector and the action a t-1 is embedded in the second latent space as a second latent variable vector.

[0055] The second calculation unit 123 further inputs the second latent variable vector output by the first temporal encoder MDL2-1 to the second temporal encoder MDL2-2. In response to the input of the second latent variable vector at time t, the second temporal encoder MDL2-2 outputs a third latent variable vector having a smaller number of dimensions than the second latent variable vector. In other words, the second temporal encoder MDL2-2 embeds the second latent variable vector as a third latent variable vector in a third latent space. In particular, the third latent variable vector is also referred to as a query q.

[0056] The second calculation unit 123 calculates the third latent variable vector at time t output by the second temporal encoder MDL2-2 (query q at time t t ) is calculated as the second feature amount at time t.

[0057] Next, the behavior decision unit 124 calculates the second feature (i.e., the query q t ) based on the robot's action a at time t. t is determined (step S106).

[0058] For example, the behavior decision unit 124 determines the behavior a of the robot at time t using the deep reinforcement learning model MDL4. t may be determined.

[0059] The deep reinforcement learning model MDL4 may be, for example, an Actor-Critic that combines a value and a policy. Examples of the Actor-Critic include Twin Delayed DDPG (TD3), Soft Actor-Critic (SAC), and Proximal Policy Optimization (PPO). In this embodiment, the deep reinforcement learning model MDL4 will be described as TD3, which is one of the Actor-Critics, as an example.

[0060] For example, the behavior decision unit 124 determines the environmental state s detected by various sensors at time t. tObservation data showing the detection results of t Instead of directly inputting the data into the deep reinforcement learning model MDL4, we use the query encoder MDL1 and the temporal encoder MDL2 to generate the observed data O t is compressed to a low dimension and input to the deep reinforcement learning model MDL4. In response to this, the deep reinforcement learning model MDL4 calculates the environmental state s at time t. t One or more actions (behavior variables) a that the robot 10 can take under t Among them, the action (action variable) a with the highest value (Q value) is t The value is the reward r t is added. Reward r t is the environmental state s at time t-1 before time t. t-1 and Action a ′-1 The reward function R(s t-1 ,a t-1 ) is calculated by the behavior (behavior variable) a t The action may be various actions such as grasping the object OB with the end effector 11, changing the grip, or moving the object OB.

[0061] Next, the command generation unit 125 generates the action a determined by the action determination unit 124. t Based on the above, a command for controlling each actuator 16 of the robot 10 is generated, and the communication control unit 126 transmits the command to the robot 10 (step S108).

[0062] When the drive control unit 17 of the robot 10 receives a command from the control device 100, it controls the actuator 16 based on the command. This causes the robot 10 to act, and the object OB is lifted or moved, thereby changing the environmental state s surrounding the robot 10. t ga s t+1 will be updated to.

[0063] Next, the acquisition unit 121 acquires the environmental state s t+1 Observation data O at time t+1 from robot 10 placed below t+1is acquired (step S110).

[0064] Next, the acquisition unit 121 acquires the observation data O t , action a at time t t , the actor-critic reward function R(s t ,a t ), and the observation data O at time t+1 t+1 Combination of (O t , a t , R(s t ,a t ), O t+1 ) is stored in the storage unit 130 as a time series data set D (step S112). The time series data set D may include, for example, t-1 , a t-1 , R(s t-1 ,a t-1 ), O t ) includes past information.

[0065] Next, the learning unit 127 uses the observation data O at the time t described above. t and the observation data O at time t-1 t-1 Two types of observation data with different times are extracted from the time series data set D containing the above (step S114).

[0066] For example, the learning unit 127 may select observation data O at reference time t+j from the time series data set D. t+j and the observation data O at time t′ before the reference time t+j t′ For example, the learning unit 127 may extract the observed data O t′ As the observation data O t+j The observation data one step before O t+j-1 may be extracted, or the observation data O t+j Observation data 10 steps before t+j-10 The reference time t+j is an example of a "first reference time," and the time t' that is earlier than the reference time t+j is an example of a "second reference time."

[0067] Next, the learning unit 127 uses the key encoder MDL3 to extract the two pieces of observation data O t+j and O t′ From these, a fourth latent variable (hereinafter also referred to as key k) is calculated (step S116).

[0068] The key encoder MDL3 is a machine learning model that is trained to compress the dimensionality of input data, similar to the query encoder MDL1, and may be implemented by, for example, an MLP or a CNN. The key encoder MDL3 is an example of a "third encoder."

[0069] First, the learning unit 127 uses two observation data O t+j and O t′ One of the observation data O t+j is input to the key encoder MDL3. At this time, the learning unit 127 inputs the observation data O t+j In addition, the reward r at reference time t+j t+j The reward r at reference time t+j is input. t+j is the environmental state s at time t+j-1 before the reference time t+j. t+j-1 and Action a t+j-1 The reward function R(s t+j-1 ,a t+j-1 ) is calculated as follows.

[0070] The key encoder MDL3 is t+j and reward r t+j In response to the input of the fourth latent variable vector (i.e., key k), the key encoder MDL3 outputs the fourth latent variable vector (i.e., key k). In other words, the key encoder MDL3 outputs the fourth latent variable vector (i.e., key k) in response to the input of the fourth latent variable vector (i.e., key k). t+j and reward r t+j is embedded in the fourth latent space as a fourth latent variable vector.

[0071] Furthermore, the learning unit 127 uses two observation data O t+j and O t′ The other observation data O t′ is input to the key encoder MDL3. At this time, the learning unit 127 inputs the observation data O t′In addition, the reward r at reference time t′ t′ The reward r at reference time t' is input. t′ is the environmental state s at time t′-1 before the reference time t′. t′-1 and Action a t′-1 The reward function R(s t′-1 ,a t′-1 ) is calculated as follows.

[0072] The key encoder MDL3 is t′ and reward r t′ In response to the input of the fourth latent variable vector (i.e., key k), the key encoder MDL3 outputs the fourth latent variable vector (i.e., key k). In other words, the key encoder MDL3 outputs the fourth latent variable vector (i.e., key k) in response to the input of the fourth latent variable vector (i.e., key k). t′ and reward r t′ is embedded in the fourth latent space as a fourth latent variable vector.

[0073] Next, the learning unit 127 uses contrastive learning to learn the query encoder MDL1, the temporal encoder MDL2, and the key encoder MDL3 (step S118).

[0074] For example, the learning unit 127 uses two observation data O t+j and O t′ One of the observation data O t+j The fourth latent variable vector derived from the first latent variable vector of the positive example is used as the first target latent variable vector of the other observed data O t′ The resulting fourth latent variable vector is the second target latent variable vector of the negative example. The first target latent variable vector is an example of a "first target feature," and the second target latent variable vector is an example of a "second target feature."

[0075] Then, the learning unit 127 learns the query encoder MDL1, the temporal encoder MDL2, and the key encoder MDL3 so that the query q (i.e., the third latent variable vector) output by the second temporal encoder MDL2-2 approaches the first target latent variable vector of a positive example, and the query q moves away from the second target latent variable vector of a negative example.

[0076] For example, the learning unit 127 calculates a contrastive loss L TCL Calculate the loss L TCL By backpropagating the error based on the loss L, the parameters of each encoder (weighting coefficients and bias components) are adjusted. TCL can be expressed by, for example, equations (1)-(3).

[0077]

number

[0078]

number

[0079]

number

[0080] ψ represents parameters such as weighting coefficients and bias components, and ψ′ is the exponential moving average of the parameters ψ. ψ represents the query encoder MDL1, and g ψ represents the key encoder MDL3, and h ψ represents the temporal encoder MDL2. α is an arbitrary coefficient, e.g., 1. d q is the number of dimensions of the query q, and d k is the dimension number of the key k. As shown in Equation (1), the symmetric loss L TCL is calculated as the sum of two types of regularization loss.

[0081] Next, the learning unit 127 uses deep reinforcement learning to learn a query encoder MDL1, a temporal encoder MDL2, and a deep reinforcement learning model MDL4 (step S120).

[0082] For example, when the deep reinforcement learning model MDL4 is Actor-Critic, the learning unit 127 calculates the actor loss Lactor And Critic's Loss L critic Then, by performing backpropagation based on these two losses, the parameters of the query encoder MDL1, the temporal encoder MDL2, and the deep reinforcement learning model MDL4 are adjusted. This completes the processing of this flowchart.

[0083] Figure 5 shows a schematic diagram of how each encoder is trained using two approaches: contrastive training and deep reinforcement learning.

[0084] As shown in the figure, first, the control loss L TCL The parameters of the query encoder MDL1, the temporal encoder MDL2, and the key encoder MDL3 are adjusted based on the gradients of actor The actor parameters are adjusted based on the gradient, and the critic loss L critic The parameters of the Critic are adjusted by the gradient based on the critic The gradient based on is also propagated to the query encoder MDL1 and the temporal encoder MDL2. As a result, the query encoder MDL1 and the temporal encoder MDL2 TCL Once trained with the gradient based on critic It is further trained with gradients based on

[0085] [Comparison results] 6 to 9 are diagrams showing examples of tasks. FIG. 6 shows a task of manipulating an object OB like a pendulum (hereinafter referred to as "Pendulum"), and its observation data O t is expressed in one dimension, and the behavior a t is also expressed in one dimension. Figure 7 shows a task to land an object OB on the moon (hereafter referred to as Lunar Lander), and its observation data O t is expressed in five dimensions, and the behavior a tis expressed in two dimensions. Figure 8 shows a task of manipulating a pen-like object OB (hereafter referred to as Pen (touch)), and its observation data O t is expressed in 51 dimensions, and the action a t is expressed in 24 dimensions. Figure 9 shows a task similar to quadruped walking (hereafter referred to as Quadruped walk), and its observed data O t is represented in 64x64x3 dimensions, and the action a t is expressed in 12 dimensions.

[0086] 10 to 13 are diagrams showing the results of comparing the learning method of this embodiment with other methods for each task. The horizontal axis represents the number of steps in which the environment s is updated, i.e., the number of times learning is repeated. The vertical axis represents the reward r.

[0087] The learning method of this embodiment is represented as TD3 (Twin Delayed DDPG) + TCL (Temporal Contrastive Learning). Other learning methods include, for example, TD3(fc) using a fully connected layer in TD3, TD3+LSTM using LSTM in TD3, SAC(fc) using a fully connected layer in SAC (Soft Actor-Critic), SAC+LSTM using LSTM in SAC, PPO(fc) using a fully connected layer in PPO (Proximal Policy Optimization), and PPO+LSTM using LSTM in PPO.

[0088] 10, when the task is Pendulum, TD3+TCL, which is the learning method of this embodiment, shows performance comparable to conventional TD3(fc), SAC(fc), SAC+LSTM, etc. Specifically, the reward r converges to a high value in a small number of steps, which means that the task is realized early.

[0089] As shown in FIG. 11, even when the task is Lunar Lander, which is more complex than Pendulum, the TD3+TCL learning method of this embodiment shows performance comparable to that of conventional TD3(fc), TD3+LSTM, and SAC+LSTM.

[0090] As shown in Figures 12 and 13, when the task is Pen (touch), which is more complicated than Lunar Lander, or Quadruped walk, which is even more complicated than Lunar Lander, TD3+TCL, which is the learning method of this embodiment, shows significantly higher performance than conventional learning methods. Specifically, among multiple learning methods, it converges to a high value of reward r in the fewest number of steps, which means that the task is realized earlier than conventional learning methods. As such, TD3+TCL, which is the learning method of this embodiment, introduces contrastive learning to train each encoder in the latent space, which makes the task more complex and the observed data O t and action a t Even if the dimension becomes high, the robot 10 can be made to quickly execute the task.

[0091] According to the embodiment described above, the processing unit 120 of the control device 100 calculates the environmental state s at time t of the robot 10 that can manipulate the object OB. t Observation data showing the observation results of t The processing unit 120 obtains the observation data O at time t using the query encoder MDL1. t From the observation data t The processing unit 120 calculates a first latent variable vector having fewer dimensions than the first feature at time t. ... t-1 From the second feature at time t-1 and the first feature at time t, the query q at time t is calculated. t is calculated as the second feature amount at time t.

[0092] The processing unit 120 uses the deep reinforcement learning model MDL4 to calculate the robot's action a at time t. t From the above, the action a of the robot 10 at time t t Then, the processing unit 120 uses contrastive learning to train at least the query encoder MDL1 and the temporal encoder MDL2 (and actually also the key encoder MDL3).

[0093] In this way, since each encoder is trained using contrastive learning, even if the information input to each encoder is only partial observation (sparse), the robot 10 can be made to execute the target task robustly and with high accuracy using the encoder. In other words, the target task becomes easier to achieve. In particular, since each encoder is trained by contrastive learning and then trained again by deep reinforcement learning, as mentioned in the comparison with conventional learning methods, the task becomes complicated and the observation data O t and action a t Even if the dimension becomes high, the robot 10 can be made to quickly execute the task.

[0094] Furthermore, according to the above-described embodiment, the position of the object OB can be detected by using the observation data O obtained from each sensor of the end effector 11 without using the visual sensor 12 or the motion capture sensor. t Robot 10 behaviors a t can be determined.

[0095] Furthermore, according to the above-described embodiment, if an object OB having a certain shape is successfully operated, other objects having a similar shape can be operated thereafter. For example, if a rod-shaped screwdriver is successfully operated, a similar rod-shaped pencil can also be operated using the same machine learning model (query encoder MDL1, temporal encoder MDL2, key encoder MDL3, and deep reinforcement learning model MDL4).

[0096] <Modifications of the embodiment> In the above-described embodiment, the control device 100 is described as training the query encoder MDL1, the temporal encoder MDL2, the key encoder MDL3, and the deep reinforcement learning model MDL4, but this is not limiting. For example, these models may be trained offline by a server on the network NW.

[0097] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Obtaining observation data indicating an observation result of a state of a robot capable of manipulating an object at a first time; calculating a first feature amount at the first time from the observation data at the first time using a first encoder; calculating the second feature amount at the first time from the behavior of the robot at the second time, the second feature amount at the second time, and the first feature amount at the first time, using a recursive second encoder that stores a second feature amount at a second time that is earlier than the first time; determining an action of the robot at the first time based on the second feature amount at the first time; training the first encoder and the second encoder using contrastive training; Operation learning system.

[0098] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0099] 1...operation method learning system, 10...robot, 11...end effector, 12...visual sensor, 13...tactile sensor, 14...force sensor, 15...posture sensor, 16...actuator, 17...drive control unit, 100...control device, 110...communication interface, 120...processing unit, 121...acquisition unit, 122...first calculation unit, 123...second calculation unit, 124...action decision unit, 125...command generation unit, 126...communication control unit, 127...learning unit, 130...memory unit, NW...network, MDL1...query encoder, MDL2...temporal encoder, MDL3...key encoder, MDL4...deep reinforcement learning model

Claims

1. an acquisition unit that acquires observation data indicating an observation result of a state of a robot capable of manipulating an object at a first time; a first calculation unit that calculates a first feature amount at the first time from the observation data at the first time using a first encoder; a second calculation unit that calculates the second feature amount at the first time from a behavior of the robot at the second time, the second feature amount at the second time, and the first feature amount at the first time, using a recursive second encoder that stores a second feature amount at a second time that is earlier than the first time; a decision unit that decides an action of the robot at the first time point based on the second feature amount at the first time point; a learning unit configured to learn at least the first encoder and the second encoder using contrastive learning; An operation method learning system comprising:

2. the learning unit learns at least the first encoder and the second encoder as the contrastive learning so that the second feature calculated using the second encoder approaches a first target feature and the second feature moves away from the second target feature. The operation method learning system according to claim 1 .

3. The learning unit extracting the observation data at a first reference time and the observation data at a second reference time that is earlier than the first reference time from the time series observation data including the observation data at the first time and the second time; inputting at least the observation data at the first reference time to a third encoder; a latent variable output by the third encoder in response to at least the observation data at the first reference time being input is defined as the first target feature; inputting at least the observation data at the second reference time to the third encoder; the latent variable output by the third encoder in response to at least the observation data at the second reference time being input is defined as the second target feature. The operation method learning system according to claim 2 .

4. the determination unit determines an action of the robot at the first time point using reinforcement learning; the learning unit learns the first encoder and the second encoder using a reward of the reinforcement learning. The operation method learning system according to claim 3 .

5. The learning unit inputting the reward at the first reference time into the third encoder in addition to the observation data at the first reference time; the latent variable output by the third encoder in response to input of the observation data and the reward at the first reference time is set as the first target feature; inputting the reward at the second reference time into the third encoder in addition to the observation data at the second reference time; the latent variable output by the third encoder in response to input of the observation data and the reward at the second reference time is set as the second target feature. The operation method learning system according to claim 4 .

6. The first calculation unit inputting the observation data at the first time to the first encoder; calculating a latent variable output by the first encoder in response to the input of the observation data at the first time as the first feature at the first time; 3. The operation method learning system according to claim 1 or 2.

7. The second calculation unit inputting the first feature amount at the first time to the second encoder; calculating a latent variable output by the second encoder in response to input of the first feature amount at the first time as the second feature amount at the first time; 3. The operation method learning system according to claim 1 or 2.

8. acquiring observation data indicating an observation result of a state of a robot capable of manipulating an object at a first time; calculating a first feature amount at the first time from the observation data at the first time using a first encoder; calculating the second feature amount at the first time from the behavior of the robot at the second time, the second feature amount at the second time, and the first feature amount at the first time, using a recursive second encoder that stores the second feature amount at a second time before the first time; determining an action of the robot at the first time point based on the second feature amount at the first time point; training the first encoder and the second encoder using contrastive learning; Includes learning methods for operation.

9. A program to be executed by a computer, acquiring observation data indicating an observation result of a state of a robot capable of manipulating an object at a first time; calculating a first feature amount at the first time from the observation data at the first time using a first encoder; calculating the second feature amount at the first time from the behavior of the robot at the second time, the second feature amount at the second time, and the first feature amount at the first time, using a recursive second encoder that stores the second feature amount at a second time before the first time; determining an action of the robot at the first time point based on the second feature amount at the first time point; training the first encoder and the second encoder using contrastive learning; Programs including.

Images