Motion retargeting device for motion generation of robot, machine learning method for neural network, generation method for motion data of robot, and control method for robot

Abstract

The present invention provides a new control technique for a robot without using inverse kinematics. A disclosed motion retargeting device for motion generation of a robot is configured to provide first motion data of a person to a neural network that has undergone machine learning and generate second motion data for operating the robot by using the neural network. The neural network comprises: an encoder that encodes the first motion data into latent motion data on a latent space represented by a primitive skeleton that is a common skeleton structure between the person and the robot; and a decoder that generates the second motion data of the robot on the basis of the latent motion data and the skeleton information of the robot.

Description

Motion retargeting device for generating robot motion, neural network machine learning method, robot motion data generation method, and robot control method

[0001] The present disclosure relates to a motion retargeting device for generating robot movements, a machine learning method for neural networks, a method for generating robot motion data, and a method for controlling a robot.

[0002] Teleoperation technology, in which humans remotely operate robots in distant locations, is becoming increasingly common. When teleoperating robots such as humanoids, it is important that the operator's movements are reproduced in the same way even for robots with different skeletal and joint configurations.

[0003] Conventionally, to teleoperate a robot, the correspondence between the operator and the robot's main body positions, such as the wrists and face, is established in advance. Then, the robot's joint angles are calculated by numerical calculation (inverse kinematics) to follow the operator's positions, and the robot is operated accordingly.

[0004] International Publication No. 2019 / 059364

[0005] K. Aberman, P. Li, D. Lischinski, O. Sorkine-Hornung, D. Cohen-Or, and B. Chen, “Skeleton-aware networks for deep motion retargeting,” ACM Transactions on Graphics(TOG), vol.39, no.4, pp.62-1, 2020.

[0006] In general, teleoperation based on inverse kinematics has the following two problems: (1) The correspondence between body positions must be set each time the operator changes. (2) Depending on the operator's movements, the numerical calculations of inverse kinematics can become unstable, causing the robot to run out of control. Therefore, it is necessary to avoid operations that destabilize the robot's movements.

[0007] Therefore, a new robot control method that does not use inverse kinematics is desired.

[0008] One aspect of the present disclosure is a motion retargeting device for generating robot motion. The disclosed motion retargeting device is configured to provide first motion data of a human to a machine-learned neural network and generate second motion data for operating a robot using the neural network, the neural network including an encoder that encodes the first motion data into latent motion data in a latent space represented by a protoskeleton that is a skeletal structure common to the human and the robot, and a decoder that generates the second motion data of the robot based on the latent motion data and skeletal information of the robot.

[0009] Another aspect of the present disclosure is a machine learning method for a neural network. In the disclosed machine learning method, the neural network includes a first encoder that encodes human motion data into latent motion data in a latent space represented by a primitive skeleton that is a skeletal structure common to the human and the robot, a first decoder that generates motion data for the human based on the latent motion data, a second encoder that encodes robot motion data into latent motion data in the latent space represented by the primitive skeleton, and a second decoder that generates motion data for the robot based on the latent motion data. The machine learning method includes causing the neural network to perform machine learning using motion data based on actual movements of the human and motion data based on actual movements of the robot.

[0010] Yet another aspect of the present disclosure is a method for generating robot motion data based on human motion data. The disclosed generation method includes generating robot motion data using at least the first encoder and the second decoder in the machine-learned neural network, where generating the robot motion data using at least the first encoder and the second decoder includes providing first motion data of a human to the first encoder to obtain latent motion data, and generating the robot motion data using the second decoder from the latent motion data and a skeletal structure of the robot. Yet another aspect of the present disclosure is a method for controlling a robot, the disclosed robot control method including operating the robot using the robot motion data generated by the generation method.

[0011] Further details will be described in the following embodiments.

[0012] FIG. 1 is a configuration diagram of a system equipped with a motion retargeting device. FIG. 2 is a flowchart showing the procedure for controlling a robot. FIG. 3 is an explanatory diagram of motion data. FIG. 4 is an explanatory diagram of a human skeletal structure and a common skeletal structure. FIG. 5 is an explanatory diagram of neural network learning. FIG. 6 is an explanatory diagram of an encoder and a decoder. FIG. 7 is an explanatory diagram of neural network learning. FIG. 8 is a flowchart showing the neural network learning procedure. FIG. 9 is an explanatory diagram of a robot's pseudo-structure. FIG. 10 is an explanatory diagram of motion retargeting when controlling a robot.

[0013] <1. Overview of the motion retargeting device for generating robot movements, the neural network machine learning method, the robot motion data generation method, and the robot control method>

[0014] (1) According to an embodiment, a motion retargeting device for generating robot motion may be configured to provide first motion data of a person to a machine-learned neural network and generate second motion data for moving the robot using the neural network. The neural network may include an encoder that encodes the first motion data into latent motion data in a latent space represented by a primitive skeleton, which is a skeletal structure common to the person and the robot, and a decoder that generates the second motion data of the robot based on the latent motion data and skeletal information of the robot. In the latent space, a deep motion representation corresponding to the primitive skeleton is obtained. The neural network encodes the person's motion (first motion data) into latent motion data representing the deep motion and decodes the latent motion data into robot motion (second motion data).

[0015] (2) The skeletal information of the robot is preferably data representing the robot's skeleton using a pseudo-structure that approximates the actual structure of the robot to the human body structure.

[0016] (3) It is preferable that the neural network is machine-learned using data on the robot's movements that satisfy the structural constraints of the robot, and is configured to generate the second motion data that satisfy the structural constraints of the robot.

[0017] (4) It is preferable that the neural network is machine-trained using a loss function that penalizes the generation of data that causes the robot to perform actions that are impossible due to structural constraints of the robot.

[0018] (5) In a machine learning method for a neural network according to an embodiment, the neural network may include: a first encoder that encodes human motion data into latent motion data in a latent space represented by a primitive skeleton that is a skeletal structure common to the human and the robot; a first decoder that generates human motion data based on the latent motion data; a second encoder that encodes robot motion data into latent motion data in the latent space represented by the primitive skeleton; and a second decoder that generates robot motion data based on the latent motion data. The machine learning method may include training the neural network using motion data based on actual movements of the human and motion data based on actual movements of the robot. In typical machine learning (supervised learning), pairs of human motion data and robot motion data must be associated. However, in machine learning using a generative adversarial network (GAN) described below, pairs of data do not need to be associated, enabling unsupervised learning.

[0019] (6) It may include using a loss function that penalizes the generation of data that causes the robot to perform an action that is impossible due to structural constraints of the robot.

[0020] (7) The motion data based on the actual movement of the robot may include skeletal information of the robot. The skeletal information of the robot may be data representing the skeletal structure of the robot using a pseudo-structure that approximates the actual structure of the robot to the structure of the human body.

[0021] (8) A method for generating robot motion data according to an embodiment may include generating motion data of the robot using at least the first encoder and the second decoder in the machine-learned neural network. Generating the robot motion data using at least the first encoder and the second decoder may include providing first motion data of a person to the first encoder to obtain the latent motion data, and generating the robot motion data using the second decoder from the latent motion data and a skeletal structure of the robot. A method for controlling a robot according to an embodiment may include operating a robot using the robot motion data generated by the generation method.

[0022] 2. Examples of a motion retargeting device for generating robot motion, a neural network machine learning method, a method for generating robot motion data, and a method for controlling a robot

[0023] Hereinafter, the embodiment will be described in more detail with reference to the drawings.

[0024] 1 shows a system 10 including a motion retargeting device 20 according to an embodiment. In addition to the motion retargeting device 20, the system 10 also includes an operation device 30 and a robot 40. The system 10 is used by a human operator 100 to operate the robot 40 using the operation device 30. The robot 40 is controlled via the motion retargeting device 20 in accordance with the operation of the operator 100 using the operation device 30.

[0025] The motion retargeting device 20, the operation device 30, and the robot 40 are connected to one another via a network 15 such as a wireless or wired LAN or the Internet. The operator 100 can, for example, use the operation device 30 to remotely control the robot 40 connected via the network 15 online. Note that the movements of the human operator 100 and the movements of the robot 40 do not have to be performed simultaneously, and offline operation is also possible. For example, the human movements may be acquired in advance using the operation device 30 or the like to prepare data on the human movements, and the robot may be operated later based on the prepared data on the human movements.

[0026] 2 , the operating device 30 generates and outputs motion data representing the motion of the operator 100 (step S21). The motion data representing the motion of the human operator 100 may be referred to as human motion data, operator motion data, human motion data, or first motion data. The operator motion data is provided to the motion retargeting device 20.

[0027] The motion retargeting device 20 converts (motion retargets) the operator motion data into motion data representing the movement of the robot 40 (step S22). The motion data representing the movement of the robot 40 may be called robot control data, robot motion data, or second motion data.

[0028] The motion retargeting device 20 provides robot motion data, which is robot control data, to the robot 40. The robot 40 operates in accordance with the robot motion data. Because the robot motion data represents operations corresponding to the operator motion data, the robot 40 operates in accordance with the operations of the operator 100.

[0029] In the embodiment, the correspondence between the body parts (joints, etc.) of the operator 100 and the body parts (joints, etc.) of the robot does not need to be set in advance. The motion retargeting device 20 can convert (retarget) the motion of the robot 40 even if such a correspondence is not set in advance. This conversion is performed by a neural network that has been trained in advance (machine learning).

[0030] When using the motion retargeting device 20 according to the embodiment, there is no need to reset the body position correspondence for the operator 100 every time the operator 100 of the robot 40 changes. Therefore, various operators 100 can easily operate the robot 40 without having to set the correspondence in advance. Therefore, various people can operate the robot 40 with their own movements. Furthermore, because the movements of the robot 40 are output by a neural network, there is no instability in the movements of the robot 40 due to the numerical calculation of inverse kinematics.

[0031] The robot 40 is, for example, a humanoid (humanoid robot) or a dual-arm manipulator. The robot 40 does not need to imitate the entire human body, but may imitate only a part of the human body (for example, an arm). Furthermore, the robot 40 does not need to imitate a human, but may be one having one or more joints that are operated by the operator 100. In other words, the robot 40 is sufficient as long as it is a joint structure having one or more joints. The number of joints that the robot 40 has may be different from the number of joints that a human has. Furthermore, the length between each joint in the robot 40 may be different from the length between each joint in a human.

[0032] 1 , the robot 40 may include, for example, a robot controller 50 and a robot main body 60. The controller 50 may be provided outside the robot main body 60 or may be built into the robot main body 60. As an example, the controller 50 is a personal computer (control PC) that is provided outside the robot main body 60 and connected to the robot main body 60 wirelessly or via a wire.

[0033] The robot main body 60 has, for example, a humanoid body. The robot main body 60 is an articulated structure having one or more joints driven by a drive unit 61. The drive unit 61 is, for example, configured with a motor that rotates the joint. The drive unit 61 is not limited to a motor, and may be other actuators that can drive the joints, such as a hydraulic cylinder. Furthermore, the movement mechanism of the robot main body 60 does not need to be a two-legged mechanism like a human, but may also be a wheel-driven movement mechanism. An example of the robot main body 60 is "Torobo" by Tokyo Robotics Co., Ltd. "Torobo" has 22 joints.

[0034] The controller 50 generates drive signals (e.g., motor control commands) for driving each drive unit 61 from the robot motion data, which is robot control data provided by the motion retargeting device 20 (see step S23 in FIG. 2). The drive signals are provided to each drive unit 61. Upon receiving the drive signals, each drive unit 61 operates in accordance with the drive signals. This causes the robot main body 60 to operate in accordance with the robot motion data (see step S24 in FIG. 2).

[0035] As shown in FIG. 1 , the operation device 30 includes, for example, a motion capture device 30A and a processing device 30B. The motion capture device 30A captures the motion of the operator 100. The motion capture device 30A may be configured to be worn by the operator 100 and detect the motion of the operator 100, or may be configured to detect the motion of the operator 100 from an image of the operator 100 captured by a camera. One example of the motion capture device 30A is the mobile motion capture device "mocopi" by Sony Corporation. "mocopi" includes six sensors attached to the body of the operator 100 (head, both wrists, waist, and both ankles), and these sensors detect the entire body motion of the operator 100.

[0036] The processing device 30B acquires the detection signal from the motion capture device 30A and generates motion data of the operator 100 from the detection signal (see step S21 in FIG. 2 ). The processing device 30B is configured by a computer such as a smartphone, tablet, or personal computer. As an example, the processing device 30B is a smartphone paired with "moccopi." An application program (smartphone app) dedicated to "moccopi" is installed on the smartphone that is the processing device 30B. The application program acquires, via wireless communication, the detection signal from the sensor (moccopi) worn by the operator 100 and generates motion data (motion capture data) of the operator 100.

[0037] For calibration purposes, the mocopi application program estimates the posture (static posture; basic posture) of the operator 100. The operator 100 sets his or her own height using the mocopi application program. Furthermore, when the operator 100 wearing the mocopi sensor performs a predetermined movement for calibration, the mocopi application program acquires the length of each part of the body of the multi-jointed operator 100 (for example, a part (link) from one joint to another), as well as the quaternion (quaternion) of the joint angle. The quaternion (quaternion) represents rotation (posture) in three-dimensional space.

[0038] 2, after calibration, when the operator 100 wearing the sensor moves, the smartphone (application program) serving as the processing device 30B generates motion data of the operator 100 based on the detection signal from the sensor (step S21 in FIG. 2). The motion data is, for example, a BVH file (Bio Vision Hierarchy file). The BVH file is a format for motion capturing and the like.

[0039] As shown in FIG. 3, motion data such as a BVH file may include skeletal information (static posture) and movement information (posture at each time; amount of rotation of each joint at each time).

[0040] The skeletal information represents an initial posture (static posture; basic posture). In a BVH file, the skeletal information represents a skeletal structure having a plurality of joints in a hierarchical structure.

[0041] The movement information indicates the movement in each time frame of the movement of the operator 100. In the BVH file, the movement information is represented by the posture displacement from the initial posture (the dynamic rotation amount of each joint in three-dimensional space) in each time frame of the movement.

[0042] As shown in Fig. 3, in this embodiment, the motion data of the robot 40 is represented by data in the same format (e.g., a BVH file) as the motion data of the operator 100. Fig. 3(A) shows the operator motion data, and Fig. 3(B) shows the robot motion data. Like the operator motion data shown in Fig. 3(A), the robot motion data shown in Fig. 3(B) may also include skeletal information (static posture) and operation information (posture at each time; amount of rotation of each joint at each time).

[0043] The skeletal structures of the operator 100 (person) and the robot 40, as shown in FIG. 3, can be represented by a tree-structured graph. A graph is a data structure consisting of nodes (vertices) and edges connecting the nodes. The nodes represent joints, etc. FIG. 4A shows a graph representation of a human skeletal structure. When a human skeletal structure is represented by a tree-structured graph, the root of the tree structure is located, for example, at the waist joint, and a chain of nodes, which are joints, points toward end effectors such as the hands and head. The graph structure of the robot 40 is similar. Each node has information such as the relative distance from the parent joint (e.g., the distance in a Cartesian coordinate system), the rotation angle (e.g., Euler angle), and the name of the child joint. The graph structure containing this information indicates a specific initial posture (e.g., a T-pose) of the operator 100 (person) or the robot 40. The T-pose is a posture with the legs spread and arms stretched out on both sides, as shown in FIG. 3.

[0044] The skeletal structure of the human operator 100 and the skeletal structure of the humanoid robot 40 may be topologically equivalent graph structures. For example, the operator 100 and the robot 40 may have common parts such as both hands and a head. The common parts of the operator 100 and the robot 40 may be represented by a common skeletal structure. The common skeletal structure may be, for example, a stick figure-like skeletal structure. FIG. 4B shows an example of a common skeletal structure between a human and a robot. The upper and lower skeletal structures on the far left of FIG. 4B (the human skeletal structure and the robot skeletal structure) are different from each other. However, by deleting (pooling) nodes with degree 2 in the graph structure representing these skeletal structures, a common minimal graph (primitive skeleton) can be obtained. The primitive skeleton shown in FIG. 4B is an example; the primitive skeleton does not need to be a minimal graph, as long as it is a skeletal structure common to both the human and the robot.

[0045] By adopting a common skeletal structure such as a primitive skeleton, it is possible to obtain a common deep representation from skeletal structures with different numbers of joints and different lengths of each bone (body segment). In this embodiment, a deep motion representation (latent motion data; feature data) for the common skeletal structure is obtained from the motion data of the operator 100 using a neural network, and motion data for the robot 40 is generated from the deep motion representation (latent motion data; feature data).

[0046] Generally, since the skeletal structure of each person is different, when the operator 100 of the robot 40 changes, it is necessary to reset the correspondence between the parts of the robot 40 and the parts of the operator 100. However, in this embodiment, the movements of the operator 100 are converted into a deep motion representation (latent motion data; feature data) for a common skeletal structure, so that the robot 40 can be operated even if the skeletal structure of the operator 100 is different. Therefore, there is no need to set in advance the correspondence between the parts of the robot 40 and the parts of the operator 100.

[0047] In this embodiment, as an example, the skeletal structures of the human operator 100 and the robot 40 are represented as graph structures, and a deep motion representation in the common skeletal structure is obtained by a graph neural network. In this embodiment, a graph convolutional network can be used as an example of a graph neural network. The graph convolutional network is a network that convolves a graph structure. As shown in FIG. 4 , the graph convolutional network obtains a common graph structure (primitive skeleton) by deleting (pooling) nodes in a graph that represents the skeletal structure.

[0048] In this embodiment, as an example, the neural network may include two pairs of generative adversarial networks (GANs). Each of the two pairs of generative adversarial networks handles the motion data of the operator 100 and the motion data of the robot 40. As shown in Fig. 3, the motion data such as a BVH file includes skeletal information indicating an initial posture and operation information indicating the displacement of each joint from the initial posture.

[0049] The neural network of this embodiment is based on Cycle GAN. Cycle GAN is a method for converting the style of data using two generators. As shown in FIG. 5 , the neural network may include two generators 210 and 220. The two generators 210 and 220 include a first generator 210 and a second generator 220. The first generator 210 retargets the motion of the operator 100 to the motion of the robot 40. The second generator 220 retargets the motion of the robot 40 to the motion of the operator 100.

[0050] The first generator 210 is designated by "G ｈ→ｒ ” and “G” indicating the second generator 220 ｒ→ｈ " etc., h indicates a person (operator 100) and r indicates a robot 40.

[0051] 5, the first classifier 230 is a classifier corresponding to the first generator 210, and the second classifier 240 is a classifier corresponding to the second generator 220. The classifiers 230 and 240 are used in a dynamic encoder E (described later). Ｑ The neural network has a similar structure to the neural network 210 and 220, except that the classifiers 230 and 240 do not take skeletal information into account. Similar to a typical generative adversarial network, this neural network is trained (machine learning) to distinguish between real training data (training motion data representing actual movements) and motion data generated by the generators 210 and 220. The training (machine learning) of this neural network is performed using, for example, the loss function L shown in FIG. 5 (see equations (1) to (6) in FIG. 5). The loss function L will be described later.

[0052] As shown in FIG. 6, the generators each have two encoders E Ｓ , E Ｑ and a decoder D. Ｓ , E Ｑ is the static encoder E Ｓ and dynamic encoder E Ｑ Static encoder E Ｓ handles skeleton information, and the dynamic encoder E Ｑ handles motion information. Static encoder E Ｓand dynamic encoder E Ｑ The two encoders E operate in parallel, and can process skeleton information and motion information in parallel. Ｓ , E Ｑ The decoder D is preferably multi-stage (for example, three or more stages) and large in scale. Ｓ , E Ｑ In decoder D, the number of convolution operations required to obtain the original skeleton or to transform it into the retarget skeleton is performed.

[0053] Dividing the encoder into one for the skeleton (posture) and one for the movement makes it possible to accommodate a variety of operators 100 / robots 40. Generally, the movement of the operator 100 / robot 40 is characterized by the skeletal structure, but it is preferable to extract characteristics of movement that do not depend on the skeletal structure (movement in a common skeletal structure) in the latent space. In other words, it is preferable for the encoder to compress the movement into a movement that is independent of the specific skeleton (posture) of the person or robot. As described above, dividing the encoder into one for the skeleton (posture) and one for the movement is preferable because it allows a common latent movement representation (latent motion data in the latent space represented by a primitive skeleton) to be obtained even if the operator 100 changes.

[0054] Furthermore, by training the network so that the latent motion expressions of the operator 100 and the robot 40 become closer, the network learns the mapping of motion from the operator 100 to the robot 40, and can output the motion of the robot 40 using the motion of the operator 100 as input.

[0055] As shown in FIG. Ｓ is a linear transformation of the skeleton information, and the dynamic encoder E Ｑ processes motion and skeletal information in parallel. Ｓ Skeleton information (initial pose) is input to the static encoder E, and a deep skeletal representation (common skeletal feature data; deep pose representation) is obtained by the fully connected layer. ＳThis obtains a common skeletal structure (common skeletal feature data) in which the skeletal structures of the operator 100 / robot 40 are pooled. This makes it possible to obtain deep motion independent of the skeletal structure while preserving the features of the skeletal structure.

[0056] Since the motion data for a certain skeleton is composed of its skeleton structure (initial posture) and motion information (movement of joint angles), the static encoder E Ｓ encodes the skeleton information, and the encoded skeleton information is sent to the dynamic encoder E Ｑ Give to.

[0057] Dynamic Encoder E Ｑ performs convolution processing and pooling. Pooling, which is performed after the convolution processing, is performed to obtain a deep motion representation using an abstracted skeletal structure (see FIG. 4B). By pooling, motion feature data (latent motion data; common latent motion representation) for the common skeletal structure of the operator 100 / robot 40 is obtained.

[0058] The decoder D (dynamic decoder D) decodes the action feature data (latent motion data) and generates reconstructed motion data from the action feature data (latent motion data). The decoder D may also use skeleton information such as common skeleton feature data to generate the reconstructed motion data. The decoder D performs unpooling. Unpooling is the inverse process of pooling, and restores human / robot motion data from common skeleton information.

[0059] In Non-Patent Document 1, the movement of the root joint is also learned, but in the complex skeleton of a human-robot, there is no need to retarget the movement of the root joint (i.e., if the foot (posture) moves, the root joint moves), and therefore, in this embodiment, which controls a robot, there is no need to learn the movement of the root joint.

[0060] As shown in FIG. 7, the neural network generates a first encoder E for the motion data of the human operator 100. ｈ and the first decoder D ｈThe neural network also includes a second encoder E for the motion data of the robot 40. ｒ and a second decoder D ｒ The first encoder E ｈ and the second encoder E ｒ Each of them is a static encoder E shown in FIG. Ｓ and dynamic encoder E Ｑ Equipped with.

[0061] First encoder E ｈ is the operator motion data (first motion data) x ｈ The first feature data E ｈ (x ｈ ) The second encoder E ｒ is the robot motion data (second motion data) x ｒ The second feature data E ｒ (x ｒ ) where x ｈ indicates the operator motion data for learning, and x ｒ denotes the robot motion data for learning. Also, x^ denotes the decoder D ｈ , D ｒ It should be noted that "x^" is the same as the notation in the drawings (FIGS. 1, 7, and 10) where "^" is placed directly above x (the same applies hereinafter).

[0062] The neural network in Fig. 7 is trained using the motion data for training (operator motion data and robot motion data) with the loss function L exemplified in Fig. 5. The loss function L exemplified in Fig. 5 includes four types of loss functions L rec ,L ltc ,L adv ,L ee The loss function L rec is the loss function of the reconstruction, and the three functions L 1 rec ,L 2 rec ,L 3 recEach function included in the loss function L illustrated in FIG. 5 is multiplied by a coefficient of "1.0" as a hyperparameter, but the hyperparameter is not limited to 1.0 and can be set appropriately.

[0063] Function L 1 rec can be expressed by equation (1) in FIG. 5. Equation (1) is a function of the operator motion data x ｈ 1 shows the mean square error between the reconstructed data and the original data for all time series joint quaternions of .

[0064] Function L 2 rec can be expressed by equation (2) in FIG. 5. Equation (2) is a function of the operator motion data x ｈ The root mean square error (rms error) between the reconstructed data and the original data for all time-series joint positions f in the joint pose, where f represents the forward kinematics (FK) function for calculating joint positions from joint angles (joint quaternions).

[0065] Function L 3 rec can be expressed by equation (3) in FIG. 5. Equation (3) is used by the generator 210 (decoder D ｒ ), a penalty is imposed when a joint of the robot 40 moves in a direction that it cannot move in. While a human's joints can move relatively freely, the joints of the robot 40 can only move in a fixed direction (in most cases, one direction). Equation (3) increases the penalty when each joint of the robot 40 moves in a direction other than the movable direction (unmovable direction).

[0066] In equation (3), the function fr indicates the joint position calculated from the joint angle (rotation angle) using forward kinematics (FK) for the movable direction of the joint of the robot 40. Furthermore, equation (3) indicates the joint position calculated from the forward kinematics (FK) by setting the joint angle (rotation angle) to zero for the movable direction of the joint of the robot 40. As described above, since the function f indicates the joint position calculated from the joint angle (rotation angle) using forward kinematics (FK), the root mean square error of fr - f becomes large in the case of reconstructed data that moves the robot 40 in a direction in which the robot 40 cannot move. Therefore, equation (3) imposes a large penalty on reconstructed data that moves the robot 40 in a direction in which the robot 40 cannot move.

[0067] As a result, the neural network trained using the loss function of equation (3) is configured to generate robot motion data (second motion data) that does not include data that causes the robot 40 to move in a direction in which the joints of the robot 40 cannot move.

[0068] The loss function of Equation (3) imposes a penalty on movements in an impossible direction as an example of movements that are impossible due to structural constraints. However, the penalty is not limited to movements in an impossible direction. Movements that are impossible due to structural constraints may be, for example, movements that exceed the rotational angle range (movement range) of the robot's joints, joint movements at rotational speeds (movement speeds) that exceed the upper limit of the possible movement speed, movements in which multiple parts of the robot interfere with (contact with) each other, or combinations of these movements. The loss function of Equation (3) may impose a penalty on movements that are impossible due to these structural constraints. In this case, the neural network can generate robot motion data that does not include data that causes movements that are impossible due to the robot's structural constraints.

[0069] Loss function L ltc can be expressed by equation (4) in Figure 5. Equation (4) is for obtaining a common latent space between the human operator 100 and the robot 40. Equation (4) is used to encode the retargeted robot motion data into the second encoder Eｒ The encoded data (latent action representation) and the original operator motion data are encoded by the first encoder E ｈ It acts to reduce the difference between what is encoded by (latent action representation) and . In equation (4), || ||1 indicates the L1 norm.

[0070] Loss function L adv can be expressed by equation (5) in FIG. 5. In this embodiment, the classifier C ｒ (First classifier 230, see FIG. 5) is used. Classifier C ｒ is the actual robot motion x ｒ And fake motion x^ ｈ－＞ｒ The loss function L is evaluated. adv is the least squares loss function of LSGAN (Least Square Generative Adversarial Networks).

[0071] Loss function L ee can be expressed by equation (6) in Figure 5. Equation (6) quantifies the corresponding end effector velocity error between the operator 100 and the robot 40. Here, the function f ee As an example, calculate the positions of three end effectors. The positions of the end effectors are obtained from the calculation results of forward kinematics (FK) performed by the function f in equation (2). Δ represents the difference from the value one step before. In other words, Δf ee / Δt represents the velocity of the end effector calculated from the current and previous steps. ee By learning using this method, the speeds of the end effectors (both arms, head, etc.) of the operator 100 and the robot 40 correspond to each other, making it easier for the operator 100 to intuitively operate the robot 40.

[0072] 8 shows an example of a procedure for neural network machine learning. In neural network machine learning, operator motion data for learning is generated (step S81), and robot motion data for learning is generated (step S83), and both sets of data are used to perform neural network machine learning (step S84). The neural network machine learning is performed, for example, by loading BVH files of a person and a robot collected in advance into the motion retargeting device.

[0073] To generate operator motion data for learning, the movements of a plurality of people 100 are detected by a motion capture device 30A and generated as motion data by a processing device 30B.

[0074] To generate robot motion data for learning, a robot trainer moves each part of the robot 40 (Torobo) (the left and right arms, and the "torso to neck region"). One or more human robot trainers hold parts of the robot 40 and manually make the robot perform various movements. The movements of the robot 40 (joint rotation angles) caused by external human force are detected by sensors, and the resulting movement signals become the movement information that constitutes the robot motion data.

[0075] By generating the learning robot motion data by actually operating the robot 40, the learning robot motion data does not include data for movements that are impossible due to constraints such as the structure of the robot 40. In other words, the learning robot motion data is data for movements that satisfy the structural constraints of the robot 40. This prevents the neural network from learning movements that the robot 40 cannot perform. This, combined with learning using the loss function of the above-described equation (3), more reliably prevents the neural network from generating motions that the robot 40 cannot perform, and allows for the generation of motion data that satisfies the structural constraints of the robot 40.

[0076] In addition, skeletal information of the robot 40 is also required to generate the robot motion data for training (step S82). For example, the skeletal information of the robot 40 is generated so as to resemble as closely as possible the skeletal information of a human in a T-pose (see FIG. 3 ). According to experiments conducted by the inventors, the closer the T-pose between the human (operator 100) and the robot 40, the better the motion retargeting. Because the robot 40 has a structure different from that of a human, the initial posture (skeletal structure) of the robot 40 may differ significantly from that of a human. For example, if a skeletal structure is generated based on the actual structure of the robot 40 as shown in FIG. 9(A), a structure that does not exist in a human (e.g., an L-shaped structure between the arm and hand as shown in FIG. 9(B)) may result. The inventors discovered that if the skeletal information of the robot 40 includes such a structure that does not exist in a human, motion retargeting may not be performed appropriately.

[0077] Therefore, it is preferable that the skeletal information of the robot 40 is not based directly on the actual structure of the robot 40, but is data that represents the skeleton of the robot 40 using a pseudo structure that approximates the skeleton of a human, as shown in Fig. 9(C). For example, in Fig. 9(B), the actual structure of the robot 40 has an L-shaped structure between the arms and hands, but in the pseudo structure of Fig. 9(C), the arms and hands are linear, more like a human's. In this way, by defining the structure of the robot so that the structure and orientation of each part of the robot approximates the structure and orientation of each part of a human, good motion retargeting can be achieved.

[0078] The skeletal information of the robot 40 based on the pseudo structure can be used both as data for learning and as data during control. That is, the neural network learns using the skeletal information based on the pseudo structure of the robot 40, and generates motion data based on the pseudo structure of the robot 40 for controlling the robot 40. Note that, because the joints of the pseudo structure are common to the joints of the actual structure, even motion data based on the pseudo structure can be used to control (operate) the actual robot 40 without any problems.

[0079] The motion retargeting device 20 according to this embodiment uses a neural network that has been machine-learned as described above. Fig. 10 shows the control of the robot 40 using the machine-learned neural network. For motion retargeting, the control of the robot 40 uses at least the first encoder E ｈ and a second decoder D ｒ and the first encoder E of the neural network can be used. ｈ operator motion data x ｈ The operator motion data x ｈ is obtained by motion capturing the motion of the operator 100 operating the robot 40. This operator motion data x ｈ The correspondence between the robot 40 and the parts of the robot 40 does not need to be set in advance.

[0080] First encoder E ｈ is the operator motion data x ｈ The feature data (latent motion data) E of the motion in the common skeletal structure ｈ (x ｈ ) The second decoder D ｒ Based on the skeletal information of the robot, feature data (latent motion data) E ｈ (x ｈ ) into robot motion data. This results in the retargeted robot motion data x^ ｈ－＞ｒ The robot motion data x^ is obtained. ｈ－＞ｒ is converted into a drive signal for the robot 40, and the robot 40 operates based on the drive signal.

[0081] According to this embodiment, the operator 100 can intuitively operate the robot 40 online in real time using his / her gestures, which is preferable.

[0082] In addition, when the robot 40 to be controlled is determined in advance, the second decoder D ｒThe network may be provided with pre-encoded skeletal information of the robot 40 to be controlled. Note that the skeletal information of the robot 40 may be encoded at the time of control, as shown in FIG. 10 . Even if the robot 40 to be controlled changes, motion retargeting to that robot 40 is possible by encoding the skeletal information of the new robot 40. In addition, in the embodiment, in order to collect learning data, a person operates the robot 40, such as a humanoid, to generate motion data. This makes it possible to prevent the network from outputting motions that exceed the range of motion of the humanoid's joints during actual operation.

[0083] The present invention is not limited to the above-described embodiment, and various modifications are possible.

[0084] 10: System 15: Network 20: Motion retargeting device 30: Operation device 30A: Motion capture device 30B: Processing device 40: Robot 50: Controller 60: Robot body 61: Driving unit 100: Operator 210: First generator 220: Second generator 230: First classifier 240: Second classifier D: Decoder D ｈ : First decoder D ｒ : Second decoder E Ｑ : Dynamic Encoder E Ｓ : Static encoder E ｈ : First encoder E ｒ : Second encoder

Claims

1. A motion retargeting device for generating robot movements, configured to provide first motion data of a person to a machine-learned neural network and generate second motion data for operating a robot using the neural network, the neural network comprising: an encoder that encodes the first motion data into latent motion data in a latent space represented by a primitive skeleton that is a skeletal structure common to the person and the robot; and a decoder that generates the second motion data of the robot based on the latent motion data and skeletal information of the robot.

2. The motion retargeting device of claim 1, wherein the skeletal information of the robot is data representing the robot's skeleton using a pseudo-structure that approximates the actual structure of the robot to the human body structure.

3. The motion retargeting device of claim 1, wherein the neural network is machine-trained using data on the robot's movements that satisfy the structural constraints of the robot, and is configured to generate the second motion data that satisfies the structural constraints of the robot.

4. The motion retargeting device of claim 1, wherein the neural network is machine-trained using a loss function that penalizes the generation of data that causes the robot to perform movements that are impossible due to structural constraints of the robot.

5. A machine learning method for a neural network, wherein the neural network comprises: a first encoder that encodes human motion data into latent motion data in a latent space represented by a primitive skeleton that is a skeletal structure common to the human and a robot; a first decoder that generates motion data for the human based on the latent motion data; a second encoder that encodes robot motion data into latent motion data in the latent space represented by the primitive skeleton; and a second decoder that generates motion data for the robot based on the latent motion data, and the machine learning method includes causing the neural network to learn machine learning using motion data based on actual movements of the human and motion data based on actual movements of the robot.

6. The neural network machine learning method described in claim 5, wherein training the neural network includes using a loss function that penalizes the generation of data that causes the robot to perform actions that are impossible due to structural constraints of the robot.

7. The neural network machine learning method described in claim 5, wherein the motion data based on the actual movement of the robot includes skeletal information of the robot, and the skeletal information of the robot is data representing the skeletal structure of the robot using a pseudo-structure that represents the actual structure of the robot in a manner that approximates the structure of the human body.

8. A method for generating motion data of a robot, comprising generating motion data of the robot using at least the first encoder and the second decoder in the neural network trained by machine learning using the method of any one of claims 5 to 7, wherein generating motion data of the robot using at least the first encoder and the second decoder comprises providing first motion data of a person to the first encoder to obtain the latent motion data, and generating motion data of the robot by the second decoder from the latent motion data and the skeletal structure of the robot.

9. A method for controlling a robot, comprising operating the robot using robot motion data generated by the method according to claim 8.

Images