Learning device

Abstract

To implement a learning device that improves estimation accuracy of a grip posture by a robot hand being symmetry.SOLUTION: A learning device 130 according to an embodiment of the present disclosure includes a learning unit 134 that performs machine learning on a posture of gripping an object by a robot hand 10 by using training data in which a first posture and a second posture are represented by one parameter set, the first posture being of the robot hand 10 being symmetry to rotation for two times, the second posture being established by rotating the first posture around a rotation symmetry axis by 180°.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a learning device.

Background Art

[0002] Patent Document 1 describes a gripping device that grips an object using a robot hand having symmetry.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When machine learning the gripping posture by a robot hand having rotational symmetry (e.g., a two-finger hand), the symmetry may destabilize the learning and reduce the estimation accuracy. For example, if a gripping posture of 0° rotation is output in a certain area and a gripping posture of 180° rotation is output in another area, there is a risk that an incorrect gripping posture will be output at the boundary surface between the two areas.

[0005] The present disclosure has been made in view of such problems, and an object thereof is to provide a learning device that improves the estimation accuracy of the gripping posture by a robot hand having symmetry.

Means for Solving the Problems

[0006] The learning device according to the present disclosure includes a learning unit that machine-learns a posture of gripping an object by the robot hand using training data representing a first posture of a robot hand having two-fold rotational symmetry and a second posture obtained by rotating the first posture 180° about an axis of rotational symmetry as one parameter set.

Effects of the Invention

[0007] According to this disclosure, it is possible to provide a learning device that can improve the accuracy of estimating the gripping posture of a symmetrical hand. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of the gripping system according to Embodiment 1. [Figure 2] This figure shows an example of the configuration of a robot hand according to Embodiment 1. [Figure 3] This is a diagram illustrating a method for representing the posture of a robot hand according to Embodiment 1. [Modes for carrying out the invention]

[0009] The following describes specific embodiments applying this disclosure with reference to the drawings. However, this disclosure is not limited to the following embodiments. Also, for clarity, the following descriptions and drawings have been simplified as appropriate.

[0010] Embodiment 1 Figure 1 is a block diagram showing the configuration of the gripping system 1 according to Embodiment 1. The gripping system 1 includes a detection device 2, a robot hand 10, and a control system 100. The control system 100 is connected to the detection device 2 and the robot hand 10 via a wireless or wired communication network.

[0011] The grasping system 1 estimates the grasping posture of the robot hand using an inference model generated in advance by machine learning. The grasping system 1 grasps the object with the estimated grasping posture. Here, the inference model can be learned by, for example, a neural network.

[0012] Detection device 2 detects each position in the three-dimensional space where an object is placed. In other words, detection device 2 detects (captures, measures) the position where an object exists in three-dimensional space. Detection device 2 detects whether or not an object exists at each position in three-dimensional space. Detection device 2 is, for example, a three-dimensional camera such as an RGB-D camera or stereo camera, a depth camera, or LiDAR (Light Detection And Ranging), but is not limited to these. Also, in Embodiment 1, the position in three-dimensional space is represented by a voxel, but is not limited to this.

[0013] The robot hand 10 is configured to grasp objects placed in three-dimensional space. The movement of the robot hand 10 is controlled by the control system 100. In other words, the robot hand 10 grasps objects under the control of the control system 100. The robot hand 10 may be an end effector, such as one located at the tip of a robot arm (not shown).

[0014] Figure 2 illustrates a robot hand 10. The robot hand 10 has a hand body 12, two finger parts 14, a plurality of links 16, and a plurality of joint parts 18. The finger parts 14 are connected to the hand body 12 via the plurality of links 16 and joint parts 18. The finger parts 14 move when one or more joint parts 18 are driven. Here, some of the joint parts 18 may be driven. A drive device such as a motor is incorporated into the driveable joint parts 18. The robot hand 10 can assume six degrees of freedom in grasping postures in three-dimensional space.

[0015] Here, a reference point Pr is set for the robot hand 10. The reference point Pr is also called the TCP (Tool Center Point). The reference point Pr is the origin of the hand coordinate system (x,y,z). The (+x) direction is the direction in which the robot hand 10 approaches the object. The y direction is the direction along which the finger portion 14 moves (opens and closes). The z direction, which is perpendicular to the xy plane, is the direction of the normal vector of the plane on which the finger portion 14 moves. The reference point Pr can be determined arbitrarily. In the example in Figure 2, the reference point Pr is located near the center in the y direction on the front surface 12a of the hand body 12, but it is not limited to this. The reference point Pr may be located outside the hand body 12 or inside the hand body 12.

[0016] The robot hand 10 has twofold rotational symmetry with respect to the x-axis as the axis of rotational symmetry. Furthermore, the robot hand 10 has plane symmetry with respect to a plane of symmetry containing the axis of rotational symmetry (e.g., the xy-plane, the xz-plane).

[0017] Referring again to Figure 1, the control system 100 is, for example, a computer such as a server. The control system 100 can be implemented, for example, by cloud computing. Alternatively, the control system 100 can be implemented by multiple computers. In this case, the multiple components of the control system 100, which will be described later, may each be implemented on physically different computers.

[0018] The control system 100 has a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 (IF) as its main hardware components. The control unit 102, storage unit 104, communication unit 106, and interface unit 108 are interconnected via a data bus or the like. If the control system 100 is implemented using multiple computers, each of the computers may have the hardware configuration shown in Figure 1.

[0019] The control unit 102 is a processor such as a CPU (Central Processing Unit), etc. The control unit 102 has a function as an arithmetic unit that performs control processing, arithmetic processing, etc. Note that the control unit 102 may have a plurality of processors. The storage unit 104 is a storage device such as a memory or a hard disk, for example. The storage unit 104 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory), etc. The storage unit 104 has a function for storing control programs and arithmetic programs executed by the control unit 102. That is, the storage unit 104 (memory) stores one or more instructions. Also, the storage unit 104 has a function for temporarily storing processing data, etc. The storage unit 104 may include a database. Also, the storage unit 104 may have a plurality of memories.

[0020] The communication unit 106 performs the processing necessary for communicating with other devices via a network. The communication unit 106 may include a communication port, a router, a firewall, etc. The interface unit 108 is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, a touch panel, or a mouse, and an output device such as a display or a speaker. The interface unit 108 may be configured such that the input device and the output device are integrated, such as a touch panel, for example. The interface unit 108 receives an operation of inputting data by the user and outputs information to the user.

[0021] The control system 100 has a learning device 130 and a control device 140. The learning device 130 and the control device 140 may be physically separate devices. In this case, each of the learning device 130 and the control device 140 has the hardware configuration described above. Also, the learning device 130 and the control device 140 may be physically the same device. For example, the function of the control device 140 may be incorporated into the learning device 130.

[0022] The learning device 130 includes, as components, a training data acquisition unit 132 and a learning unit 134. The control device 140 includes, as components, a position acquisition unit 142, an estimation unit 144, and a hand control unit 146.

[0023] Each of the above-described components can be realized, for example, by causing a program to be executed under the control of the control unit 102. More specifically, each component can be realized by the control unit 102 executing a program (instruction) stored in the storage unit 104. Further, a necessary program may be recorded on an arbitrary non-volatile storage medium and installed as needed to realize each component. Also, each component is not limited to being realized by software by a program, and may be realized by any combination of hardware, firmware, and software. Further, each component may be realized using a user-programmable integrated circuit such as, for example, an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, a program composed of the above-described components may be realized using this integrated circuit.

[0024] The training data acquisition unit 132 generates training data used to generate an inference model. The training data acquisition unit 132 generates, for example, training data indicating a gripping posture when the reference point Pr of the robot hand 10 is at each position in the three-dimensional space where the object is arranged. The training data acquisition unit 132 can, for example, acquire opposing points on the surface of the object and determine the gripping posture of the robot hand 10 such that the opposing points become the contact points of the respective finger portions 14. The training data acquisition unit 132 may determine a gripping posture that does not cause the gripped object to fall in consideration of gravity.

[0025] The training data acquisition unit 132 generates training data using the position of the reference point Pr in each grasping posture and posture data indicating that grasping posture. Here, the posture data is a parameter set that includes parameters relating to the unit vector of the direction in which the robot hand 10 approaches the object when realizing the corresponding grasping posture (the direction when the x direction in Figure 2 is projected into three-dimensional space) and parameters relating to the normal vector of the plane on which the finger portion 14 moves (the plane when the xy plane in Figure 2 is projected into three-dimensional space). Note that the parameter set may also include parameters relating to the normal vector of the plane when the xz plane in Figure 2 is projected into three-dimensional space, instead of parameters relating to the normal vector of the plane when the xy plane in Figure 2 is projected into three-dimensional space.

[0026] Referring to Figure 3, the method for representing posture data according to Embodiment 1 will be explained. Referring to (a), the posture of the robot hand 10 is represented by a rotation matrix formed by arranging three mutually perpendicular vectors ex, ey, and ez. Vector ex is a unit vector in the direction in which the robot hand 10 approaches the object (a direction parallel to the axis of rotational symmetry). Vector ez (normal vector ez) is a unit vector perpendicular to the plane on which the finger portion 14 moves, that is, one of the planes of symmetry of the robot hand 10. Note that once vector ex and the normal vector ez are determined, ey is also determined, and the posture of the robot hand 10 is determined.

[0027] Conventionally, posture data including the vector ex and the normal vector ez was used. However, the posture data representing one posture of the robot hand 10 and the posture data representing the posture obtained by rotating that posture 180° around the axis of rotational symmetry were different from each other, which could lead to a decrease in the accuracy of gripping posture estimation.

[0028] Referring to (b), the training data acquisition unit 132 assumes that the vector ez is distributed symmetrically with respect to the symmetry plane, and uses a parameter set including the parameters of that distribution and the vector ex as pose data. In this way, one pose of the robot hand 10 and the pose obtained by rotating that pose 180° around the axis of rotational symmetry are represented by a single parameter set.

[0029] As the distribution of the normal vector ez, a distribution defined on a sphere and symmetric with respect to the plane of symmetry (the plane on which the finger portion 14 moves) may be used (e.g., a two-dimensional Bingham distribution). A two-dimensional Bingham distribution can be represented, for example, by six parameters which are elements of a cubic symmetric matrix. The six parameters determine the position of the peak of the distribution and the shape of the distribution.

[0030] By using a distribution such as a two-dimensional Bingham distribution, information regarding the reliability of the estimated pose data can also be obtained. For example, if the distribution has a large variability, it may be judged as having low reliability. (c) shows multiple poses represented by pose data with high reliability. Multiple vectors ez randomly selected from the distribution are almost identical, and thus represent almost identical poses. (d) shows multiple poses represented by pose data with low reliability. On the other hand, when grasping a cylindrical object from one end, the robot hand 10 can grasp the object from any direction. In this case, as shown in (e), the distribution of the normal vector ez is uniform.

[0031] Referring again to Figure 1, the training data acquisition unit 132 generates a TSDF volume for each voxel from a depth image obtained by, for example, capturing (rendering) a scene in three-dimensional space (e.g., virtual space) with objects placed in it from a predetermined direction, and uses this as input data for the training data. The TSDF volume indicates the distance from each voxel in three-dimensional space to the object closest to that voxel.

[0032] For example, the training data acquisition unit 132 uses the posture data for each position of the robot hand 10 with a reference point Pr as output data in the training data. The training data acquisition unit 132 can, for example, select a grasping posture for each position of the robot hand 10 with a reference point Pr, and calculate the parameters of a two-dimensional Bingham distribution using the normal vector ez in that grasping posture and an appropriate loss function.

[0033] The output data in the training data may further include a score and a mask value for each voxel where the robot hand 10 has a reference point Pr. The mask value indicates "true" (e.g., "1") when the object can be grasped (i.e., a grasping posture exists) when the reference point Pr is present at each position (voxel) in three-dimensional space. On the other hand, the mask value indicates "false" (e.g., "0") when the object cannot be grasped (i.e., a grasping posture does not exist) when the reference point Pr is present at that position. In the output data, the score represents the quality of grasping when the reference point Pr is present in three-dimensional space. The higher the quality of grasping, the more robustly the robot hand 10 can grasp the object.

[0034] The training data is not limited to the examples described above. Positions in three-dimensional space may be represented by point cloud data instead of voxels. In addition, in the examples described above, attitude data was calculated assuming a reference position Pr at each position in the input data, but any known technique may be used to determine the attitude data for the input data. For example, the gripping attitude may be determined using the positions of objects shown in the point cloud data as contact points.

[0035] The learning unit 134 trains an inference model by performing machine learning, so that it takes input data from the training data as input and outputs output data from the training data as output. In this way, the learning unit 134 generates a trained inference model. The inference model can be implemented by, for example, a neural network such as an FCN (Fully Convolutional Network), but is not limited to this.

[0036] The input to the neural network may be, for example, voxel data (TSDF volume) of a scene with dimensions of 40x40x40, which may contain multiple objects. In this case, the output of the neural network may be, for example, a score with dimensions of 40x40x40, a mask value with dimensions of 40x40x40, and pose data with dimensions of 40x40x40x9. The pose data may include, for example, a 3D vector ex and six elements of a 3x3 symmetric matrix representing a 2D Bingham distribution. For each of the multiple voxels, the score, mask value, and pose data are output.

[0037] The control device 140 controls the robot hand 10 to grasp an object placed in three-dimensional space. The position acquisition unit 142 acquires TSDF volume or point cloud data for each voxel in three-dimensional space based on the detection results from the detection device 2.

[0038] The estimation unit 144 estimates the posture data of the robot hand 10 using an inference model. The estimation unit 144 may also input the TSDF volume acquired by the position acquisition unit 142 into the inference model and acquire the posture data output from the inference model.

[0039] The estimation unit 144 may determine one or more normal vectors ez based on the distribution of normal vectors ez. The estimation unit 144 may determine normal vectors ez corresponding to the peaks of the distribution, or it may sample normal vectors ez from the distribution. The estimation unit 144 may then estimate the gripping posture based on the determined normal vectors ez and the unit vectors ex included in the posture data.

[0040] The estimation unit 144 may determine whether the estimated posture data is highly reliable based on the variability of the distribution of the normal vector ez. The estimation unit 144 may output a gripping posture to the hand control unit 146 based on posture data with a reliability higher than a predetermined value.

[0041] The estimation unit 144 may sample multiple normal vectors ez from the distribution of normal vectors ez and estimate multiple gripping postures corresponding to each of the multiple normal vectors ez. This is useful when considering other constraints such as placement and collision avoidance.

[0042] The hand control unit 146 controls the robot hand 10 based on the determined gripping posture. The hand control unit 146 may, for example, place a reference point Pr in the voxel where the gripping posture is determined, and reproduce the posture of the robot hand 10 using a rotation matrix obtained according to the estimated gripping posture.

[0043] The learning device according to Embodiment 1 can improve the accuracy of estimating the grasping posture of a robot hand by representing two symmetrical postures of the robot hand with a single parameter set.

[0044] The program described above includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0045] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, in the embodiments described above, the gripping posture was represented by three orthogonal unit vectors, but the gripping posture may be represented by other methods. [Explanation of Symbols]

[0046] 1. Gripping System 2. Detection device 10 Robot Hand 12 Hand Body 14 Finger section 100 control systems 130 Learning device 132 Training Data Acquisition Unit 134 Learning Department 140 Control device 142 Position acquisition part 144 Estimation Department 146 Hand Control Unit

Claims

1. A learning unit that uses training data representing a first posture of a robot hand that is rotationally symmetrical twice, and a second posture obtained by rotating the first posture 180° around the axis of rotational symmetry, as a single parameter set, to machine learn the posture of the robot hand grasping an object. Equipped with, The robot hand has plane symmetry with respect to a plane of symmetry that includes the axis of rotational symmetry, The parameter set includes parameters for the distribution of normal vectors perpendicular to the plane of symmetry and parameters for vectors parallel to the axis of rotational symmetry. Learning device.

2. The distribution is defined on a sphere and is symmetric with respect to the plane of symmetry. The learning device according to claim 1.

3. An estimation unit estimates a parameter set representing the posture of the robot hand grasping the object using an inference model generated by the learning unit, and determines whether the reliability of the parameter set is high or low based on the variability of the distribution based on the parameter set. The learning device according to claim 2, comprising:

4. Estimation Unit: Estimates a parameter set representing the posture of the robot hand grasping the object using an inference model generated by the learning unit, samples a plurality of normal vectors from the distribution based on the parameter set, and estimates a plurality of postures corresponding to each of the plurality of normal vectors. The learning device according to claim 2, comprising:

Images