Robot hand and robot hand control method

Abstract

To provide a robot hand that can obtain the shapes of a target object and an environment at the same time with high accuracy just before and after work, and a robot hand control method.SOLUTION: A robot hand comprises a substrate, finger parts connected to the substrate, and sensors installed in the vicinity of contact points of the finger parts.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] ,

[0006] , , ,

[0005] , , ,

[0001] The present invention relates to a robot hand and a method for controlling a robot hand.

Background Art

[0002] A robot that performs work includes a hand that functions like a human hand and an arm that functions like a human arm. Such work is controlled, for example, by a control device. The control device performs work based on an image captured by a imaging device installed in the work space, for example. In this case, when the hand is brought close to the target object in order to grasp the target object, the arm or the hand becomes a blind spot of the imaging device and blocks the field of view. For this reason, it has been proposed to attach an imaging device to the tip of a robot arm (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0007] (2) In addition, in a robot hand according to one aspect of the present invention, the sensor may be installed at a position offset from the central axis connecting the fingertip to the base body in the direction of bending the finger, from the center of the fingertip of the fingertip.

[0008] (3) In addition, in a robot hand according to one aspect of the present invention, the finger portion may be provided with a skin portion and a nail portion, and the sensor may be installed between the skin portion and the nail portion.

[0009] (4) In addition, in a robot hand according to one aspect of the present invention, the sensor may be installed at a location offset from the center of the fingertip of the finger portion in a direction perpendicular to the bending direction of the finger from the central axis connecting the finger portion and the base body.

[0010] (5) In addition, in a robot hand according to one aspect of the present invention, there are two or more fingers, and the sensors are installed near the contact points of at least two of the fingers, and the sensors are cameras, and stereo vision is performed using images captured by the sensors installed on the two fingers.

[0011] (6) To achieve the above objective, a robot hand control method according to one aspect of the present invention is an operation method for operating a robot hand comprising a base body and finger parts connected to the base body and capable of manipulating an object, and having a camera near the contact point of the finger parts (the place where the finger and the object come into contact for the purpose of gripping), wherein the method involves determining an object, determining an environmental target which is the target for manipulating the object, determining a gripping method for gripping the object, capturing images of the object and the environment which is the target for manipulating the object with the camera while the object to be gripped is being held by the finger, acquiring positional information of the object based on the captured images, and determining the amount of manipulation of the object based on the acquired positional information.

[0012] (7) In addition, in a robot hand control method according to one aspect of the present invention, there may be a plurality of robot hands, at least two of the plurality of robot hands may be equipped with cameras, the cameras equipped on the at least two robot hands may capture images of the same target object, and the at least two robot hands may be controlled in coordination using the captured images. [Effects of the Invention]

[0013] (1) to (7) are designed with sensors at the fingertips of the fingers, allowing for simultaneous and highly accurate acquisition of the shape of the target object and the environment immediately before and after work. Furthermore, according to (1) to (7), when the sensor is a camera, blind spots can be reduced compared to conventional designs. [Brief explanation of the drawing]

[0014] [Figure 1] This figure shows an example configuration of a robot hand according to the embodiment. [Figure 2] This is a diagram illustrating an example of where to place a camera on your fingertip. [Figure 3] This diagram illustrates an example of the mounting position of a fingertip camera as seen from the palm side. [Figure 4] This figure shows an example of the shooting range when using a camera on your fingertip. [Figure 5] This figure shows an example of the first image captured by the camera when the object being handled is grasped. [Figure 6] This figure shows a second example of an image captured by the camera when the object being handled is grasped. [Figure 7] This diagram illustrates an example of control using images captured by a camera placed on the fingertip. [Figure 8] This figure shows an example configuration of a robot equipped with a robot hand having a camera according to the embodiment. [Figure 9] This is a flowchart of an example of a control procedure for a robot hand using a robot control device according to the embodiment. [Figure 10] This is a diagram showing an example in which the robot hand according to the embodiment also has a camera on the palm. [Figure 11] This is a diagram showing an example of the shooting ranges of the camera on the palm and the cameras on the fingertips. [Figure 12] This is a diagram for explaining a comparative example of a robot hand in which a camera is attached to a base.

Embodiment for Carrying out the Invention

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scales of each member are appropriately changed in order to make each member recognizable.

[0016] (Example of the Configuration of a Robot Hand) FIG. 1 is a diagram showing an example of the configuration of a robot hand according to the present embodiment. The robot hand 1 in FIG. 1 has an example in which there are four finger parts. The robot hand 1 (end effector) in the configuration example of FIG. 1 includes finger parts 101, finger parts 102, finger parts 103, finger parts 104, a base 111, a camera 141 (sensor), a camera 142 (sensor), a camera 143 (sensor), and a camera 144 (sensor). The robot hand 1 is connected to an arm 121 via a joint. Note that the robot hand 1 includes at least two finger parts. Also, the robot hand 1 includes at least one camera. In the embodiment, a camera is described as an example of a sensor, sensor but it is not limited to this. For example, the sensor may be an RGBD camera that can acquire RGB images and depth information, or a combination of an RGB camera and a distance sensor, etc.

[0017] The finger part 101 corresponds to, for example, the thumb of a human. The finger part 102 corresponds to, for example, the index finger of a human. The finger part 103 corresponds to, for example, the middle finger or the ring finger of a human. The finger part 104 corresponds to, for example, the little finger or the ring finger of a human. Each finger part includes joints and articulations. The base 111 includes a position corresponding to the back of a human hand and the palm. Note that the finger parts 101 to 104 are connected to the base 111.

[0018] Cameras 141-144 are RDG-D cameras that can obtain RGB information and depth information. Alternatively, cameras 141-144 may be, for example, a combination of a CCD (Charge Coupled Device) imaging device, a CMOS (Complementary MOS) imaging device, and a depth sensor.

[0019] In this embodiment, the first finger portion is one of finger portions 101, 102, 103, and 104. The second finger portion is any finger portion other than the first finger portion. In this embodiment, the first camera is one of cameras 141, 142, 143, and 144. The second camera is any camera other than the first camera.

[0020] Camera 141 is positioned, for example, at the fingertip of a human thumb. Alternatively, camera 141 may be positioned at the fingertip, distal phalanx, or region 161 including the distal phalanx, corresponding to a human thumb.

[0021] Camera 142 is positioned, for example, at the fingertip of a human index finger. Alternatively, camera 142 may be positioned at the fingertip, distal phalanx, or region 162 including the distal phalanx, corresponding to a human index finger.

[0022] Camera 143 is positioned, for example, at the fingertip of a human middle or ring finger. Alternatively, camera 143 may be positioned at the fingertip, distal phalanx, or region 163 including the distal phalanx, corresponding to a human middle or ring finger.

[0023] Camera 144 is positioned, for example, at the fingertip of a human little finger or ring finger. Camera 144 is positioned at the fingertip, distal phalanx, and region including the distal phalanx, corresponding to a human little finger or ring finger. 6 It may also be installed at location 4.

[0024] Cameras 141-144 are installed near the contact points of the fingers (the places where the fingers and the target object come into contact for gripping purposes).

[0025] Figure 2 illustrates an example of the placement of a camera at the fingertip. As shown in Figure 2, the camera 142 at the fingertip is installed, for example, between the epidermal portion 1021 and the nail portion 1022 of the finger 102. Alternatively, the camera 142 may be installed, for example, at the tip of the nail portion 1022. Thus, each finger portion comprises an epidermal portion and a nail portion. Note that the placement shown in Figure 2 is just an example and is not limited to this.

[0026] Figure 3 illustrates an example of the mounting position of a fingertip camera as seen from the palm side. Note that the example shown in Figure 3 is just one example and is not limited to this. As shown in Figure 3, the turtle of the finger part 101 La The camera is positioned offset from the central axis connecting the finger and the base in the direction of arrow g11, for example, relative to the longitudinal centerline of the finger, and outward in the direction of the finger's bending. Furthermore, the camera on the finger 101 is positioned with the camera's z-axis g12 tilted towards the center of the finger.

[0027] The camera on the finger portion 102 is positioned offset from the central axis connecting the finger portion and the base body in the direction of finger bending, for example, in the direction of arrow g13 (direction of finger portion 103), relative to the longitudinal centerline of the finger portion.

[0028] The camera on the finger portion 103 is positioned offset from the central axis connecting the finger portion and the base body in the direction of finger bending, for example, in the direction of arrow g14 (direction of finger portion 104), relative to the longitudinal centerline of the finger portion.

[0029] The camera on the finger portion 104 is positioned offset outward from the central axis connecting the finger portion and the base body in the direction of finger bending, for example, in the direction of arrow g15 (outward) relative to the longitudinal centerline of the finger portion.

[0030] Furthermore, each camera is positioned at a location offset from the center of the fingertip of the finger part, perpendicular to the bending direction of the finger from the central axis connecting the finger part and the base 11. The reason for this positioning is to minimize contact between the camera and the object when the object is grasped by the finger part.

[0031] (Example of shooting range) Figure 4 shows an example of the shooting range with a camera at the fingertip. Note that the example in Figure 4 is one in which there are four fingers, each equipped with a camera. Camera 141 is mounted on the fingertip of finger part 101. Shooting range g101 is an example of the shooting range of camera 141. Camera 142 is mounted on the fingertip of finger part 102. Shooting range g102 is an example of the shooting range of camera 142. Camera 143 is mounted on the fingertip of finger unit 103. Shooting range g103 is an example of the shooting range of camera 143. Camera 144 is mounted on the fingertip of finger unit 104. Shooting range g104 is an example of the shooting range of camera 144. Furthermore, as shown in Figure 4, the shooting range may overlap when the fingers are in contact. Also, the shooting range shown in Figure 4 is just an example and is not limited to this.

[0032] (Example of images taken) Next, we will explain examples of images captured by the camera when grasping an object, using Figures 5 and 6. Figures 5 and 6 show the finger portion 101 and the finger portion 10 2 This is an example of attempting to grasp an object.

[0033] Figure 5 shows an example of a first image captured by a camera when grasping an object. Image g201 is a side view of the robot hand 1 before it grasps the coin-shaped object Obj. Image g211 is an example of an image captured by camera 142 installed on finger 102. Image g212 is an example of an image captured by camera 143 installed on finger 103. Image g213 is an example of an image captured by camera 144 installed on finger 104. Image g214 is an example of an image captured by camera 141 installed on finger 101. Note that images g211 to g214 are examples where the fingertips are tilted relative to the object.

[0034] Figure 6 shows a second example of images captured by the camera when grasping an object. Image g221 is an example of an image captured by camera 142 installed on finger 102. Image g222 is an example of an image captured by camera 143 installed on finger 103. Image g223 is an example of an image captured by camera 144 installed on finger 104. Image g224 is an example of an image captured by camera 141 installed on finger 101. Note that images g221 to g224 are examples where the fingertip is directly above the target object.

[0035] As shown in Figures 5 and 6, according to this embodiment, when grasping an object, the fingertips to The camera that is installed can capture images, which can then be used by the control device of the robot hand 1 to precisely control the gripping operation of the robot hand 1.

[0036] Next, we will explain an example of control using images captured by a camera mounted on the fingertip. Figure 7 illustrates an example of control using images captured by a camera placed on the fingertip. Images g301 to g303 show an example where a person changes the gripping angle when connecting a connector cable to its terminal with their finger. As shown, a person may rotate the object within their finger after gripping it. When performing such an action with the robot hand 1, for example, as shown in images g310 to g320, the target object Obj is grasped by the three fingers 101, 102, and 103 and rotated, for example, clockwise as shown in g330. In this case, as shown in Figure 7, allowing the contact points to roll makes it possible to change the orientation of small target objects. In such control, it is necessary to acquire the contact position between the fingertips and the target object Obj.

[0037] As described above, in this embodiment, a camera is installed at the fingertip of the robot hand. This makes it possible to simultaneously and accurately acquire the shape of the target object and the environment immediately before and after grasping.

[0038] Furthermore, as in this embodiment, when a camera is mounted on the fingertip, the captured images can be used for grasping small objects in an in-hand state, controlling the posture before pegging into a hole in an in-hand state, making precise positional adjustments in a visual servo, correcting the hand posture in a visual servo, and so on.

[0039] While the finger unit 102 is used in a variety of tasks, the finger units 103 and 104 may not be used depending on the task. However, since the finger units 103 and 104 are used when the target object is not small or is spherical, it is desirable in this embodiment to also install cameras on the finger units 103 and 104.

[0040] (Robot hand system) Here, we will describe an example of a robot configuration that includes a robotic hand equipped with a camera. Figure 8 shows an example configuration of a robot equipped with a camera-equipped robot hand according to this embodiment. As shown in Figure 8, the robot 10 comprises a robot hand 1A-1 and a control device 5. Note that the robot 10 may also be equipped with two robot hands (1A-1, 1A-2). The robot hand 1A-1 includes, for example, an actuator 201-1, a sensor 202-1, and a camera 203-1. The robot hand 1A-2 includes, for example, an actuator 201-2, a sensor 202-2, and a camera 203-2. The control device 5 includes, for example, an motion control unit 501, a camera motion control unit 502, an image processing unit 503, and a storage unit 504.

[0041] In the following description, unless one of the robot hands 1A-1 or 1A-2 is specified, it will be referred to as robot hand 1. A If one of actuators 201-1 and 201-2 is not specified, it will be referred to as actuator 201. If one of sensors 202-1 and 202-2 is not specified, it will be referred to as sensor 202. If one of cameras 203-1 and 203-2 is not specified, it will be referred to as camera 203.

[0042] The robot hand 1A and the control unit 5 are connected by wire or wireless connection. The control unit 5 may be located inside or outside the robot 10. Furthermore, all or some of the functions of the control unit 5 may be implemented on the cloud.

[0043] The actuators 201 are installed, for example, at the joints of the fingers, the joints between the fingers and the base, and the joints between the base and the arm. The actuators 201 operate in accordance with the control of the motion control unit 501.

[0044] Sensors 202 include, for example, encoders installed at each joint, angular velocity sensors installed at each part, six-axis sensors, gyro sensors, etc.

[0045] Camera 203 is at least one, for example, the cameras (141-144) in Figure 1.

[0046] The motion control unit 501 controls the operation of the actuator 201 using the image captured by the camera 203 and the detected value detected by the sensor 202.

[0047] The camera motion control unit 502 controls the operation of the camera 203. The camera motion control unit 502 may, for example, change the angle of the camera according to the gripping state. The camera motion control unit 502 may, for example, change the field of view of the camera 203 according to the work content or gripping state. The camera motion control unit 502 may, for example, switch the sensitivity of the camera 203 according to the work content or gripping state.

[0048] The image processing unit 503 performs image processing on the images captured by the camera 203 using well-known methods. If there are multiple cameras 203, the image processing unit 503 may process the images individually, overlay them, or process them as stereo images.

[0049] The memory unit 504 stores programs, thresholds, formulas, predetermined values, etc., used by the control device 5 for processing. The memory unit 504 may also store learned object identification models for identifying target objects, learned gripping plan models for generating gripping plans, etc. All or part of the information stored in the memory unit 504 may be stored on other servers or in the cloud.

[0050] (Example of robot hand control procedure) Here, the robot hand 1 is controlled by the robot control device 5. A An example of the control procedure will be explained. Figure 9 is a flowchart of an example of the control procedure for the robot hand by the robot control device according to this embodiment.

[0051] (Step S1) The control device 5 determines the object to be worked on. The determination of the object may be based on, for example, instructions from the worker, or on images captured by a camera. Alternatively, the control device 5 may determine the object by inputting, for example, the captured image and the positional information of each object in the image into a trained model.

[0052] (Step S2) The control device 5 determines the target object as the environmental target to be worked on.

[0053] (Step S3) The control device 5 determines a gripping method for gripping the target object, for example, by a taxonomy method (see Reference 1, for example), depending on the target object.

[0054] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human Grasp Types” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

[0055] (Step S4) The control device 5, while gripping the object to be grasped with its fingers, captures images of the object and the environment in which the object is being manipulated together using the camera. The control device 5 obtains the position information of the object from the information contained in the captured images.

[0056] (Step S5) The control device 5 generates the amount of manipulation of the gripped object based on the acquired position information.

[0057] (Step S6) The control device 5 controls the movement of the robot hand based on the determined operation amount to grasp the target object, for example.

[0058] The work is not limited to gripping; it may also include tasks such as opening bottle caps or pressing buttons.

[0059] Furthermore, if the robot hand system has at least two robot hands, each robot hand is equipped with a camera. The cameras of each robot hand capture images of the same target object. The control device 5 uses the captured images to perform coordinated control of at least two robot hands.

[0060] (An example of a camera mounted on the palm of the hand) Next, we will describe an example of a robotic hand equipped with a camera in its palm. Figure 10 shows an example in which the robot hand according to this embodiment is also equipped with a camera in the palm. As shown in Figure 10, the robot hand 1B is equipped with cameras 131 to 134 in the palm in addition to cameras 141 to 144. The robot hand 1B is equipped with at least two finger sections. The robot hand 1B is also equipped with at least one camera.

[0061] Cameras 131-134 are RDG-D cameras that can obtain RGB information and depth information. Alternatively, cameras 131-134 may be, for example, a combination of a CCD imaging device, a CMOS imaging device, and a depth sensor.

[0062] Camera 131 is positioned, for example, at a location corresponding to the thenar eminence of a human thumb. Alternatively, camera 131 may be positioned, for example, at a location corresponding to the outer side of the proximal phalanx of a human thumb, not on the index finger side. In this way, camera 131 is positioned between finger portion 101 and finger portion 104.

[0063] Camera 132 is positioned, for example, at a location corresponding to the thenar eminence of a human thumb. Alternatively, camera 132 may be positioned at a location corresponding to the index finger side of the proximal phalanx of a human thumb. In this way, camera 132 is positioned between finger portion 101 and finger portion 102.

[0064] Camera 133 is positioned, for example, on the thumb side of the base of the four fingers of a human, corresponding to the sides of the thumb and index finger on the thenar eminence. Thus, camera 133 is positioned between finger portion 101 and finger portion 102.

[0065] The camera 134 is positioned, for example, on the side of the little finger or ring finger at the base of the four fingers of a human, or on the side of the hypothenar eminence. Thus, the camera 134 is positioned between the finger portion 104 and the finger portion 101.

[0066] Note that the placement of cameras 131-134 shown in Figure 10 is just an example and is not limited to this. Also, although the example in Figure 10 shows a robot hand with four fingers, the number of fingers can be two or more.

[0067] Figure 11 shows an example of the shooting range of the palm camera and the fingertip camera. Note that the example in Figure 11 has two cameras in the palm and four cameras in the fingers. The shooting range g151 is an example of the shooting range of camera 133. The shooting range g152 is an example of the shooting range of camera 134. Note that the shooting ranges of each camera shown in Figure 11 are examples only and are not limited to these.

[0068] (Example of a stereo camera) Next, the robot hand butThis section explains an example of using the two cameras provided as a stereo camera. The robot hand may achieve stereo vision using any two of the cameras 141-144 installed on the fingertips, for example. For example, camera 141 and camera 142 may be used to achieve stereo vision. Alternatively, stereo vision may be achieved using any two of the cameras 131-134 installed on the palm as described above, or using any two of the cameras installed on the fingertips and the cameras installed on the palm.

[0069] Thus, the robot hand is equipped with at least two cameras, and uses two of these cameras to achieve stereo vision. In this case, stereo vision of the target object is possible even if the wrist angle changes, and it has the advantage of making it easier to observe the object before grasping it compared to when stereo cameras are attached to the forearm or wrist.

[0070] (Comparison with conventional technology) Next, we will compare the robot hand of this embodiment with a conventional robot hand.

[0071] Figure 12 is a diagram illustrating a comparative example of a robot hand in which the camera is attached to the base body. In the robot hand 901A in which the camera 921 is attached to the palm of the base body 941, for example, when grasping an object Obj with the fingers 911 and 912 as shown in Figure 12, the view is obstructed by the object Obj, and the tip of the hand (finger's end) becomes invisible. In contrast, in this embodiment, for example, the camera is mounted on the fingertip of the finger, which reduces the obstruction of the hand by the object being grasped.

[0072] Furthermore, a program to implement all or part of the functions of the control device 5 in this invention may be recorded on a computer-readable recording medium, and all or part of the processing performed by the control device 5 may be performed by loading the program recorded on this recording medium into a computer system and executing it. Herein, "computer system" includes hardware such as an OS and peripheral devices. Furthermore, "computer system" also includes a WWW system equipped with a homepage provisioning environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" also includes volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, which holds the program for a certain period of time.

[0073] Furthermore, the above program may be transmitted from a computer system that stores the program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. In addition, the above program may be for the purpose of realizing a part of the functions described above. Furthermore, it may be a so-called differential file (differential program) that can realize the functions described above in combination with a program already recorded in the computer system.

[0074] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0075] 1,1A,1B...Robot hand, 10...Robot, 101~10 4 ...Finger part, 111...Base, 131~134, 141~144...Camera, 121...Arm, 201...Actuator, 202...Sensor, 203...Camera, 5...Control device, 501...Motion control unit, 502...Camera motion control unit, 503...Image processing unit, 504 … storage section

Claims

1. Substrate and, A finger portion connected to the base body, The system includes a sensor installed near the contact point of the finger, The aforementioned finger portion comprises an epidermal portion and a nail portion. The sensor is installed at a position between the skin portion and the claw portion. The sensor is installed at a position offset outward in the bending direction of the finger from the central axis connecting the finger and the base, from the center of the fingertip of the finger portion. Robot hand.

2. A substrate and, A finger portion connected to the base body, The system includes a sensor installed near the contact point of the finger, The aforementioned finger portion comprises an epidermal portion and a nail portion. The sensor is installed at a position between the skin portion and the nail portion, offset from the center of the fingertip portion of the finger portion in a direction perpendicular to the bending direction of the finger from the central axis connecting the finger portion and the base body. Robot hand.

3. A substrate and, A finger portion connected to the base body, The system includes a sensor installed near the contact point of the finger, The aforementioned finger portion comprises an epidermal portion and a nail portion. The sensor is installed at a position between the skin portion and the claw portion. The aforementioned finger portion consists of two or more parts. The sensors are installed near the contact points of at least two of the multiple finger parts. The aforementioned sensor is a camera, Stereoscopic vision is performed using images captured by sensors placed on the two fingers. Robot hand.

4. A substrate and, A finger portion connected to the base body, The system includes a sensor installed near the contact point of the finger, The aforementioned finger portion comprises an epidermal portion and a nail portion. The sensor is installed at a position between the skin portion and the claw portion. A method for operating a robot hand, Determine the target object, Determine the environmental target that will be used to manipulate the target object, Determine the gripping method for grasping the target object. The aforementioned sensor, the camera, captures images of the object to be grasped and the environment in which the object is being manipulated, while the object to be grasped is being held with the fingers. Based on the captured image, the position information of the target object is obtained. Based on the acquired position information, the amount of manipulation of the target object is determined. Robot hand control method.

5. A method for operating a robot hand according to any one of claims 1 to 3, Determine the target object, Determine the environmental target that will be used to manipulate the target object, Determine the gripping method for grasping the target object. The aforementioned sensor, the camera, captures images of the object to be grasped and the environment in which the object is being manipulated, while the object to be grasped is being held with the fingers. Based on the captured image, the position information of the target object is obtained. Based on the acquired position information, the amount of manipulation of the target object is determined. Robot hand control method.

6. The aforementioned robot hands are multiple, Of the aforementioned plurality of robot hands, at least two robot hands are equipped with cameras. The cameras provided by the at least two robot hands capture images of the same target object. The captured images are used to coordinate the control of at least two robot hands. The robot hand control method according to claim 4 or 5.

Images