Robot teaching through human demonstration

Abstract

To provide a method for teaching a robot to perform an operation based on human demonstration with images from a camera.SOLUTION: The method includes a teaching phase where a 2D or 3D camera detects a human hand grasping and moving a workpiece, and images of the hand and workpiece are analyzed to determine a robot gripper pose and positions which equate to the pose and positions of the hand and corresponding pose and positions of the workpiece. Robot programming commands are then generated from the computed gripper pose and position relative to the workpiece pose and position. In a replay phase, the camera identifies workpiece pose and position, and the programming commands cause the robot to move the gripper to pick, move and place the workpiece as demonstrated. A teleoperation mode is also disclosed, where camera images of a human hand are used to control movement of the robot in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of industrial robot programming, and more particularly to a method for programming a robot to perform workpiece pick, move and place operations, including a teaching phase in which a single camera detects the grasping and movement of a workpiece by a human hand, determines a grasp pose corresponding to the hand pose, and generates robot programming commands from the calculated grasp pose. [Background technology]

[0002] The use of industrial robots to perform a variety of repetitive manufacturing, assembly, and material transfer operations is well known. However, teaching a robot to perform even a fairly simple operation, such as picking a workpiece that is in an irregular position and orientation on a conveyor and transferring the workpiece to a bin or another conveyor, can be counterintuitive, time-consuming, and / or costly using conventional methods.

[0003] Robots have traditionally been taught to perform these types of pick-and-place operations by human operators using a teach pendant. The operator uses the teach pendant to instruct the robot to perform incremental moves, such as "nudge in the X direction" or "rotate the gripper around the local Z axis," until the robot and its gripper are correctly positioned and oriented to grasp the workpiece. The robot's configuration and the workpiece's location and orientation are then recorded by the robot controller and used for the "pick" operation. Nearly identical teach pendant commands are then used to define the "move" and "place" operations. However, using a teach pendant to program a robot is often counterintuitive, error-prone, and time-consuming, especially for non-expert operators.

[0004] Another known technique for teaching a robot to perform pick-and-place operations involves the use of a motion capture system. A motion capture system consists of multiple cameras positioned around a work cell to record the position and orientation of a human operator and a workpiece as the operator manipulates the workpiece. The operator and / or the workpiece may be affixed with uniquely recognizable marker dots to more accurately locate key locations on the operator and workpiece in the camera images as the operation is performed. However, this type of motion capture system is costly and difficult and time-consuming to precisely set up and configure so that the recorded positions are accurate.

[0005] In light of the above, there is a need for improved robot teaching techniques that are simple and intuitive for a human operator to implement. Summary of the Invention

[0006] In accordance with the teachings of the present disclosure, a method for teaching and controlling a robot to perform operations based on human demonstration using images from a single camera is described and illustrated. The method includes a teaching phase in which the single camera detects a human hand grasping and moving a workpiece, and the images of the hand and workpiece are analyzed to determine the pose and position of the robot's gripper corresponding to the pose and position of the hand, and the corresponding pose and position of the workpiece. Techniques are disclosed for using either a 2D or 3D camera. Robot programming commands are then generated from the calculated gripper pose and position relative to the workpiece pose and position. In a playback phase, the camera identifies the workpiece pose and position, and the programming commands cause the robot to move the gripper to pick, move, and place the workpiece as demonstrated by the human hand. Additionally, a teleoperation mode is disclosed in which the camera image of the human hand is used to control the robot's movements in real time.

[0007] Additional features of the presently disclosed apparatus and methods will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1] 1A-1C illustrate a method for analyzing an image of a human hand to determine the corresponding position and orientation of a fingered robotic grasper, according to one embodiment of the present disclosure. [Figure 2] FIG. 1 illustrates a method for analyzing an image of a human hand to determine the corresponding position and orientation of a magnetic or suction cup type robotic gripper, according to one embodiment of the present disclosure. [Figure 3] FIG. 1 is a diagram of a system and steps for teaching a robot to perform a pick-and-place operation using camera images of a human hand, according to one embodiment of the present disclosure. [Figure 4] FIG. 1 is a flowchart diagram of a method for teaching a robot to perform a pick-and-place operation using camera images of a human hand and a workpiece, according to one embodiment of the present disclosure. [Figure 5A] FIG. 1 is a diagram of one of four parts of a first step for detecting hand pose and determining hand size from 2D camera images, according to one embodiment of the present disclosure. [Figure 5B] FIG. 1 is a diagram of one of four parts of a first step for detecting hand pose and determining hand size from 2D camera images, according to one embodiment of the present disclosure. [Figure 5C] FIG. 1 is a diagram of one of four parts of a first step for detecting hand pose and determining hand size from 2D camera images, according to one embodiment of the present disclosure. [Figure 5D] FIG. 1 is a diagram of one of four parts of a first step for detecting hand pose and determining hand size from 2D camera images, according to one embodiment of the present disclosure. [Figure 6A] FIG. 10 is a diagram of a portion of a second step for detecting hand pose from 2D camera images using hand size data determined in the first step, according to an embodiment of the present disclosure. [Figure 6B] FIG. 10 is a diagram of a portion of a second step for detecting hand pose from 2D camera images using hand size data determined in the first step, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a diagram of a system and steps for a robot to perform a pick-and-place operation using camera images of a workpiece and programming previously taught by images of a human hand, according to one embodiment of the present disclosure. [Figure 8] FIG. 1 is a flowchart diagram of a method for a robot to perform a pick-and-place operation using camera images of a workpiece and programming previously taught by images of a human hand, according to one embodiment of the present disclosure. [Figure 9] FIG. 1 is a diagram of a system and steps for teleoperating a robot using camera images of a human hand and visual feedback via the human eye, according to one embodiment of the present disclosure. [Figure 10] FIG. 1 is a flowchart diagram of a method for teleoperating a robot using camera images of a human hand and visual feedback via the human eye, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following discussion of embodiments of the present disclosure directed to teaching a robot by human demonstration using a single camera is merely exemplary in nature and is in no way intended to limit the disclosed apparatus and techniques or their applications or uses.

[0010] The use of industrial robots for a variety of manufacturing, assembly, and material transfer operations is well known. One known type of robotic operation is sometimes known as "pick, move, and place." In this case, a robot picks a part or workpiece from a first location and moves and places the part at a second location. The first location is often a conveyor belt carrying randomly oriented parts, such as parts just removed from a mold. The second location may be another conveyor leading to a different operation or a shipping container, but in either case, the part must be placed at a specific location and oriented to a specific pose at the second location.

[0011] To perform the above types of pick, move, and place operations, a camera is typically used to determine the position and orientation of the incoming part and then the robot must be taught to grasp the part in a specific way using a finger-type gripper or a magnetic or suction-cup gripper. Teaching a robot how to grasp a part depending on its orientation has traditionally been performed by a human operator using a teach pendant. The operator uses the teach pendant to instruct the robot to perform incremental movements, such as a "nudge in the X direction" or a "rotate the gripper around the local Z axis," until the robot and its gripper are in the correct position and orientation to grasp the workpiece. The robot's configuration and the workpiece's position and orientation are then recorded by the robot controller and used for the "pick" operation. Nearly identical teach pendant commands are then used to define the "move" and "place" operations. However, using a teach pendant to program a robot is often unintuitive, error-prone, and time-consuming, especially for non-expert operators.

[0012] Another known technique for teaching a robot to perform pick, move, and place operations involves the use of a motion capture system. A motion capture system consists of multiple cameras positioned around a work cell to record the position and orientation of a human operator and workpiece as the operator manipulates the workpiece. The operator and / or workpiece may be affixed with uniquely recognizable marker dots to more accurately locate key locations on the operator and workpiece in the camera images as the operations are performed. However, this type of motion capture system is costly and difficult and time-consuming to precisely set up and configure to ensure the recorded positions are accurate.

[0013] The present disclosure overcomes the limitations of existing robot teaching methods by providing a technique that uses a single camera to capture images of a human performing the natural actions of grasping and moving a part, and analyzes the image of the human hand and its position relative to the part to generate robot programming commands.

[0014] 1 illustrates a method for analyzing an image of a human hand to determine the corresponding position and orientation of a fingered robotic grasper, according to one embodiment of the present disclosure. A hand 110 has a hand coordinate system 120 defined thereon. The hand 110 includes a thumb 112 with a thumb tip 114 and an index finger 116 with an index finger tip 118. Other points on the thumb 112 and index finger 116 may also be identified in the camera image, such as the locations of the bases of the thumb 112 and index finger 116 and the first knuckles of the thumb 112 and index finger 116.

[0015] Point 122 is located midway between the base of thumb 112 and the base of index finger 116. Here, point 122 is defined as the origin of hand coordinate system 120. The orientation of hand coordinate system 120 may be defined using any convention suitable for correlating with the orientation of the robotic grip. For example, the Y-axis of hand coordinate system 120 may be defined as being perpendicular to the plane of thumb 112 and index finger 116 (the plane is defined by points 114, 118, and 122). Thus, the X-axis and Z-axis lie in the plane of thumb 112 and index finger 116. Furthermore, the Z-axis may be defined as bisecting the angle formed by thumb 112 and index finger 116 (angle 114-122-118). The orientation of the X-axis may then be found from the known Y-axis and Z-axis using the right-hand rule. As noted above, the conventions defined here are merely exemplary, and other coordinate system orientations may be used instead. Importantly, the position and orientation of the coordinate system may be defined based on the primary recognizable points of the hand, and the position and orientation of the coordinate system can be related to the position and orientation of the robotic gripper.

[0016] A camera (not shown in FIG. 1 and discussed below) may be used to provide an image of the hand 110. The image can then be analyzed to determine the spatial locations (e.g., within a workcell coordinate system) of the thumb 112 and index finger 116, along with the knuckles, thumb tip 114 and index finger tip 118, and thus, the origin location 122 and orientation of the hand reference coordinate system 120. In FIG. 1, the location and orientation of the hand reference coordinate system 120 is relative to a gripper coordinate system 140 of a gripper 150 attached to a robot 160. The gripper coordinate system 140 has an origin 142 corresponding to the origin 122 of the hand reference coordinate system 120, and points 144 and 146 corresponding to the index finger tip 118 and thumb tip 114, respectively. Therefore, the two fingers of the finger-shaped gripper 150 lie in the XZ plane of the gripper coordinate system 140, with the Z axis bisecting the angle 146-142-144.

[0017] The origin 142 of the gripper coordinate system 140 is also defined as the tool center point of the robot 160. The tool center point is a point whose location and orientation is known to the robot controller, where the controller can provide command signals to the robot 160 to move the tool center point and its associated coordinate system (the gripper coordinate system 140) to a defined location and orientation.

[0018] Figure 2 illustrates a method for analyzing an image of a human hand to determine the corresponding position and orientation of a magnetic or suction cup type robotic gripper, according to one embodiment of the present disclosure. Figure 1 illustrates a method for relating hand pose to the orientation of a mechanical gripper with movable fingers, while Figure 2 illustrates a method for relating hand pose to a flat gripper (e.g., circular) that picks a part by its flat surface with either an attractive or magnetic force.

[0019] Again, hand 210 comprises thumb 212 and index finger 216. Point 214 is located where thumb 212 contacts part 220. Point 218 is located where index finger 216 contacts part 220. Point 230 is defined as being midway between points 214 and 218. Here, point 230 corresponds to tool center point (TCP) 240 of surface gripper 250 on robot 260. For the surface gripper 250 shown in FIG. 2 , the plane of gripper 250 may be defined as the plane that contains line 214-218 and is perpendicular to the plane of thumb 212 and index finger 216, based on knuckle and fingertip detection. Tool center point 240 of gripper 250 corresponds to point 230, as described above. This generally defines the location and orientation of surface gripper 250 relative to the position and pose of hand 210.

[0020] FIG. 3 illustrates a system 300 and steps for teaching a robot to perform a pick-and-place operation using camera images of a human hand, according to one embodiment of the present disclosure. The teaching steps illustrated in FIG. 3 are performed in a work cell 302. The work cell 302 includes a camera 310 for capturing images of the teaching steps. The camera 310 may be a three-dimensional (3D) camera or a two-dimensional (2D) camera, so long as it can identify coordinates of specific points and features of the hand, such as the fingertips, thumb tip, and knuckles discussed above. Techniques for using a 3D camera for hand pose detection and alternative techniques for using a 2D camera for hand pose detection are discussed below. The camera 310 is configured to provide an image of the portion of the work cell 302 where part movement activities (all of steps (1)-(3) discussed below) are being performed.

[0021] The camera 310 communicates with the robot controller 320. The controller 320 analyzes the images from the teaching step and generates robot programming commands, as discussed below. These robot programming commands are then used to control the robot's motion to perform pick-and-place operations during the reclaiming step. Alternatively, a separate computer may be provided between the camera 310 and the controller 320. This separate computer analyzes the camera images and transmits the gripper position to the controller 320. The teaching step in FIG. 3 consists of three steps. The picking step (step (1)) determines the position and pose (pose is an interchangeable term with orientation) of the workpiece and determines the position and pose of the hand gripping the workpiece. The moving step (step (2)) tracks the positions of the workpiece and the hand as the workpiece moves from the picking location to the placing location. The placing step (step (3)) determines the position and pose of the workpiece when it is placed at its destination (placement) location. The three steps are discussed in detail below.

[0022] In step (1) (removal), the camera images are used to identify the position and orientation of the workpiece 330. As the workpiece flows in the incoming direction on the conveyor, one coordinate (e.g., Z) of the workpiece 330's position is tracked according to the conveyor position index. The workpiece 330 shown in FIG. 3 is a simple cube. However, any type or shape of workpiece 330 may be identified from the camera images. For example, if the workpiece 330 is known to be a particular injection-molded part, which may have any irregular orientation and position on the conveyor, the images from the camera 310 can be analyzed to identify key features of the workpiece 330, and the position and orientation of the workpiece coordinate system 332 can be determined from such features.

[0023] Additionally, in step (1) (retrieval), the camera image is also used to identify the position and orientation of the hand 340 as it grasps the workpiece 330. The image of the hand 340 is analyzed to determine the position and orientation of the hand coordinate system 342, in the manner discussed above with respect to FIG. 1 and described in detail below (using two different techniques). The hand 340 approaches the workpiece 330 in the direction indicated by arrow 344. When the tip of the thumb contacts the workpiece 330, as indicated by point 350, and the fingertips contact the workpiece 330 (this contact point is not visible), the controller 320 saves all data from this particular image as retrieved data. The retrieved data includes the position and orientation of the workpiece 330 as defined by its workpiece coordinate system 332 and the position and orientation of the hand 340 as defined by its hand coordinate system 342.

[0024] Step (2) (movement) uses camera images to track the positions of both the workpiece 330 and the hand 340 as they move along a path 360. Multiple images of the workpiece 330 and the hand 340 are recorded to define the path 360, which may not be a straight line. For example, the path 360 may include a long, extensive curve, or the path 360 may involve moving the workpiece 330 upward over some kind of barrier. In either case, the path 360 includes multiple points that define the position (and possibly orientation) of the workpiece 330 and the hand 340 during the movement step. The same techniques discussed previously are used to identify the position and pose of the workpiece coordinate system 332 and the hand coordinate system 342 from the camera images. Techniques for determining hand pose are also discussed in more detail below.

[0025] In step (3) (Placement), the camera image is used to identify the final position and orientation of the workpiece 330 (as defined by its workpiece coordinate system 332) after it has been placed in the desired location, as indicated by arrow 370. The same techniques discussed above are used to identify the position and orientation of the workpiece coordinate system 332 from the camera image. When the thumb tip and fingertips break contact with the workpiece 330, the controller 320 saves the workpiece coordinate system data from this particular image as placement data. Placement data may also be recorded and saved based on the workpiece 330 stopping movement; that is, the workpiece coordinate system 332 remains in the exact same position and orientation for a period of time (e.g., 1-2 seconds).

[0026] 4 is a flowchart 400 of a method for teaching a robot to perform a pick-and-place operation using camera images of a human hand and workpiece, according to one embodiment of the present disclosure. Flowchart 400 is arranged in three vertical columns corresponding to the pick step (right side), the move step (center), and the place step (left side), as shown in FIG. 3 . The pick step begins at start box 402. In box 404, workpiece 330 and hand 340 are detected in an image from camera 310. Analysis of the images is performed in controller 320 or a separate computer in communication with controller 320. The position and orientation of workpiece coordinate system 332 is determined from analysis of the image of workpiece 330, and the position and orientation of hand coordinate system 342 is determined from analysis of the image of hand 340.

[0027] In decision diamond 406, it is determined whether the fingertips (thumb tip 114 and index finger tip 118 in FIG. 1 ) have contacted the workpiece 330. This is determined from the camera image. Once the fingertips have contacted the workpiece 330, the gripping pose and position of the workpiece 330 and hand 340 are recorded by controller 320 in box 408. It is important that the pose and position of the hand 340 relative to the workpiece 330 be identified. That is, the position and orientation of hand coordinate system 342 and workpiece coordinate system 332 must be defined relative to some global fixed reference coordinate system, such as the workcell coordinate system. This allows controller 320 to determine how to position the gripper to grip the workpiece in a later playback phase, as discussed below.

[0028] After the gripping pose and position of workpiece 330 and hand 340 are recorded by controller 320 in box 408, the pick step ends in end box 410. The process then proceeds to the transfer step beginning in box 422. In box 424, workpiece 330 is detected in the camera image. In decision diamond 426, if workpiece 330 is not detected in the camera image, the process loops back to box 424 to take another image. Once workpiece 330 is detected in the camera image, the position (and possibly pose) of the workpiece is recorded by controller 320 in box 428.

[0029] In box 434, the hand 340 is detected in the camera image. In decision diamond 436, if the hand 340 is not detected in the camera image, the process loops back to box 434 to capture another image. Once the hand 340 is detected in the camera image, the hand position (and possibly pose) is recorded by the controller 320 in box 438. Once both the workpiece position (from box 428) and the hand position (from box 438) are detected and recorded from the same camera image, the hand position and workpiece position are combined and recorded in box 440. Combining the hand position and workpiece position may be achieved by simply averaging the two. For example, if the midpoint between the thumb tip 114 and the index finger tip 118 needs to coincide with the center / origin of the workpiece 330, the average location can be calculated between the midpoint and the center of the workpiece.

[0030] Preferably, multiple positions along the move step are recorded to define a smooth move path by repeating the activity from start move box 422 through combine hand and work position box 440. After the hand and work position are combined and recorded in box 440 and no more move step positions are needed, the move step ends in end box 442. The process then proceeds to the placement step beginning in box 462.

[0031] In box 464, the position of workpiece 330 is detected in the image from camera 310. As with all of the image analysis steps, the analysis of the image in box 464 may be performed on controller 320, or on a graphics processor connected to controller 320, or an intermediate computer, if desired. In decision diamond 466, it is determined whether workpiece 330 is present in the camera image and whether workpiece 330 is stationary. Alternatively, it may be determined whether the fingertips have lost contact with workpiece 330. When workpiece 330 is determined to be stationary and / or when the fingertips have lost contact with workpiece 330, the desired pose and position of workpiece 330 is recorded by controller 320 in box 468. The placement step, and the entire teaching phase, concludes in end box 470.

[0032] Throughout the foregoing discussion of Figures 1-4, much reference has been made to the concept of detecting hand pose (the position of the thumb and index finger pivot points) from a camera image and defining a hand coordinate system from the hand pose. To define the origin and axis orientation of a hand coordinate system (such as hand coordinate system 120 in Figure 1), it is necessary to determine the three-dimensional (X, Y, and Z) coordinates of the hand pivot points. Two different techniques for determining the 3D coordinates of the hand pivot points are described in the following discussion. Either of these hand pose detection techniques may be used in an overall method for teaching a robot from human demonstrations.

[0033] One embodiment of hand pose detection uses a three-dimensional (3D) camera to directly detect the 3D coordinates of the hand's key points. The 3D camera (which could be camera 310 in FIG. 3) not only provides an image that allows key points to be identified within the image plane, but can also detect depth (distance perpendicular to the image plane). Some 3D cameras use two or more lenses, or even a single lens that shifts its position to record multiple viewpoints, where combining the two viewpoints enables the perception of depth. Other 3D cameras use range imaging to determine the range (distance) from the camera's lens to various points in the image. Range imaging may be achieved using time-of-flight or other techniques.

[0034] Data directly available from the 3D camera may be used to determine the X / Y / Z coordinates of the hand pivot points (such as points 112 and 118 in FIG. 1 ). From the X / Y / Z coordinates of the hand pivot points, the origin and orientation of the hand coordinate system may be calculated globally, as previously discussed. Robot motion commands may then be defined based on the correspondence between the hand and gripper coordinate systems, as previously discussed.

[0035] Another embodiment of hand pose detection may use a two-dimensional (2D) camera to detect the 3D coordinates of the hand's key points using a two-step process. In some applications, it may be advantageous to use a 2D camera for robot teaching by human demonstration, since 2D cameras are more widely available and cheaper than 3D cameras, and 2D cameras have much faster image frame rates.

[0036] Existing techniques for determining the X / Y / Z coordinates of key points on a hand from 2D images can be unreliable. These methods typically rely on deep learning image analysis techniques using databases containing both real and synthetic hand images. However, hand size varies greatly from person to person, which can lead to inaccuracies in these existing methods.

[0037] To overcome the limitations of existing methods, we present a new technique for detecting hand pose from 2D camera images, where one preliminary step of measuring hand size is performed, and then the 2D camera images during robot teaching can be accurately analyzed for the 3D coordinates of the hand's key points.

[0038] FIG. 5A is a diagram of a first portion of a first step for detecting hand poses from 2D camera images, according to one embodiment of the present disclosure. In this first portion of the first step, referred to as step 1A, a camera 510 is used to capture images of an operator's hand 520, which will later teach a robot program via human demonstration. The camera 520 may be the same as the camera 310 in FIG. 3. The hand 520 is placed on a grid 530 of ArUco markers, and a photograph (digital image) is taken in step 1A. A traditional chessboard for camera calibration is another option that can be used in place of the ArUco markers. The digital image is analyzed in the remainder of step 1 to determine the size of the individual elements or segments of the hand 520. Step 1 is performed only once for each operator's hand. Step 2 (robot path and process teaching via human hand demonstration) can be performed multiple times for the hand 520 without having to repeat step 1.

[0039] The process of step 1 is based on finding correspondences between points in the real / physical environment and their 2D image projections. To facilitate this step, synthetic or fiducial markers are used. One approach involves using binary square fiducial markers, such as ArUco markers. The main advantage of such markers is that a single marker provides sufficient correspondences (its four corners) to obtain the camera pose relative to the plane of the marker. Using many markers in the grid 530 provides enough data to determine the actual size of each finger segment of the hand 520.

[0040] An ArUco marker is a square visual marker consisting of a wide black border and an inner binary matrix that determines its identity. The black border facilitates fast detection within an image, and the binarization allows for its identification and the application of error detection and correction techniques. Grid 530 in FIG. 5A includes a 5×7 array of ArUco markers, including individual markers 532, 534, and 536. Each ArUco marker within grid 530 is unique. The 5×7 array used in grid 530 is sized to facilitate visual measurement of a human hand and is merely illustrative. The grid 530 of ArUco markers is used to provide sufficient information “bandwidth” in the digital image to allow both calibration of the image plane to the marker plane and determination of the coordinates within the marker plane of multiple key points on the hand 520, as discussed below.

[0041] The image from step 1A is provided to both step 1B (FIG. 5B) and step 1C (FIG. 5C), with the hand 520 resting flat on a grid 530. Here, two different types of image analysis are performed.

[0042] Figure 5B is a diagram of a second portion of the first step for detecting hand pose from 2D camera images, according to one embodiment of the present disclosure. At box 540, Figure 5B depicts step 1B of the hand size determination process. In step 1B, analysis of the image from step 1A is performed to determine rotational and translational transformations from image (virtual) coordinates to marker (physical) coordinates. Either the grid 530, or the individual markers (532, 534, etc.), are defined as having a marker coordinate system 538 with local X and Y axes in the plane of the marker, and a local Z axis perpendicular to the marker, pointing upward as shown.

[0043] The camera 510 (from FIG. 5A) has a camera coordinate system 518 defined as fixed to itself, with the local X and Y axes in the screen or image plane, and the local Z axis normal to the screen and oriented in the direction of the camera's field of view (towards the marker grid 530). Note that with this technique, the camera does not need to be precisely aligned with the camera image plane parallel to the marker plane. The transformation calculations will handle rotational and translational misalignment. A point in the idealized image in the camera coordinate system 518 has coordinates (x c ,y c ) The transformation from the marker coordinate system 538 to the camera coordinate system 518 is defined by a rotation vector R and a translation vector t.

[0044] We define the actual observed image coordinate system 548 as having local X and Y axes in the plane of the actual observed image, where the local Z axis is irrelevant because the image is two-dimensional. The actual observed image coordinate system 548 differs from the camera coordinate system 518 due to image distortion. A point in the actual image in the actual observed image coordinate system 548 has coordinates (x d ,y d ) The transformation from the camera coordinate system 518 to the actual observed image coordinate system 548 is defined by the intrinsic camera parameter matrix k. The camera intrinsic parameters include focal length, image sensor type, and principal point.

[0045] 5C is a diagram of a third portion of the first step for detecting hand pose from a 2D camera image, according to one embodiment of the present disclosure. At box 550, FIG. 5C depicts step 1C of the hand size determination process. In step 1C, analysis of the image from step 1A is performed to identify key points of the hand 520 within the image.

[0046] A convolutional layer 560 is used to analyze the image to identify and locate key features of the hand 520. Convolutional layers, as known in the art, are a class of deep neural networks commonly applied to analyzing visual images. The convolutional layer 560 is specially trained to identify the structural topology of a human hand based on factors including known hand anatomy and proportions, visual cues such as the flexion and bulge of knuckles, and more. In particular, the convolutional layer 560 can identify and locate points in the image, including fingertips and knuckles.

[0047] The output of convolutional layer 560 is shown in box 570. The hand 520 and grid 530 of ArUco markers can be seen in the image within box 570. Also superimposed on the image is the topology of the hand 520, including the bones in the hand 520 and the joints between the bones, represented as points. For example, the tip of the thumb is identified as point 572, and the tip of the index finger is identified as point 574. Points 572 and 574, as well as all of the points shown in the topology within box 570, have known locations in image coordinates, specifically the coordinates of the actual observed image coordinate system 548. This is because convolutional layer 560 is applied to the image from step 1A, which has not yet been rectified or transformed in any way.

[0048] 5D is a diagram of a fourth portion of the first step for detecting hand pose from 2D camera images, according to one embodiment of the present disclosure. The transformation parameters k, R, and t from step 1B (FIG. 5B) and the key points of the hand 520 in image coordinates from step 1C (FIG. 5C) are both provided to step 1D, where calculations are performed to determine the true distances between the key points of the hand 520.

[0049] The calculations performed to generate the topology of the hand 520 in true world / marker coordinates are defined as in equation (1).

[0050]

number

[0051] where X d is the set of hand points in real image coordinates from step 1B, and X w is the currently computed set of hand key points in world / marker coordinates, and k, R and t are as defined above.

[0052] Using equation (1), the key point X of the hand w The set of move points X shown in box 580 in Step 1D can be calculated. w is in world coordinates and is calibrated by a number of ArUco markers in the same coordinate system.

[0053] Thus, the true distance between point 582 (the tip of the thumb) and point 584 (the knuckle closest to the tip of the thumb) can be calculated as the square root of the sum of the squares of the differences in coordinates of points 582 and 584. This distance, shown at 590, is the true length of the outer segment (bone) of the thumb. Similarly, the distance between points 586 and 588 can be calculated as shown at 592, and this distance is the true length of the outer segment of the index finger. The true length of each bone (each segment of each finger) in hand 520 can be calculated in this manner.

[0054] Step 1 of the method for detecting hand pose from 2D camera images discussed above provides the true lengths of each segment of each finger in hand 520. Hand 520 can then be used for robot teaching, with images from the 2D camera being analyzed to determine the 3D locations of key points, according to step 2 of the method discussed below.

[0055] 6A is a diagram of a first portion of a second step for detecting hand poses from 2D camera images, according to one embodiment of the present disclosure. In this first portion of the second step, a camera image of a hand 520 in an arbitrary pose, such as grasping an object, during robot teaching is analyzed to identify key points as discussed above. In box 610, an image of the hand 520 is provided. The image can be provided by camera 510 or another camera. In box 620, image analysis is performed using neural network convolutional layers to identify key points on the hand 520, in the manner discussed above. Because the hand is typically not placed flat on the surface of the image from box 610, portions of the hand may be occluded (not visible in the image). Furthermore, the finger segments in the image from box 610 will generally not be at their true length, as will be discussed later.

[0056] In box 630, any recognizable pivot points on the hand 520 are identified, along with their locations in screen coordinates. For example, pivot point 632 on the tip of the index finger may be identified by finding the central tip of the finger proximal to the thumb. Pivot point 634 on the outer knuckle of the index finger may be identified as being between the knuckle's side points 636 and 638, which are identifiable in the image. The neural network convolutional layer identifies points based on its past training, for example, by looking for bulges and bends in the finger image. Other recognizable pivot points are similarly identified, for example, pivot point 640 on the middle knuckle of the index finger, which is between the side points 642 and 644.

[0057] Next, the screen coordinates of each of the finger segments (individual bones) in hand 520 can be determined. The outer segment 652 of the index finger is determined as a line extending from pivot 632 to pivot 634 in screen coordinates of the image from box 610. Similarly, the middle segment 654 of the index finger is determined as a line extending from pivot 634 to pivot 640, and the inner segment 656 of the index finger is determined as a line extending from pivot 640 to pivot 646.

[0058] The diagram in box 630 depicts hand 520 in a flat position simply to clearly show all of the various gist points. Gist points may be identified if they are visible in the image from box 610. In fact, box 630 involves identifying and locating gist points of hand 520 (e.g., in a grasping pose) in an image that looks exactly like the image in box 610. Hand 520 is not flattened or distorted in any way from the camera image in box 610 to the gist point identification in box 630.

[0059] The location and identification of the screen coordinates of all visible and recognizable features and finger segments (finger bones) of the hand 520 are provided to the additional analysis step of Figure 6B, where the 3D pose of the hand 520 will ultimately be determined.

[0060] Figure 6B is a diagram of a second portion of the second step for detecting hand pose from 2D camera images, according to one embodiment of the present disclosure. In box 660, the hand size from step 1 (Figure 5D) is provided as an input. Specifically, the true lengths of each bone (each segment of each finger) in hand 520 are provided, as discussed in detail above. In addition, the identification and location in screen coordinates of each recognizable key point in hand 520 in the pose from box 630 in Figure 6A are provided as inputs on the left. Using these inputs, a series of Perspective-n-Point calculations are performed, as shown at 662.

[0061] Perspective-n-Point (PnP) is the problem of estimating the pose of a calibrated camera given a set of n 3D points in the world and their corresponding 2D projections in an image. The camera pose consists of six degrees of freedom (DOF), consisting of the camera's rotation (roll, pitch, and yaw) and 3D translation relative to the "world" or workcell coordinate system. A frequently used solution to this problem is for n=3, called P3P, although many solutions are available for the general case of n≧3. Because at least 4-6 key points on each segment of the index finger and thumb will be identified in box 630 of FIG. 6A (along with possibly several other bones in other fingers), more than enough points exist for the PnP calculation in 662.

[0062] The identification and location in screen coordinates of the pivot point on the outer segment 652 of the index finger is provided by box 630 of FIG. 6A , as shown. Similarly, the identification and location in screen coordinates of the pivot points on the middle segment 654 of the index finger and the inner segment 656 of the index finger are provided. The true lengths of each of the finger segments (652, 654, 656) are provided by box 660. Next, for each of the finger segments (652, 654, 656), the PnP problem is solved by box 664. Solving the PnP problem provides the pose of the camera 510 for each of the finger segments of the hand 520. Since the pose of the camera 510 in world or workcell coordinates is known, the pose of the finger segments of the hand 520 in world or workcell coordinates can be calculated. In this way, the 3D poses of the finger segments (652, 654, 656) are obtained by box 666. In box 668, the 3D poses of the individual segments 652, 654, and 656 are combined to obtain the 3D pose of the entire index finger.

[0063] In box 672, identification and location in screen coordinates of the key points on the thumb segment and any other visible finger segments are provided from box 630 of FIG. 6A. In box 674, the PnP problem is solved for each individual finger segment using the key point screen coordinates and the finger segment's true length as input, as discussed above. In box 676, the 3D pose of each finger segment visible in the image from box 610 is obtained. The thumb segments are combined in box 678 to obtain the 3D pose of the entire thumb and any other similarly visible fingers.

[0064] In box 680, the 3D poses of the fingers and thumb from boxes 668 and 678 are combined to obtain the 3D pose of the entire hand 520. In box 690, the final output of the method and calculations of step 2 is shown. This includes the position and orientation of the camera 510, represented as a pyramid shape 692, and the pose of the hand 520, represented as a wireframe 694. The pose of the hand 520 includes the 3D localization in world coordinates (or workcell coordinate system) of each distinct finger segment (bone) on the hand 520. This is sufficient to define the hand coordinate system 120, as shown in FIG. 1, as long as the thumb and index finger are identified.

[0065] The second step of the method for detecting hand pose from 2D camera images discussed above with reference to Figures 6A and 6B is repeatedly and continuously performed during the entire process of robot teaching by human demonstration. That is, referring again to Figures 3 and 4, when a 2D camera is used to detect a hand 520 grasping and moving a workpiece, the hand pose at each intermediate position is determined using the Step 2 sequence of Figures 6A and 6B. In contrast, Step 1, which determines the hand size (the true length of each finger segment, Figures 5A-5D), is performed only once, prior to hand pose detection for robot teaching.

[0066] The 2D camera image analysis steps to determine hand size (FIGS. 5A-5D) and to determine hand pose from the 3D coordinates of the hand's key points (FIGS. 6A and 6B) may be performed in a robot controller, such as controller 320 of FIG. 3, or in a separate computer in communication with controller 320.

[0067] 7 is a diagram of a system 700 and steps for a robot to perform a pick-and-place operation using camera images of a workpiece and programming previously taught by images of a human hand, according to one embodiment of the present disclosure. System 700 is located within a work cell 702 that includes a camera 740 in communication with a controller 720. Such components may or may not be the same as work cell 302, camera 310, and controller 320 previously shown in FIG. 3. In addition to camera 740 and controller 720, work cell 702 includes a robot 710 that communicates with controller 720, typically via a physical cable 712.

[0068] System 700 is designed to "play back" pick, move, and place operations taught by a human operator within system 300. The hand and workpiece data recorded in the pick, move, and place steps of flowchart 400 are used to generate robot programming instructions as follows: Robot 710 positions gripper 720 in a home position, as known to those skilled in the art. Camera 740 identifies the location and orientation of a new workpiece 730 that may be placed on the incoming conveyor. From box 408 of the pick step, controller 320 (FIG. 3) knows the position and orientation of hand 340 relative to workpiece 730 (and thus, from FIG. 1, the position and orientation of gripper 720) to properly grip workpiece 730. Path 1 is calculated as the path for moving gripper 720 from the home position to a pick position calculated based on the position and orientation of workpiece 730. The pick operation is performed at the end of Path 1. Gripper 720A is shown along path 1 near the pick-up position, and workpiece 730A is shown at the pick-up position.

[0069] From box 440 of the movement step (FIG. 4), controller 320 knows the position of workpiece 330 (and therefore workpiece 730) at multiple locations along the movement. Path 2 is calculated as the path that moves gripper 720A and workpiece 730A from the pick position along the movement path. In FIG. 7, one of the intermediate positions along the movement path shows gripper 720B.

[0070] Path 2 terminates at the placement position recorded in box 468, which includes both the position and posture (orientation) of workpiece 330 corresponding to workpiece 730C. After gripper 720 places workpiece 730C in the placement position and orientation, gripper 720 releases workpiece 730C and returns to its home position via path 3.

[0071] 8 is a flowchart diagram 800 of a method for a robot to perform a pick-and-place operation using camera images of a workpiece and programming previously taught by images of a human hand, according to one embodiment of the present disclosure. In box 802, data from a human demonstration of the pick-and-place operation is provided to the robot controller 720, as discussed in detail above. This data can be obtained from a 2D or 3D camera and includes pick, move, and place motion commands directed by the human hand. In box 804, images from the camera 740 are analyzed by the controller 720 to identify the location and orientation (position and pose) of the workpiece 730. In box 606, the controller 720 provides motion commands to the robot 710 to move the gripper 720 from its home position and grasp the workpiece 730. The motion commands are calculated by controller 720 based on the position and orientation of workpiece 730 known from analysis of the camera images in box 806 and the relative position and orientation of gripper 720 with respect to workpiece 730 known from human demonstration of the pick operation (Figures 3 and 4).

[0072] In box 808, in response to instructions from the controller, the robot 710 moves the gripper 720 holding the workpiece 730 along a travel path as taught during human demonstration. The travel path may include one or more intermediate points between the pick position and the place position, so that the travel path may track any three-dimensional curve. In box 810, the robot 710 places the workpiece 730 at the position and orientation taught during the human demonstration of the place operation. The workpiece placement position and orientation are known from the human demonstration of the place operation, and the relative position and orientation of the gripper 720 with respect to the workpiece 730 are known from the human demonstration of the pick operation. In box 812, the robot 710 returns to its home position in preparation for receiving instructions to pick the next workpiece.

[0073] The following is a summary of the disclosed techniques for robot programming by human demonstration, discussed in detail above. · Demonstrate gripping, moving and placing of workpieces using the human hand. Analyzes camera images of the hand and workpiece to determine the appropriate posture and position of the robot gripper relative to the workpiece based on the hand's key points. The hand features may be determined from images from a 3D camera or from images from a 2D camera with a preliminary step of determining the size of the hand. Generate robot motion commands based on camera images of the new workpiece in the regeneration phase using the appropriate pose and position of the robot gripper relative to the workpiece. This involves first grasping a new workpiece, then moving and placing the new workpiece based on the movement and placement data acquired during the human demonstration.

[0074] The preceding discussion describes an embodiment of teaching a robot by human demonstration, where a teach phase is performed upfront and the human is no longer in the loop during a play phase, where the robot uses camera images of the incoming workpiece to perform the operation in a production environment. The pick-and-place operation described above is just one example of teaching a robot by human demonstration. Other robot operations may be taught in the same manner, where camera images of a hand are used to determine the relative position of a gripper or other tool with respect to the workpiece. Another embodiment is disclosed below, where a human remains in the loop during production operations, but where camera images of the incoming workpiece are not required.

[0075] FIG. 9 is a diagram of a system 900 and steps for remote operation of a robot using a camera image of a human hand and visual feedback via the human eye, according to an embodiment of the present disclosure. A human operator 910 is positioned so that a camera 920 can capture an image of the operator's hand. The camera 920 may be a 2D or 3D camera. The camera 920 provides images to a computer 930, which analyzes the images and identifies key features of the hand, as described in detail with reference to FIGS. 1 and 2, as well as FIGS. 5 and 6 (for the 2D camera embodiment). From the analysis of the images, the position and orientation of the hand coordinate system, and therefore the grip coordinate system, can be determined. The position and orientation of the grip coordinate system are provided to a robot controller 940. In a slight variation of this embodiment, the computer 930 may be eliminated, and the controller 940 may perform the image analysis function to determine the position and orientation of the grip coordinate system.

[0076] The controller 940 is in communication with the robot 950. The controller 940 calculates robot motion commands to cause the robot 950 to move its gripper 960 to a position and orientation in the gripper coordinate system identified from the image. The robot 950 responds to commands from the controller 940, which in turn moves the gripper 960 in response to the hand position from the camera image. Optionally, a transposition may be included between the hand pose and the gripper pose. For example, the gripper pose and movement may mirror the hand pose and movement. The situation in FIG. 9 is that the gripper 960 is to grasp the workpiece 970 and perform some operation on the workpiece 970, such as moving the workpiece 970 to a different position and / or pose. While the gripper 960 is shown as a finger-type gripper, it may alternatively be a suction cup or magnetic surface gripper, as previously described.

[0077] A human operator 910 is positioned to view the robot 950, gripper 960, and workpiece 970. The operator 910 may be in the same work cell as the robot 950 or in a separate room separated by a windowed wall. The operator 910 may be remote or otherwise visually obstructed from the robot 950. In that case, real-time video images of the robot 950, gripper 960, and workpiece 970 would be provided to the operator 910 as a means of visual feedback.

[0078] System 900 does not use a camera to provide an image of workpiece 970. Instead, operator 910 monitors the movement of gripper 960 and moves his / her hand to cause gripper 960 to perform an operation on workpiece 970. In one possible situation, operator 910 moves his / her hand toward workpiece 970 with index finger and thumb spread apart, continues moving his / her hand to bring gripper 960 closer to workpiece 970 from a desired orientation, grasps workpiece 970 between his / her index finger and thumb, and then moves workpiece 970 to another position and / or orientation while holding it.

[0079] In system 700, operator 910 moves his / her hands based on visual feedback of the gripper / workpiece scene, camera 920 provides continuous images of the hands, and controller 940 moves gripper 960 based on the hand movements as analyzed in the camera images.

[0080] Figure 10 is a flowchart diagram 1000 of a method for teleoperation of a robot using camera images of a human hand and visual feedback via the human eye, according to one embodiment of the present disclosure. The method of Figure 10 uses system 900 of Figure 9. The method begins at start box 1002. At box 1004, a human hand is detected in the camera image. At decision diamond 1006, if a hand is not detected, the process loops back to box 1004 to continue taking images. At box 1008, if a hand is successfully detected in the camera image, computer 930 calculates the position and pose of gripper 960 based on the position and pose of the hand as determined by hand feature identification.

[0081] In box 1010, the gripper position and pose are transferred from computer 930 to controller 940. Alternatively, image analysis and calculation of gripper position and pose may be performed directly on controller 940, in which case box 1010 is eliminated. In box 1012, controller 940 provides movement commands to robot 950 to move gripper 960 to a new position based on the most recently analyzed hand image. In box 1014, human operator 910 visually confirms gripper position and pose. In decision diamond 1016, operator 910 determines whether the target position of gripper 960 has been achieved. If not, operator 910 moves his hand in box 1018, causing gripper 960 to make a corresponding movement via closed-loop feedback to the new camera image. Once the target position is reached, the process ends at terminal 1020.

[0082] In actual practice, the operator 910 may perform a continuous sequence of hand movements to control the robot via teleoperation. For example, the operator 910 may move his or her hand to move the gripper 960 toward the workpiece 970, grasp the workpiece 970, move the workpiece 970 to a desired location, place and release the workpiece 970, move the gripper 960 back to the starting location, and approach a new workpiece.

[0083] Various computers and controllers have been described or alluded to throughout the preceding discussion. It should be understood that the software applications and modules of such computers and controllers execute on one or more computing devices having a processor and memory modules. In particular, this includes the processors of the robot controllers 320, 720, and 940 discussed above, as well as the computer 930. Specifically, the processors of the controllers 320, 720, and 940 and the computer 930 are configured to perform robot teaching via human demonstration in the manner discussed above.

[0084] As outlined above, the disclosed techniques for teaching robots by human demonstration make robot motion programming faster, easier, and more intuitive than prior techniques, while offering the simplicity of requiring only a single camera.

[0085] While several exemplary aspects and embodiments of robotic teaching by human demonstration have been discussed above, those skilled in the art will recognize modifications, permutations, additions, and subcombinations thereof. Therefore, it is intended that the following appended claims and the claims introduced below be interpreted to include any and all such modifications, permutations, additions, and subcombinations as are within their true spirit and scope.

Claims

1. 1. A method for programming a robot to perform an operation via human demonstration, the method comprising: demonstrating the operation on a workpiece by a human hand; A step of analyzing a camera image of the hand demonstrating the operation on the workpiece by a computer to create demonstration data; analyzing camera images of a new workpiece to determine an initial position and orientation of the new workpiece; generating robot motion commands based on the demonstration data and the initial position and orientation of the new workpiece to cause the robot to perform the operation on the new workpiece; performing the operation on the new workpiece by the robot; the step of demonstrating the operation on the workpiece by the human hand and the step of performing the operation on the new workpiece by the robot are both performed in a robot work cell, and the camera images are taken by a single camera; the camera is a three-dimensional camera that directly acquires an image and X, Y, and Z coordinates of a plurality of identifiable points on the hand within the image; analyzing a camera image of the hand demonstrating the manipulation includes identifying locations of a plurality of points on the hand, including the tips of each of the thumb and index finger, the base knuckles, and other knuckles; the demonstration data includes positions and orientations of a hand coordinate system, a gripping unit coordinate system corresponding to the hand coordinate system, and a workpiece coordinate system in a gripping step of the operation; the hand coordinate system has an origin at a midpoint between the proximal knuckles of the thumb and index finger, a Z axis passing through a midpoint between the tips of the thumb and index finger, and a Y axis perpendicular to a plane containing the thumb and index finger.

2. The method of claim 1 , wherein the demonstration data further includes positions of the hand coordinate system and the work coordinate system for intermediate steps of the operation, and a position and orientation of the work coordinate system for a final step of the operation.

3. The method of claim 1 , wherein the new workpiece is placed on a conveyor prior to the manipulation by the robot, and the initial position of the new workpiece is a function of a conveyor position index.

4. 2. The method of claim 1, wherein generating robot motion commands includes generating, by a robot controller having a processor and memory, commands to move the robot gripper to a gripping position and orientation based on the initial position and orientation of the new workpiece and a position and orientation of the robot gripper relative to the workpiece included in the demonstration data.

5. 5. The method of claim 4, wherein generating a robot motion command further comprises generating a command to the robot gripper to move the new workpiece from the gripping position to another position included in the demonstration data.

6. The method of claim 4 , wherein the robotic gripper is a finger gripper or a surface gripper that uses suction or magnetic forces.

7. 1. A system for programming a robot to perform an operation via human demonstration, the system comprising: A camera and Industrial robots and A robot controller having a processor and a memory, the controller communicating with the robot and receiving images from the camera, the controller comprising: analyzing camera images of a human hand demonstrating the operation on a workpiece to generate demonstration data; analyzing camera images of a new workpiece to determine an initial position and orientation of the new workpiece; generating robot motion commands that cause the robot to perform the operation on the new workpiece based on the demonstration data and the initial position and orientation of the new workpiece; performing the operation on the new workpiece by the robot; and a robot controller configured to perform steps including: The camera is a three-dimensional camera that directly captures images and the X, Y, and Z coordinates of a plurality of key points on the hand, respectively, in the images. analyzing a camera image of the hand demonstrating the manipulation includes identifying locations of the plurality of key points on the hand, including the tips of the thumb and index finger, the base knuckles, and other knuckles; the demonstration data includes positions and orientations of a hand coordinate system, a gripping unit coordinate system corresponding to the hand coordinate system, and a workpiece coordinate system in a gripping step of the operation; The hand coordinate system has an origin at the midpoint between the proximal knuckles of the thumb and index finger, a Z axis passing through the midpoint between the tips of the thumb and index finger, and a Y axis perpendicular to a plane including the thumb and index finger. system.

8. 8. The system of claim 7, wherein the demonstration data further includes positions of the hand coordinate system and the work coordinate system for intermediate steps of the operation, and a position and orientation of the work coordinate system for a final step of the operation.

9. 8. The system of claim 7, further comprising a conveyor on which the new workpiece is placed prior to the manipulation by the robot, and wherein the initial position of the new workpiece is a function of a conveyor position index.

10. 8. The system of claim 7, wherein the step of generating robot motion commands includes the steps of: generating a command to move the robot gripper to a gripping position and orientation based on the initial position and orientation of the new workpiece and a position and orientation of the robot gripper relative to the workpiece included in the demonstration data; and generating a command to move the robot gripper to move the new workpiece from the gripping position to another position included in the demonstration data.

11. The system of claim 10 , wherein the robotic gripper is a finger gripper or a surface gripper that uses suction or magnetic forces.

Images