Two-Hand Detection in the Teaching According to the Demonstration

Abstract

A method for performing two - hand detection in robot teaching based on human demonstrations. The camera images of the demonstrator's hands and the workpiece are provided to a first neural network, which determines the identities of the left and right hands of the human demonstrator based on the images and also provides cropped sub - images of the recognized hands. The first neural network is trained using images in which the left and right hands are pre - recognized. Then, the cropped sub - images are provided to a second neural network, which detects the poses of both the left and right hands based on the images, where, if the second neural network is trained using the right - hand images, the sub - image of the left hand is horizontally flipped before and after hand - pose detection. The hand - pose data is converted into robot - gripper pose data and used to teach the robot to perform operations through human demonstrations.

Description

Technical Field

[0001] The present disclosure relates to the field of industrial robot programming, and more particularly, to a method for determining the identity of the left and right hands of a human demonstrator based on a camera image, and then detecting the postures of both the left and right hands according to the image, wherein the hand identity and posture data are used to teach or program a robot to perform operations through human demonstration. Background Art

[0002] It is well known to use industrial robots to repeatedly perform various manufacturing, assembly, and material handling operations. However, there are problems in teaching a robot to perform even fairly simple operations - such as picking up a workpiece at a random position and orientation on a conveyor and moving the workpiece to a container - using traditional methods.

[0003] One traditional method of robot teaching includes an operator using a teach pendant to instruct the robot to make incremental moves - such as "slow advance in the X direction" or "rotate the gripper about the local Z axis" - until the robot and its gripper are in the correct position and orientation to perform the operation, then storing the operation data, and repeating this multiple times. Another known technique for teaching a robot to perform an operation is to use a motion capture system in combination with human demonstration. Since it has been found that robot programming using a teach pendant and a motion capture system is non-intuitive, time-consuming, and / or expensive, techniques for robot teaching based on human demonstration using camera images have been developed.

[0004] In some types of operations, such as assembling a device that includes many parts, a person naturally uses both hands to perform the operation task. To make robot teaching accurate in these cases, it is necessary to reliably detect the left and right hands of the human demonstrator. A known method for determining the identity of the left and right hands of a human demonstrator involves providing a camera image of the entire body of the human, performing anthropomorphic analysis of the image to identify the left and right arms, and then identifying the left and right hands based on the arm identity. However, this technique requires a camera image separate from the image required for hand posture detection to identify the arm / hands, and also requires additional computational steps for body skeleton analysis.

[0005] Other techniques that can be used to identify the left and right hands of a human demonstrator involve requiring each hand to maintain a relative position with respect to the other hand, or requiring each hand to remain within position boundaries for all teaching operations. However, these techniques impose unsustainable constraints on the natural hand movements of the human demonstrator, and there is a risk of misidentifying the hands if the constraints are violated.

[0006] In view of the above, there is a need for an improved technique for dual hand detection in robot teaching based on human demonstration. Summary of the Invention

[0007] According to the teachings of the present disclosure, a method for performing two - hand detection in robot teaching based on human demonstrations is described and illustrated. Camera images of the demonstrator's hands and the workpiece are provided to a first neural network, which determines the identities of the left and right hands of the human demonstrator based on the images and also provides cropped sub - images of the identified hands. The first neural network is trained using images in which the left and right hands are pre - identified. Then, the cropped sub - images are provided to a second neural network, which detects the postures of both the left and right hands based on the images, where if the second neural network is trained using right - hand images, the sub - image of the left hand is horizontally flipped before and after hand posture detection. The hand posture data is converted into robot gripper posture data and used to teach a robot to perform an operation through human demonstration.

[0008] In combination with the accompanying drawings, additional features of the presently disclosed apparatus and methods will become apparent from the following description and the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 FIG. is a schematic diagram of how to analyze an image of a human hand to determine the corresponding position and orientation of a finger - type robot gripper according to an embodiment of the present disclosure;

[0010] Figure 2 FIG. is a schematic diagram of how to analyze an image of a human hand to determine the corresponding position and orientation of a magnetic or suction - cup - type robot gripper according to an embodiment of the present disclosure;

[0011] Figure 3 FIG. is a schematic diagram of a system and steps for identifying the positions and postures of hands based on camera images of the two hands of a human demonstrator according to an embodiment of the present disclosure;

[0012] Figure 4 FIG. is for training in accordance with an embodiment of the present disclosure Figure 3 FIG. is a schematic diagram of steps for training a hand detection and recognition neural network used in the system of

[0013] Figure 5 FIG. is a flowchart of a method for identifying hand positions and postures based on camera images of the two hands of a human demonstrator according to an embodiment of the present disclosure;

[0014] Figure 6 FIG. is a flowchart of a method for teaching a robot to perform an operation using camera images of both the hands of a human demonstrator and the corresponding workpiece according to an embodiment of the present invention; and

[0015] Figure 7 FIG. is a schematic diagram of a system for robot operation based on teaching by using human demonstrations with both hands according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The following discussion of embodiments of the present disclosure directed to hands detection when teaching a robot by human demonstration is in nature merely exemplary and is in no way intended to limit the disclosed apparatus and techniques or their application or use.

[0017] As is well known, industrial robots are used for various manufacturing, assembly, and material movement operations. One known type of robotic operation is sometimes referred to as "pick, move, and place," where the robot picks up a part or workpiece from a first location, moves the part, and places it at a second location. The first location is typically a conveyor belt where randomly oriented parts are transported in a stream, such as parts just removed from a mold. The second location can be another conveyor that leads to a different operation, or it can be a shipping container, but in either case, the part needs to be placed at a specific location and oriented in a specific pose at the second location. Other robotic operations - such as assembling multiple parts into a device such as a computer chassis - similarly require picking up parts from one or more sources and placing them in precise positions and orientations.

[0018] To perform operations of the above type, a camera is typically used to determine the position and orientation of the incoming parts, and the robot must be taught to grasp the parts in a specific manner using a finger gripper or a magnetic or suction cup gripper. Traditionally, teaching the robot how to grasp a part based on its orientation is done by a human operator using a teach pendant. The teach pendant is used by the operator to instruct the robot to make incremental movements - such as "slow advance in the X direction" or "rotate the gripper about the local Z axis" - until the robot and its gripper are in the correct position and orientation to grasp the workpiece. The robot controller then records the robot configuration as well as the workpiece position and pose for the "pickup" operation. Similar teach pendant commands are then used to define the "move" and "place" operations. However, especially for non-professional operators, it is often found that programming a robot using a teach pendant is non-intuitive, error-prone, and time-consuming.

[0019] Another known technique for teaching a robot to perform pick, move, and place operations is to use a motion capture system. The motion capture system includes multiple cameras arranged around the work cell to record the position and orientation of the operator and the workpiece as the operator manipulates the workpiece. The operator and / or the workpiece can have additional uniquely identifiable marker points to more precisely detect key positions on the operator and the workpiece in the camera images during the performance of the operation. However, this type of motion capture system is expensive, and it is difficult and time-consuming to set up and configure precisely so that the recorded positions are accurate.

[0020] Techniques have been developed to overcome the limitations of the above-described existing robot teaching methods, including methods that use a single camera to capture images of a person performing natural part grasping and moving actions, where the images of the person's hand and its position relative to the part are analyzed to generate robot programming commands.

[0021] Figure 1 FIG. is a schematic diagram of how to analyze an image of a human hand to determine the corresponding position and orientation of a finger-type robot gripper according to an embodiment of the present disclosure. The hand 110 has a hand coordinate system 120 defined to be attached thereto. The hand 110 includes a thumb 112 having a thumb tip 114 and an index finger 116 having an index finger tip 118. Other points on the thumb 112 and index finger 116 can also be identified in the camera image, such as the positions of the bases of the thumb 112 and index finger 116 and the positions of the first phalanges of the thumb 112 and index finger 116, etc.

[0022] The point 122 is located at the midpoint between the base of the thumb 112 and the base of the index finger 116, where the point 122 is defined as the origin of the hand coordinate system 120. The orientation of the hand coordinate system 120 can be defined using any convention suitable for relating to the robot gripper orientation. For example, the Y-axis of the hand coordinate system 120 can be defined to be perpendicular to the plane of the thumb 112 and index finger 116 (the plane defined by the points 114, 118, and 122). Thus, the X and Z axes lie in the plane of the thumb 112 and index finger 116. In addition, the Z-axis can be defined to bisect the angle formed by the thumb 112 and index finger 116 (angle 114-122-118). Then, the X-axis orientation can be found by the right-hand rule based on the known Y and Z axes. As described above, the convention defined here is merely exemplary, and other coordinate system orientations can alternatively be used. The key point is that the coordinate system position and orientation can be defined based on key identifiable points on the hand, and the coordinate system position and orientation can be related to the robot gripper position and orientation.

[0023] A camera ( Figure 1 not shown in FIG. and discussed later) can be used to provide an image of the hand 110, where the image can then be analyzed to determine the spatial positions of the thumb 112 and index finger 116, including the thumb tip 114 and index finger tip 118 and the phalanges, (such as in the work cell coordinate system), and thus determine the origin position 122 and orientation of the hand reference system 120. In Figure 1In this case, the position and orientation of the hand reference system 120 are related to the gripper coordinate system 140 of the gripper 150 attached to the robot 160. The gripper coordinate system 140 has an origin 142 corresponding to the origin 122 of the hand reference system 120, and points 144 and 146 corresponding to the index finger tip 118 and the thumb tip 114, respectively. Thus, the two fingers of the finger-type gripper 150 are in the X-Z plane of the gripper coordinate system 140, where the Z axis bisects the angle 146-142-144.

[0024] The origin 142 of the gripper coordinate system 140 is also defined as the tool center point of the robot 160. The tool center point is a point whose position and orientation are known to the robot controller, where the controller can provide command signals to the robot 160 to move the tool center point and its associated coordinate system (gripper coordinate system 140) to a defined position and orientation.

[0025] Figure 2 is a schematic diagram of how to analyze an image of a human hand to determine the corresponding position and orientation of a magnetic or suction cup type robot gripper according to an embodiment of the present disclosure. Figure 1 shows how a hand pose can be related to the orientation of a mechanical gripper with movable fingers, while Figure 2 shows how a hand pose can be related to a flat gripper (e.g., circular) that picks up parts through the flat surface of the part using suction or magnetic force.

[0026] The hand 210 also includes a thumb 212 and an index finger 216. The point 214 is located at the position where the thumb 212 contacts the part 220. The point 218 is located at the position where the index finger 216 contacts the part 220. The point 230 is defined as being intermediate between the points 214 and 218, where the point 230 corresponds to the tool center point (TCP) 240 of the surface gripper 250 on the robot 260. In Figure 2 the case of the surface gripper 250 shown, based on the detection of the knuckles and fingertips, the plane of the gripper 250 can be defined as the plane that contains the line 214-218 and is perpendicular to the plane of the thumb 212 and the index finger 216. As described above, the tool center point 240 of the gripper 250 corresponds to the point 230. This completely defines the position and orientation of the surface gripper 250 corresponding to the position and pose of the hand 210.

[0027] Techniques for teaching a robot to perform an operation based on a human demonstration, particularly based on an analysis of images of a human hand and a workpiece by a camera, are described in the co-assigned U.S. patent application Ser. No. 16 / 843,185, filed Apr. 8, 2020, and entitled "ROBOT TEACHING BY HUMAN DEMONSTRATION"; the '185 application is incorporated herein by reference in its entirety. The '185 application discloses techniques for determining the 3D coordinates of key points of a hand (knuckles, etc.) based on a camera image of a single hand.

[0028] In some types of operations, such as assembling a device including several components, a human demonstrator naturally uses both hands to perform the operation task. To make the robot teaching accurate in these cases, it is necessary to reliably identify the left and right hands of the human demonstrator in the image. A known method for determining the identity of the left and right hands of a human demonstrator involves providing a camera image of the entire body of the human, performing anthropomorphic analysis of the body image to identify the left and right arms, and then identifying the left and right hands based on the arm identity. However, this technique requires a camera image separate from the image required for hand pose detection to identify the arm / hands, and also requires additional computational steps for body skeleton analysis. Other two-handed teaching methods prohibit the human demonstrator from crossing their hands across to their "opposite sides".

[0029] Using the key point detection method of the '185 application, the present disclosure describes techniques for reliably determining the identity, position, and pose of both hands of a human demonstrator in a camera image, without imposing artificial restrictions on the demonstrator's use or movement of his / her hands, and without requiring a full-body image and analysis as required in existing methods.

[0030] Figure 3 FIG. 10 is a schematic diagram of a system and steps for identifying the position and pose of hands based on camera images of both hands of a human demonstrator according to an embodiment of the present disclosure. Camera 310 provides an image of a training workspace, i.e., camera 310 provides an image of the area that the operator's hands will occupy during a teaching demonstration. The training workspace can be, for example, a desktop on which the device is being assembled. Camera 310 is preferably a two-dimensional (2D) camera that provides a color image of the training workspace but does not provide depth information like a 3D camera.

[0031] Camera 310 provides an image 312 as Figure 3 shown. The processing of image 312 is described in detail in Figure 3 . Camera 310 provides a continuous stream of images, where each image is as Figure 3The shown is processed to provide a complete motion sequence for use by a robot, such as picking up a part, moving it to a new location, and placing it in a desired pose. The human demonstrator is at the top of image 312, so the right hand appears on the left side of image 312 and the left hand appears on the right side of image 312.

[0032] Image 312 is analyzed by a first neural network 320 to determine the identities of the left and right hands and their respective positions in image 312. The first neural network 320 - which is capable of identifying the left and right hands in an image of only the hands (rather than the entire body) - provides an ability not available in existing hand image analysis systems. The first neural network 320 identifies the left and right hands based on cues such as the curvature of the fingers (the fact that the fingers of a human hand can only bend in one direction) like the four fingers and the thumb and relative positions, regardless of the relative position of the hands in image 312. Through appropriate training (discussed below with reference to Figure 4 ), the first neural network 320 has been shown to be able to quickly and reliably determine the identities and positions of the left and right hands in image 312.

[0033] Based on the output of the first neural network 320 at block 330, a cropped image 332 of the right hand and a cropped image 334 of the left hand are created. Again, the image 332 of the right hand and the image 334 of the left hand are determined by the first neural network 320 through image analysis based on the actual identity of the hands, rather than simply based on the position of the hands in image 310 / 312. That is, in some images, the hands can be crossed such that the left and right hands appear in positions opposite to the expected "normal" positions.

[0034] As shown, the image 332 of the right hand and the image 334 of the left hand are cropped tightly around the hands to provide the maximum amount of image resolution and the minimum amount of redundant data for subsequent analysis. The image 332 of the right hand is provided to a second neural network 350 on line 342. The second neural network 350 analyzes image 332 to determine the three-dimensional (3D) coordinates of a plurality of key points on the right hand. The key points include the tips of the four fingers, the knuckles of the four fingers, the tip of the thumb, and the knuckle of the thumb. The second neural network 350 is trained using many images of a particular hand (assumed to be the right hand for the purposes of this discussion). Techniques for determining the 3D coordinates of key points on a hand from an image of a hand of known identity (left or right) are disclosed in the above-mentioned 16 / 843,185 application.

[0035] An image 334 of a left hand is provided on line 344. In the case of training the second neural network 350 to identify key points in an image of a right hand, the image 334 of the left hand must be horizontally flipped at frame 346 before being provided to the second neural network 350. The second neural network 350 analyzes the flipped version of the image 334 to determine the three-dimensional (3D) coordinates of a plurality of key points (finger tips, knuckles, etc.) on the left hand. Since the image 334 has been horizontally flipped, the second neural network 350 can analyze the flipped image 334 as accurately as if it were an image of a right hand.

[0036] It should be clear that an image of a left hand or a right hand can be used to train the second neural network 350. If the second neural network 350 is trained with a right hand image, the image of the left hand must be flipped for processing by the second neural network 350, and vice versa.

[0037] On line 362, a 3D "wireframe" structure of a right hand is provided to frame 372. As discussed in detail in the above-referenced application 16 / 843,185, the 3D wireframe structure of the hand output by the second neural network 350 includes as many key points and connectivities of the structure of the hand as can be determined based on visibility in the original image (e.g., an index finger bone segment connecting the finger tip at coordinates X1 / Y1 / Z1 to the first knuckle at coordinates X2 / Y2 / Z2, etc.). That is, the positions of fingers or portions of fingers that are curled and occluded below the field of view in the image cannot be distinguished.

[0038] On line 364, a 3D wireframe structure of a left hand is output from the second neural network 350. The horizontal coordinates (usually the X coordinates) of the key points of the left hand must be flipped at frame 366 before being provided to frame 374. The horizontal flip at frame 366 must be about the same mirror plane (e.g., the Y-Z plane) as the original image flip at frame 346.

[0039] As a result of the above image analysis, frame 372 contains the 3D wireframe structure of the right hand (3D coordinates of finger tips and finger joints on the four fingers and the thumb), and frame 374 similarly contains the 3D wireframe structure of the left hand. Using the 3D coordinate data from the hand, the gripper coordinates can be calculated as Figure 1 and 2 shown and as described above. In this way, the gripper position and orientation are calculated and then output on line 380.

[0040] Figure 4 is a schematic diagram of steps for training a hand detection and recognition neural network 320 used in a system in Figure 3 The first neural network 320 is shown at the center of Figure 4 As Figure 3As shown and discussed above, the first neural network 320 is responsible for determining the identity and position of the left and right hands in the image. The first neural network 320 is trained to recognize the left and right hands by providing it with a number of training images, in which the left and right hands are in a predetermined relative position in the training images.

[0041] Image 410 is an example of a training image for training the first neural network 320. Image 410 includes the left and right hands of a human demonstrator, and the left and right hands are in a known relative position, such as on a designated side of a dividing line or identified within a bounding box. One way to pre-determine the positions of the left and right hands in image 410 is to have the hands in their "normal" relative position (not crossed at the wrists). Another way to pre-determine the positions of the left and right hands in image 410 is to position each hand on its respective side of the dividing line 412. In image 410, the dividing line 412 is located at or near the center of the image, but this need not be the case. For the case where the hands are crossed at the wrists, the positions of the left and right hands are manually annotated in the bounding box.

[0042] As is known to those skilled in the art, the first neural network 320 is a multi-layer neural network, including an input layer, an output layer, and usually two or more internal hidden layers. The first neural network 320 is trained to recognize images of hands and to identify the structural characteristics of the hands that distinguish the left hand from the right hand. A combination of several factors - such as the curvature of the four fingers (which can only bend in one direction - towards the palm), the relative position of the thumb and the four fingers, etc. - can be used to distinguish the top and bottom as well as the left and right of a particular hand. Since the first neural network 320 knows the identity of the left and right hands before analyzing each image, the neural network 320 can automatically construct the structure of its layers and nodes to reliably associate the structural features with the identity of the hands. During the training process by analyzing multiple images, the first neural network 320 learns to recognize the structural features that are characteristic of the right hand and the features that are characteristic of the left hand.

[0043] The output image 420 shows the training result from image 410. The hand is detected and located in box 422, and the first neural network 320 knows it is the right hand based on its position relative to the dividing line 412 (since the human body is at the top of images 410 / 420, the right hand of the person is on the left side of images 410 / 420). Similarly, the hand is detected and located in box 424, and the first neural network 320 knows it is the left hand based on its position. Techniques are employed to crop the sub-images around the hands as shown in boxes 422 and 424, where the sub-images are cropped to include only the areas of all visible fingertips of the four fingers and the thumb tip and, for example, the position identified as the wrist joint.

[0044] Image 430 is another example of a training image for training the first neural network 320. Similarly, image 430 includes both the left and right hands of a human demonstrator, and the left and right hands are identified within bounding boxes. In image 430, a bounding box 432 is provided as an annotation or indexing attribute identifying the right hand. The demonstrator's hands are crossed in image 430, so the right hand is located where the left hand is expected to be; however, due to the bounding box identification, the first neural network 320 knows that the hand in bounding box 432 is the demonstrator's right hand. Similarly, a bounding box 434 is provided as an annotation or indexing attribute identifying the left hand.

[0045] Output image 440 shows the training result from image 430. A hand is detected and located in a box 442 that is substantially the same as bounding box 432, and even though the hands are crossed, the first neural network 320 knows based on the bounding box information that it is the right hand. Similarly, a hand is detected and located in box 444, and the first neural network 320 knows based on the bounding box information that it is the left hand. During the process of analyzing the hands in boxes 442 and 444 in images 430 / 440, the first neural network 320 is incrementally trained in hand identity detection.

[0046] Image 430 is very different from image 410. The input images include different human demonstrators, different parts, different operations and backgrounds, with and without gloves, and even slightly different camera angles (viewpoints). These differences in the input training images help train the first neural network 320 to robustly recognize hand structures and identities in images, which will be processed during the actual execution phase of robot teaching.

[0047] Many other input images 450 are provided to the first neural network 320 for training. As Figure 4 shown, each input image 450 produces an output image 460 in which the left and right hands have been located and identified. After training, the first neural network 320 is ready to be used, as Figure 3 shown, to identify the left and right hands in image 312 (even if the hands are crossed), and to provide a cropped sub-image containing the correctly identified hands. A test system has been developed that demonstrates the following capabilities of neural networks such as the first neural network 320: to quickly and accurately identify the right and left hands in images as described above, even when the left and right hands repeatedly overlap, cross, and uncross in a sequence of consecutive images.

[0048] Figure 5 is a flowchart 500 of a method for identifying the position and pose of hands based on camera images of the two hands of a human demonstrator according to an embodiment of the present disclosure. Flowchart 500 depicts the method steps corresponding to Figure 3 the system block diagram of.

[0049] At block 502, an image containing the hands of a human demonstrator is provided. An image such as Figure 3 image 312 preferably does not include the entire body of a person. The image also does not need to have the left and right hands in their "normal" or "expected" relative positions. The image depicts a human demonstrator performing operations on one or more workpieces, such as assembling a device including multiple parts, where the hands are used to pick up and place individual parts. In actual practice, images will be provided in quick succession (multiple images per second) such that a sequence of spatial grasping and placing operations can be taught. In addition to the identity, position, and pose of the hands, the workpiece position and pose will also be determined from the image and combined with the hand ("gripper") data for robot teaching.

[0050] At block 504, a first neural network 320 is used to determine the identity and position of the left and right hands in the provided image. The operations performed at block 504 have been discussed in detail above. At block 506, the original image is cropped into two sub-images, one containing the left hand and one containing the right hand. The identity of the hands is provided along with the sub-images.

[0051] At block 508, a second neural network 350 is used to analyze the right hand sub-image to detect finger structure and hand pose. The operations performed at block 508 are as described above and are detailed in the earlier-referenced patent application 16 / 843,185. Since the second neural network 350 has been trained to use images of the right or left hand to detect hand structure, it is necessary to correctly identify the sub-image before analysis in the second neural network 350. In flowchart 500, it is assumed that the second neural network 350 has been trained using right hand images; thus, the right hand sub-image from block 506 is passed directly to block 508.

[0052] At block 510, the left hand sub-image is horizontally flipped before being provided to block 508 for analysis. Again, it is assumed that the second neural network 350 has been trained using right hand images; thus, the left hand sub-image from block 506 must be horizontally flipped before being passed to block 508. The reverse process is equally applicable, where the second neural network 350 is trained using left hand images and the right hand sub-image is flipped before analysis.

[0053] At block 512, the finger structure and hand pose data (3D coordinates of the hand skeleton key points) of the right hand are used to calculate the corresponding gripper pose, and the gripper pose (along with the workpiece pose data) is output as a robot teaching step. The complete method for robot teaching based on images of a human demonstration (hands and workpieces) is discussed below.

[0054] At block 514, before calculating the corresponding gripper pose at block 512 and outputting the gripper pose as a robot teaching step, the horizontal coordinates (e.g., X coordinates) of the finger structure and hand pose data of the left hand from block 508 are horizontally flipped. The horizontal coordinate data must be flipped or mirrored with respect to the mirror plane to put the 3D coordinate data of the hand back in its proper position from the original input image.

[0055] As will be understood by those skilled in the art, throughout the process of calculating the 3D coordinates of the hand pose, the positions of the left and right hand sub-images within the original input image must be known. Additionally, the pixel coordinates of the originally provided image must be mapped to the physical workspace where the demonstration is taking place such that the 3D gripper and workpiece positions and poses can be calculated based on the image pixel coordinates.

[0056] The robot teaching step is output from block 512 and recorded for robot programming. The teaching step includes the gripper position and pose calculated based on the left and right hand pose coordinate data, as well as the corresponding workpiece position and pose. The process then loops back to block 502 to receive another input image.

[0057] Figure 6 FIG. 600 is a flow chart of a method for teaching a robot to perform an operation using camera images of the hands of a human demonstrator and corresponding workpieces according to an embodiment of the present invention. Flow chart 600 is arranged in three vertical columns corresponding to a pick-up step (on the right), a move step (center), and a place step (on the left). Three separate steps illustrate how to analyze images of the hands and workpieces to create a robot motion program, where detection of the hands in the images is an integral part.

[0058] The pick-up step begins at start block 602. At block 604, the workpiece and both hands are detected in the image from camera 310. The two-handed detection method described in detail above is used in block 604. The position and orientation of the workpiece coordinate system are determined based on the analysis of the workpiece in the image, and the position and orientation of the corresponding hand coordinate system are determined based on the analysis of the hands in the image.

[0059] At decision diamond 606, for each hand, it is determined whether the fingertips ( Figure 1 the thumb tip 114 and index finger tip 118) have contacted the workpiece. This is determined based on the camera image. When the fingertips have contacted the workpiece, the grasping pose and position of the workpiece and hand are recorded at block 608. Importantly, the pose and position of the hand relative to the workpiece are identified; that is, the positions and orientations of the hand coordinate system and workpiece coordinate system must be defined relative to some globally fixed reference frame such as the work cell coordinate system. This allows the controller to determine how to position the gripper to grasp the workpiece during a later reproduction phase. The analysis of workpiece contact is performed for each of the right and left hands.

[0060] After the grasping postures and positions of the workpiece and the hand are recorded at block 608, the pick-up step ends at end block 610. Then, the process proceeds to the movement step, which begins at block 622. The movement step can be performed separately for each hand. At block 624, the workpiece is detected in the camera image. At decision diamond 626, if the workpiece is not detected in the camera image, the process loops back to block 624 to obtain another image. When the workpiece is detected in the camera image, the workpiece position (and optionally the posture) is recorded at block 628.

[0061] At block 634, the hand (either hand - the one performing the current movement operation) is detected in the camera image. At decision diamond 636, if the hand is not detected in the camera image, the process loops back to block 634 to obtain another image. When the hand is detected in the camera image, the hand position (and optionally the posture) is recorded at block 638. When both the workpiece position (from block 628) and the hand position (from block 638) are detected and recorded from the same camera image, the hand position and the workpiece position are combined and recorded at block 640. Combining the hand position and the workpiece position can be achieved by simply taking the average of the two; for example, if the midpoint between the thumb tip 114 and the index finger tip 118 coincides with the center / origin of the workpiece, the average position between the midpoint and the workpiece center can be calculated.

[0062] Preferably, by repeating the activities from the start movement block 622 to the combine hand and workpiece position block 640, multiple positions along the movement step are recorded to define a smooth movement path. After the hand position and the workpiece position are combined and recorded at block 640 and the movement step positions are no longer needed, the movement step ends at end block 642. Then, the process proceeds to the placement step, which begins at block 662.

[0063] At block 664, the position of the workpiece is detected in the image from camera 310. At decision diamond 666, it is determined whether the workpiece is found in the camera image and whether the workpiece is stationary. Alternatively, it can be determined whether the fingertips have detached from contact with the workpiece. When the workpiece is determined to be stationary and / or the fingertips have detached from contact with the workpiece, the final posture and position of the workpiece are recorded at block 668. The entire process of the placement step and the teaching phase ends at end block 670.

[0064] Figure 6 The robot teaching process described in flowchart 600 of depends on the robust detection of the human hand posture in the image. When the human demonstration involves using both hands, Figure 3 - 5 The two - hand detection method and system of are essential parts.

[0065] Figure 7FIG. 700 is a schematic diagram of a system for robot operation based on teaching by a human demonstration using both hands. The human demonstrator 710 is positioned such that the camera 720 can capture images of the demonstrator's hands and the workpiece on which the operation is being performed. The camera 720 corresponds to Figure 3 the camera 310. The camera 720 provides the images to a computer 730, which analyzes the images to identify the 3D wireframe coordinates of the hands and the corresponding workpiece positions, as described in detail above. The analysis by the computer 730 includes Figure 3 - 5 the two-handed detection method shown.

[0066] The human demonstrator 710 demonstrates a complete operation, such as assembling multiple parts into a complete device. The camera 720 provides a continuous stream of images, and the computer 730 analyzes the images and records the robot teaching commands as identified. Each teaching step includes a gripper pose calculated based on the hand pose and the corresponding workpiece position / pose. This recording of the teaching steps includes grasping and placing operations performed by one or both hands of the human demonstrator 710.

[0067] When the robot operation is fully defined according to the human demonstration, the robot program is transferred from the computer 730 to the robot controller 740. The controller 740 communicates with the robot 750. The controller 740 computes the robot motion commands to cause the robot 750 to move its gripper 760 to the position and orientation in the gripper coordinate system identified from the images. The robot 750 moves the gripper 760 relative to the workpiece 770 according to the sequence of commands from the controller 740, thereby completing the operation demonstrated by the human demonstrator 710.

[0068] Figure 7 In the case of, the gripper 760 grasps the workpiece 770 and performs some operations on the workpiece 770, such as moving the workpiece 770 to a different position and / or orientation. The gripper 760 is shown as a finger gripper, but can also be a suction cup or a magnetic surface gripper as described above.

[0069] Figure 7 The system 700 can be used in two different modes. In one mode, the human demonstrator pre-teaches all the steps of an operation such as device assembly at once, and then the robot repeatedly performs the assembly operation based on the part movement instructions taught by the human demonstration. In another mode, called teleoperation, the human demonstrator works in real-time collaboration with the robot, where each action of the hand grasping and moving the parts is analyzed and immediately executed by the robot, and the motion of the robot is visually fed back to the human operator. Both of these operation modes can benefit from the disclosed techniques for two-handed detection in human demonstrations.

[0070] Throughout the foregoing discussion, various computers and controllers have been described and implied. It should be understood that the software applications and modules of these computers and controllers are executed on one or more computing devices having a processor and a memory module. In particular, this includes the processors in the computer 730 and the robot controller 740 discussed above. Specifically, the processor in the computer 730 is configured to perform two - hand detection in robot teaching via human demonstration in the manner described above.

[0071] As described above, the disclosed technique for two - hand detection in robot teaching via human demonstration makes robot motion programming faster, easier, and more intuitive than prior techniques, while providing reliable detection of the teacher's two hands with only a single camera.

[0072] Although multiple exemplary aspects and embodiments of two - hand detection in robot teaching via human demonstration have been discussed above, those skilled in the art will recognize its modifications, permutations, additions, and sub - combinations. Accordingly, the appended claims below and the claims introduced hereafter are intended to be construed to include all such modifications, permutations, additions, and sub - combinations in their true spirit and scope.

Claims

1. A method for detecting both hands in an image, the method comprising: Providing an image including a human left hand and a right hand; Using a first neural network running on a computer having a processor and a memory to analyze the image to determine the identities and positions in the image of the left hand and the right hand; Creating a left hand sub-image and a right hand sub-image, wherein each of the sub-images is cropped from the image; Providing the sub-images to a second neural network running on the computer, including horizontally flipping the left hand sub-image or the right hand sub-image; Analyzing the sub-images by the second neural network to determine three-dimensional (3D) coordinates of a plurality of key points on the left hand and the right hand; and Using the 3D coordinates of the key points by a robot teaching program to define a gripper pose, including horizontally flipping the coordinates of the key points on the left hand or the right hand, wherein horizontally flipping the coordinates of the key points on the left hand or the right hand includes horizontally flipping the coordinates of the key points on the hand whose sub-image has been flipped before being analyzed by the second neural network.

2. According to the method of claim 1, wherein The image is provided by a two-dimensional (2D) digital camera.

3. The method according to claim 1, wherein, The first neural network is trained to distinguish between the left hand and the right hand during a training process, during which a plurality of training images in which the left hand and the right hand are pre-identified are provided to the first neural network.

4. The method according to claim 3, wherein, The first neural network analyzes the training images to identify distinguishing features between the left hand and the right hand, the distinguishing features including the curvature and relative positions of the fingers.

5. The method according to claim 1, wherein, Each of the sub-images is cropped to include the left hand or the right hand within a predetermined boundary.

6. The method according to claim 1, wherein, Horizontally flipping the left hand sub-image or the right hand sub-image includes horizontally flipping the left hand sub-image when using training images of the right hand to train the second neural network, and horizontally flipping the right hand sub-image when using training images of the left hand to train the second neural network.

7. The method according to claim 1, wherein, The plurality of key points on the left hand and the right hand include the thumb tip, the thumb knuckle, the fingertips of the four fingers, and the knuckles of the four fingers.

8. The method according to claim 1, wherein Horizontally flipping the coordinates of the key points includes horizontally flipping the coordinates across a vertical plane to restore the coordinates to their positions in the image.

9. The method according to claim 1, wherein, The image further includes one or more workpieces, and the gripper pose and the workpiece position and pose are used by the robot teaching program to create workpiece pick-up and placement instructions for the robot.

10. The method according to claim 9, wherein, The instructions are provided from the computer to a robot controller, and the robot controller provides control commands to the robot to perform workpiece operations.

11. A method for programming a robot to perform an operation through human demonstration, the method comprising: Demonstrating an operation on a workpiece by a human using both hands; The camera images of the two hands performing the operation on the workpiece are analyzed by a computer to create demonstration data including a gripper pose calculated based on three-dimensional (3D) coordinates of key points of the two hands, wherein the 3D coordinates of the key points are determined based on the images by a first neural network and a second neural network, the first neural network being for identifying the left hand and the right hand in the image, and the second neural network being for calculating the 3D coordinates in sub-images of the identified left hand and right hand; Generating a robot motion command for causing the robot to perform the operation on the workpiece based on the demonstration data; And Performing the operation on the workpiece by the robot, wherein, before being provided to the second neural network, the left hand sub-image or the right hand sub-image is horizontally flipped, and after being calculated by the second neural network, the 3D coordinates of the key points of the left hand or the right hand are horizontally flipped.

12. The method according to claim 11, wherein, The demonstration data includes the hand coordinate system, the position and orientation of the gripper coordinate system and the workpiece coordinate system corresponding to the hand coordinate system in the grasping step of the operation.

13. The method according to claim 11, wherein, The first neural network is trained to distinguish the left hand and the right hand during the training process, and during the training process, a plurality of training images in which the left hand and the right hand are pre-identified are provided to the first neural network.

14. The method according to claim 11, wherein, When using the training image of the right hand to train the second neural network, the left hand sub-image and the 3D coordinates of the key points of the left hand are horizontally flipped.

15. A system for detecting two hands in an image, which is used for programming a robot to perform an operation through a human demonstration, the system comprising: A camera; A computer having a processor and a memory and communicating with the camera, the computer being configured to perform steps including the following: Analyzing an image including a human left hand and right hand using a first neural network to determine the identities and positions in the image of the left hand and the right hand; Creating a left hand sub-image and a right hand sub-image, wherein each of the sub-images is cropped from the image; Providing the sub-images to a second neural network running on the computer, including horizontally flipping the left hand sub-image or the right hand sub-image; Analyzing the sub-images by the second neural network to determine the three-dimensional (3D) coordinates of a plurality of key points on the left hand and the right hand; and Using the 3D coordinates of the key points to define a gripper pose for programming the robot, including horizontally flipping the coordinates of the key points on the left hand or the right hand, wherein horizontally flipping the coordinates of the key points on the left hand or the right hand includes horizontally flipping the coordinates of the key points on the hand whose sub-image has been flipped before being analyzed by the second neural network.

16. The system according to claim 15, wherein, The first neural network is trained to distinguish between the left hand and the right hand during the training process. During the training process, a plurality of training images in which the left hand and the right hand are pre-identified are provided to the first neural network, and wherein the first neural network analyzes the training images to identify the distinguishing features between the left hand and the right hand, and the distinguishing features include the curvature and relative positions of the fingers.

17. The system according to claim 15, wherein, Horizontally flipping the left hand sub-image or the right hand sub-image includes horizontally flipping the left hand sub-image when using the training images of the right hand to train the second neural network, and horizontally flipping the right hand sub-image when using the training images of the left hand to train the second neural network.

Images