Robot teaching through demonstrations using visual servos

Abstract

To provide an improved robot teaching technique which is simple and intuitive for execution of a human operator, and has the accuracy necessary for an operation of attachment and assembly of a robot.SOLUTION: A method for teaching and controlling a robot based on human demonstration includes a demonstration phase where a camera detects a human hand grasping and moving a workpiece to define a rough trajectory of the robotic movement of the workpiece. Line features or other geometric features on the workpiece collected during the demonstration phase are used in an image-based visual servoing (IBVS) approach which refines a final placement position of the workpiece, where the IBVS control takes over the workpiece placement during the final approach by the robot. Moving object detection is used for automatically localizing both object and hand position in 2D image space, and then identifying line features on the workpiece by removing line features belonging to the hand using hand keypoint detection.SELECTED DRAWING: Figure 6

Description

[[TECHNICAL FIELD]]

[0001] The present disclosure relates to the field of industrial robot programming, and more specifically to a method of programming a robot to perform workpiece pick / move / place operations, which comprises a demonstration stage in which a camera detects a human hand grasping and moving a workpiece to define a rough trajectory. Geometric features of the workpiece collected during the demonstration stage are used for image-based visual servo adjustment of the final placement position of the workpiece. [[BACKGROUND ART]]

[0002] It is well known that industrial robots are used to repeatedly perform various manufacturing, assembly and material transfer operations. However, even teaching a robot to perform fairly simple operations, such as lifting a workpiece from random positions and orientations on a conveyor and moving the workpiece to a container or a second conveyor, can be unintuitive, time-consuming and / or costly using conventional methods.

[0003] Traditionally, robots have been taught to perform pick-and-place operations of the above type by a human operator using a teach pendant. The teach pendant is used by the operator commanding the robot to make incremental movements, such as "jog in the X direction" or "rotate the gripper about the local Z axis", until the robot and its gripper are in the correct position and orientation to grip the workpiece. The configuration of the robot and the position and orientation of the workpiece are then recorded by the robot controller for use in the "pick" operation. Then, "move" and "place" operations are defined using substantially the same teach pendant commands. However, the use of teach pendants for robot programming is often unintuitive, error-prone and time-consuming, especially for non-expert operators.

[0004] Another known technique for teaching robots to perform pick-and-place operations is the use of motion capture systems. Motion capture systems use multiple cameras positioned around a work cell to record the position and orientation of the human operator and workpiece as the operator manipulates the workpiece. The operator and / or workpiece may be marked with uniquely recognizable marker dots to more accurately detect key locations on the operator and workpiece within the camera images as the operation is performed. However, this type of motion capture system is expensive, difficult to set up and configure accurately to ensure the recorded positions are precise, and time-consuming.

[0005] In addition, while robot teaching through human demonstration is known, it may lack the positional accuracy required for precise workpiece placement, such as in applications involving the attachment and assembly of parts.

[0006] In light of the above circumstances, there is a need for improved robot teaching technology that is simple and intuitive for human operators to perform and possesses the precision required for robot installation and assembly operations. [Overview of the Initiative]

[0007] This disclosure discloses a method for teaching and controlling a robot to perform operations based on a human demonstration using images from a camera, in accordance with the teachings of this disclosure. The method includes a demonstration phase in which a camera detects a human hand grasping and moving a workpiece to define the rough trajectory of the robot's movement of the workpiece. Line features or other geometric features on the workpiece collected during the demonstration phase are used in an image-based visual servo (IBVS) approach to make the final placement position of the workpiece precise, where IBVS control takes over the placement of the workpiece during the robot's final approach. Moving object detection is used to automatically locate the positions of both the object and the hand in 2D image space, and then line features on the workpiece are identified by using principal point detection of the hand to remove line features belonging to the hand.

[0008] Additional features of the currently disclosed apparatus and method will become apparent from the following description and the attached claims, in conjunction with the attached drawings. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows a method according to one embodiment of the present disclosure for determining the corresponding position and orientation of a finger-shaped robotic gripping part by analyzing an image of a human hand. [Figure 2] Figure 2 is a diagram of a system and steps for teaching a robot to perform a pick-and-place operation using camera images of a human hand and a workpiece, according to one embodiment of the present disclosure. [Figure 3] Figure 3 is a diagram of a system and steps for a robot to perform a pick-and-place operation using a camera image of a workpiece and programming previously taught by a human hand demonstration, according to one embodiment of the present disclosure. [Figure 4] Figure 4 shows a basic concept of a robot teaching technique using image-based visual servos and human demonstration, according to one embodiment of the present disclosure. [Figure 5A] Figure 5A is an image of a workpiece being attached to an assembly by a human demonstrator according to one embodiment of the present disclosure. [Figure 5B] Figure 5B is a cropped version of the image from Figure 5A, showing line features identified for an image-based visual servo according to one embodiment of the present disclosure. [Figure 6] Figure 6 shows details of a technique for human demonstration-based robot teaching using an image-based visual servo, according to one embodiment of the present disclosure. [Figure 7] Figure 7 is a flowchart illustrating a method for human-demonstrated robot teaching using an image-based visual servo, including a demonstration phase and a robot execution phase, according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0010] The following considerations of embodiments of this disclosure relating to human demonstration of robot teaching using image-based visual servos are essentially illustrative and are not intended in any way to limit the apparatus and technology of the disclosure or its uses or applications.

[0011] The use of industrial robots for various manufacturing, assembly, and material handling operations is well known. One known type of robot operation is sometimes referred to as "take-out, move-and-set-up." Here, the robot takes a part or workpiece from a first location, moves it, and sets it up in a second location. The first location may be a conveyor belt with randomly oriented parts flowing along it, such as parts just removed from a mold. The second location may be another conveyor leading to another operation, or it may be a shipping container, but in either case, the part needs to be set up in a specific location and oriented in a specific position at the second location.

[0012] To perform the above types of pick-up, move, and place operations, it is typically necessary to use a camera to determine the position and orientation of the incoming part and to teach the robot to grasp the part in a specific way using a finger-type gripper, magnetic gripper, or suction cup gripper. Traditionally, the task of teaching the robot how to grasp the part depending on its orientation was performed by a human operator using a teaching pendant. The teaching pendant is used by the operator to instruct the robot to make incremental movements, such as "jog in the X direction" or "rotate the gripper around the local Z axis," until the robot and its gripper are in the correct position and orientation to grasp the workpiece. The robot's configuration and the position and orientation of the workpiece are then recorded by the robot controller and used for the "pick-up" operation. Then, "move" and "place" operations are defined using almost the same teaching pendant commands. However, using a teaching pendant for robot programming is often unintuitive, error-prone, and time-consuming, especially for non-expert operators.

[0013] Another known technique for teaching robots to perform retrieval, movement, and placement operations is the use of motion capture systems. Motion capture systems use multiple cameras positioned around a work cell to record the position and orientation of the human operator and workpiece as the operator manipulates the workpiece. The operator and / or workpiece may be marked with uniquely recognizable marker dots to more accurately detect key locations on the operator and workpiece within the camera images as the operation is performed. However, this type of motion capture system is expensive, and accurately setting up and configuring it to ensure accurate recorded positions is difficult and time-consuming.

[0014] In addition, there is a known method of robot teaching by human demonstration, in which one or more cameras capture images of a human hand moving a workpiece from a starting (retrieval) location to a target (installation) location. The technology of robot teaching by human demonstration is disclosed in U.S. Patent Application 16 / 843,185, entitled “Robot Teaching by Human Demonstration,” filed on April 8, 2020, and assigned in common with this application. This U.S. Patent Application is incorporated herein by reference in its entirety. The aforementioned application is hereafter referred to as “Application 185.”

[0015] The technology described in application 185 works well when fine precision is not required for workpiece placement. However, in precision placement applications such as robotic mounting of components into assemblies, uncertainty in the manual gripping posture of the workpiece can cause problems. Therefore, a technology is needed to improve the precision of workpiece placement. This disclosure achieves this by using an image-based visual servo (IBVS) to accurately place a workpiece to a target location within an assembly, and further enhances the IBVS approach by automatically identifying the line features belonging to the workpiece.

[0016] Figure 1 shows a method for analyzing an image of a human hand to determine the corresponding position and orientation of a finger-shaped robotic gripping part, as disclosed in the '185 application and used in embodiments of the present disclosure. The hand 110 has a hand coordinate frame 120 defined in the manner in which it is attached to the hand. The hand 110 includes a thumb 112 having a thumb tip 114 and an index finger 116 having an index finger tip 118. Other points of the thumb 112 and index finger 116, such as the location of the base of the thumb 112 and index finger 116 and the location of the first joint of the thumb 112 and index finger 116, may also be identified in the camera image.

[0017] Point 122 is positioned midway between the base of the thumb 112 and the base of the index finger 116. Here, point 122 is defined as the origin of the hand coordinate frame 120. Other actual structural points or derived structural points of the hand 110 may be defined as the origin 122. The orientation of the hand coordinate frame 120 may be defined using any arrangement suitable for correlation with the orientation of the robot gripping part. For example, the Y-axis of the hand coordinate frame 120 may be defined as perpendicular to the plane of the thumb 112 and index finger 116 (the plane defined by points 114, 118, and 122). Thus, the X-axis and Z-axis lie within the plane of the thumb 112 and index finger 116. Furthermore, the Z-axis may be defined as bisecting the angle formed by the thumb 112 and index finger 116 (angle 114-122-118). The orientation of the X-axis may be determined from the known Y-axis and Z-axis by the right-hand rule. As mentioned above, the arrangements defined here are merely illustrative, and other coordinate frame orientations may be used instead. The key is that the position and orientation of the coordinate frame may be defined based on the main recognizable points of the hand, and that the position and orientation of that coordinate frame can be associated with the position and orientation of the robot's grasping part.

[0018] A camera (not shown in FIG. 1 and described later) may be used to provide an image of a hand 110. Here, the image can be analyzed to determine the spatial positions (such as in a work cell coordinate frame) of the thumb 112 and index finger 116, including the thumb tip 114, the index finger tip 118 and finger joints, and further determine the position 122 and orientation of the origin of the hand reference coordinate system 120. In FIG. 1, the position and orientation of the hand reference coordinate system 120 correlate with the gripper coordinate frame 140 of the gripper 150 attached to the robot 160. The gripper coordinate frame 140 has an origin 142 corresponding to the origin 122 of the hand reference coordinate system 120, and points 144 and 146 respectively corresponding to the index finger tip 118 and the thumb tip 114. Therefore, the two fingers of the finger-type gripper 150 lie within the X-Z plane of the gripper coordinate frame 140, and the Z-axis bisects the angle 146-142-144.

[0019] Alternatively, the origin 142 of the gripper coordinate frame 140 may be defined as the tool center point of the robot 160. The tool center point is a point whose position and orientation are known by the robot controller. Here, the controller may provide command signals to the robot 160 to move the tool center point and its associated coordinate frame (the gripper coordinate frame 140) to a defined position and orientation.

[0020] FIG. 1 shows how an image of a hand corresponds to a finger-type robot gripper. Similar techniques can be used to correlate the position and orientation of a human demonstrator's hand with a suction cup-type gripper or other types of grippers.

[0021] Figure 2 is a diagram of a system 200 and steps for teaching a robot to perform a pick-and-place operation using camera images of a human hand, as disclosed in the '185 application and used in embodiments of this disclosure. The teaching steps shown in Figure 2 are performed in a work cell 202. The work cell 202 includes a camera 210 for capturing images of the teaching steps. The camera 210 may be a three-dimensional (3D) camera or a two-dimensional (2D) camera, insofar as it can identify the coordinates of specific points and features of the hand, such as fingertips, thumb tips, and knuckles, as discussed herein. Different techniques using a 3D camera for hand pose detection and different techniques using a 2D camera for hand pose detection are discussed in detail in the '185 application. The camera 210 is configured to provide images of the portion of the work cell 202 in which the part movement activity (all steps (1) to (3) discussed below) is performed.

[0022] Camera 210 communicates with robot controller 220. Controller 220 analyzes images from the teaching step and generates robot programming commands, as discussed below. These robot programming commands are used to control the robot's movements when performing pick-and-place operations during the replay or execution phase. Alternatively, a separate computer may be provided between camera 210 and controller 220. This separate computer would analyze camera images and communicate the gripping position or final trajectory to controller 220. The teaching phase in Figure 2 consists of the following three steps: The pick-up step (step (1)) determines the position and orientation (orientation is a term interchangeable with orientation) of the workpiece and the position and orientation of the hand gripping the workpiece. The move step (step (2)) tracks the position of the workpiece and the hand as the workpiece moves from the pick-up location to the installation location. The installation step (step (3)) uses the position and orientation of the workpiece to install it in its intended (installation) location. The steps are discussed below.

[0023] In step (1) (picking up), the position and posture of workpiece 230 are identified using a camera image. When workpieces flow in on a conveyor, one coordinate (such as Z) of the position of workpiece 230 is tracked according to the conveyor position index. The workpiece 230 shown in Figure 2 is a simple cube. However, any type or shape of workpiece 230 may be identified from the camera image. For example, if it is known that workpiece 230 is a specific injection-molded part and may have any random orientation and position on the conveyor, the image from camera 210 can be analyzed to identify the main features of workpiece 230, and from such features, the position and orientation of workpiece coordinate frame 232 can be determined. Workpiece 230 may alternatively be a component to be attached to an assembly. Here, the components are provided in any suitable manner (individually or in groups) for picking by a robot.

[0024] Furthermore, in step (1) (picking up), the camera image is also used to identify the position and posture of hand 240 when gripping workpiece 230. The image of hand 240 is analyzed by the method (two different techniques) discussed above with reference to Figure 1 and described in detail in the '185 application to determine the position and orientation of hand coordinate frame 242. Hand 240 approaches workpiece 230 in the direction indicated by arrow 244. When the tip of the thumb contacts workpiece 230 as shown by point 250 and the fingertips contact workpiece 230 (this contact point is not visible), controller 220 extracts all data from this specific image and stores it as picking data. The picking data includes the position and posture defined by workpiece coordinate frame 232 of workpiece 230, and the position and posture defined by hand coordinate frame 242 of hand 240.

[0025] In step (2) (movement), camera images are used to track the positions of both the workpiece 230 and the hand 240 as they move along the path 260. Multiple images of the workpiece 230 and the hand 240 are recorded to define the path 260. This path is not typically a straight line. For example, the path 260 may include a long arc or may cause the workpiece 230 to move over some kind of obstacle. In any case, the path 260 includes multiple points that define the position (and optionally the orientation) of the workpiece 230 and the hand 240 during the movement step. Using the same techniques discussed so far, the position and orientation of the workpiece coordinate frame 232 and the hand coordinate frame 242 are identified from the camera images. The technique for determining the orientation of the hand is also discussed in detail in the '185 application.

[0026] In step (3) (Installation), the camera image is used to install the workpiece 230 in its intended location, as indicated by arrow 270, and then to identify the final position and orientation of the workpiece 230 (as defined by its workpiece coordinate frame 232). The same technique discussed so far is used to identify the position and orientation of the workpiece coordinate frame 232 from the camera image. When the thumb and fingertips release contact with the workpiece 230, the controller 220 saves the workpiece coordinate frame data from this particular image as installation data. Installation data may also be recorded and saved based on the fact that the workpiece 230 has stopped moving, i.e., the workpiece coordinate frame 232 has remained in the exact same position and orientation for a certain period (e.g., 1-2 seconds).

[0027] The above step of determining the hand's pose from the 3D coordinates of the hand's principal points may be performed on a robot controller such as the controller 220 in Figure 2, or on a separate computer communicating with the controller 220. Patent 185 discloses a technique for calculating the 3D coordinates of the hand's principal points using various types and numbers of cameras. Figure 2 shows a human demonstration of the steps of a retrieve-move-set operation to define a trajectory during a robot teaching phase (without robot involvement), but a corresponding set of steps may be used by the robot (without human involvement) during the playback or robot execution phase, which will be discussed below.

[0028] Figure 3 is a diagram of a system 300 and steps for a robot to perform a pick-and-place operation using a camera image of a workpiece and programming previously taught by a human hand demonstration, as disclosed in the 185 application and used in the embodiments of this application. The system 300 is located within a work cell 302 which includes a camera 330 that communicates with a controller 340. Such items may be the same as, or different from, the work cell 202, camera 210, and controller 220 previously shown in Figure 2. In addition to the camera 330 and controller 340, the work cell 302 typically includes a robot 310 that communicates with the controller 340 via a physical cable 342. The robot 310 operates a gripping unit 320.

[0029] System 300 is designed to “reproduce” the retrieval, movement, and placement operations taught by a human operator in System 200. The hand and workpiece position data recorded during the retrieval, movement, and placement steps are used to generate robot programming instructions as follows: Robot 310 positions the gripping unit 320 in place, as is known to those skilled in the art. Camera 330 identifies the position and orientation of a workpiece 350, which may be positioned, for example, on a tray of parts. The workpiece 350 is shown in positions 350A and 350C in Figure 3.

[0030] From the teaching / demonstration stage discussed above, the controller 220 (Figure 2) recognizes the position and orientation of the hand 240 relative to the workpiece 350 (and consequently, the position and orientation of the gripping unit 320 from Figure 1) and appropriately grips the workpiece 350 ("removal" operation). Path 1 is calculated as the path that moves the gripping unit 320 from a fixed position to a removal position calculated based on the position and orientation of the workpiece 350. The removal operation is performed at the end of Path 1. The gripping unit 320 is shown as 320A approaching the removal location along Path 1, and the workpiece 350 is shown as 350A at the removal location.

[0031] From the teaching / demonstration stage, the controller 220 recognizes the positions of the workpiece 230 (and consequently the workpiece 350) at multiple locations along the movement path. Path 2 is calculated as the path that moves the gripping unit 320A and the workpiece 350A from the pick-up position along the movement path. In Figure 3, one of the intermediate positions along the movement path is the gripping unit 320B.

[0032] Route 2 terminates at the installation position recorded during the training / demonstration phase. This includes both the position and orientation (orientation) of the workpiece 350, corresponding to workpiece 350C. After the gripping unit 320 has installed the workpiece 350 in its installation location and orientation, the gripping unit 320 releases the workpiece 350 and returns it to its original position via Route 3.

[0033] The following is an overview of the techniques disclosed above for robot programming through human demonstration. • During the instruction / demonstration phase, human hands are used to demonstrate the gripping, movement, and placement of the workpiece. • By analyzing camera images of the hand and workpiece, the appropriate orientation and position of the robot gripping unit relative to the workpiece are determined based on key points of the hand identified from the camera images. • Using the appropriate orientation and position of the robot gripping unit relative to the workpiece, the system generates robot motion commands for the pick-up-move-place trajectory based on the camera image of the workpiece during the replay or execution phase.

[0034] This disclosure utilizes the retrieval-transfer-placement trajectory generation from human demonstrations detailed above, and adds IBVS data acquisition during the demonstration phase and IBVS control during the replay / execution phase to provide precise workpiece placement necessary for operations such as robot component assembly.

[0035] Figure 4 shows a basic concept of a robot teaching technique using human demonstration with an image-based visual servo (IBVS) according to one embodiment of the present disclosure. In box 410, a human demonstration of workpiece picking, moving, and placing is performed using camera monitoring, as shown in Figure 2. Box 420 shows the picking, moving, and placing trajectory as determined from the analysis of camera images of the human demonstration. The raw trajectory 422 depicts the actual hand movements in 3D space (coordinate frame representing the robot gripping part), including many small swaying and inaccurate movements typical of a human demonstration. The improved trajectory 424 is calculated from the raw trajectory 422 by moving straight up from the picking point to the maximum height, then horizontally to a point directly above the placement point at the maximum height, and then straight down to the placement point.

[0036] The trajectory for removal-movement-placement from box 420 (preferably an improved trajectory 424) is used in box 430 by a robot (not shown) during the regeneration or execution phase. This is described in some detail and shown in Figure 3. Rather than using trajectory 424 to place the workpiece in its final location, this disclosure adds IBVS tracking to the final approach section 432 of trajectory 424. IBVS tracking detects the geometric features of the workpiece in camera images of the robot execution and compares the actual location of the geometric features to the target or desired location. The robot controller uses the output signals from the IBVS tracking calculation to precisely control the final approach movement of the robot and the gripping unit, and consequently the final placement of the workpiece. Details of the IBVS calculation and augmentation as an overall method for improving reliability are discussed below.

[0037] Image-based visual servo (IBVS) is a technology that controls robot movement using feedback information (visual feedback) extracted from visual sensors. IBVS calculates control signals based on the error between the current location of a selected feature in the image plane and the target location. IBVS overcomes calibration errors and depth accuracy problems that often occur in robot control methods using 3D camera data.

[0038] The basic concept of IBVS is to identify the geometric features of an object (e.g., a workpiece) and guide the object's placement so that the geometric features coincide with a predefined target location of the feature in the image plane. In the simplest case, where the geometric feature is point A, point A has a location in the xy pixel coordinates of the current camera image, and the location of the target point B has another location in the xy pixel coordinates of the camera image plane. The robot moves the object, and the location of point A relative to the target point B is calculated in each successive camera image until point A coincides with the target point B.

[0039] The robot's motion is controlled as follows to achieve the movement of point A. The 2D (xy pixel) error of point A's location relative to the target point B's location is used to calculate the 2D image space velocity vector ΔX. The Jacobian matrix J relates the image space velocity vector ΔX to the 3D robot's Cartesian space velocity vector Δq. The Jacobian matrix J is known based on the camera projection parameters. The 2D image space velocity, the function determinant, and the 3D Cartesian space velocity are related as follows. ΔX = JΔq (1) Here, only the Cartesian space velocity vector Δq of the 3D robot remains unknown.

[0040] To calculate the Cartesian space velocity vector Δq of a 3D robot, equation (1) is rearranged as follows: Δq=J -1 ΔX (2) Here, J -1 This is the reciprocal of the Jacobian J.

[0041] Many different types of geometric features of an object (workpiece) may be tracked using IBVS. Such features include points, lines, circles, other ellipses, cylinders, and even irregular shapes. Here, positional errors can be calculated by double-integration image moment calculations. In general, the use of point features is not robust to changes in the actual robotic environment. For example, the apparent position of a point in image space may change depending on lighting conditions, and it can be difficult to separate or distinguish different feature points because many point features may be located very close to each other on most objects. On the other hand, IBVS calculations related to complex 2D and 3D shapes can be redundant, and many objects do not have usable geometric features such as circles and cylinders. Ultimately, the type of feature best suited to the specific object (workpiece) and installation application may be selected.

[0042] A widely examined example using the technology of this disclosure analyzes a CPU fan for a computer assembly. The external shape of a CPU fan is generally square (in a top view), and the inner shroud surface (surrounding the fan blades) is octagonal. This makes this area suitable for using IBVS line features. As will be shown later, the robot's task is to lift the CPU fan unit and mount it into the computer case. This task requires the CPU fan unit to be precisely positioned to align the pins and slots that need to be engaged when the CPU fan unit is pushed into place.

[0043] The target location of line features on a workpiece (e.g., a CPU fan unit) can be determined during a human demonstration of component assembly. For example, if the hand movement stops at the "installation" location, or if the hand moves away from the workpiece in the camera image and the workpiece comes to a standstill, it can be determined that the workpiece is in its final mounting position, and the location of the target line feature in image space can be calculated. The determination of the target line feature location is performed only once during the human demonstration phase. This determination requires no additional steps from the human demonstrator. It simply involves additional calculations based on the same human demonstration steps shown in Figure 2.

[0044] During robot execution, IBVS tracking is initiated when the gripping unit / workpiece is in the final approach section 432 of the trajectory 424 (Figure 4). At this point, selected line features on the workpiece begin to be tracked in the camera image. To ensure and efficiently identify the appropriate line features in the image, it is useful to first crop the camera image to remove any static background, i.e., remove anything that has not moved from one image to the next.

[0045] The basic idea of ​​moving object detection is to subtract the background model of the image from the current image, and any pixels whose difference exceeds a threshold are kept as a foreground mask of the image, representing what has changed in the current image. It is known that modeling the background using a Gaussian mixture model improves the robustness of moving object detection. Here, a Gaussian distribution is calculated, and pixels within a certain statistical distance from the mean (e.g., + / - 2 sigma) are identified as unchanged, while pixels outside a predefined statistical distance are identified as changed.

[0046] Moving object detection can be calculated from 2D red / green / blue (RGB) image data or depth data from a 3D camera. In some environments, RGB image data suffers from the problem of shadows cast by human performer hands or workpieces. Shadows appear as moving objects and add unwanted extra image space to the cropped area being analyzed. Depth data does not have the problem of shadows, but is often noisy and has many small spots or pixel clusters that appear to change from image to image. In a preferred embodiment, moving object detection is performed separately on the RGB data and depth data at each image time step, and the boundaries of the image cropping are defined using pixels identified as moving in both the RGB data and depth data (by an "AND" operation). This technique has been shown to work well to provide cropped images that include human hands and objects / workpieces, rather than shadows or noise.

[0047] Another technique that has proven useful in improving the efficiency and robustness of IBVS tracking involves cropping an image using moving object detection and then removing line features of a human performer's hand. This can be done by utilizing the hand detection technique shown in Figure 1 and described in detail in the '185 application. Specifically, after identifying the major structural points of the fingers and thumb (such as the knuckles) in the image, line features of the hand can be removed in two steps using knowledge of the hand's position in the image. In the first step, line features located between the index finger and little finger of the hand are removed from the cropped image. This first step may potentially eliminate many line features associated with the edges of the fingers that are obviously not useful for IBVS positioning of the object (CPU fan). In the second step, line features that are within an acceptable width range (δ) of the index finger or thumb are removed. The idea of ​​the second step is to remove line features associated with the edges of the index finger and the thumb themselves. For example, if a line is detected in the image and that line is within the distance tolerance of the index finger's centerline defined by the joint principal points (i.e., d < δ), then that line is ignored in the IBVS calculation. The same concept applies to lines near the thumb in the second step. Removing line features from the hand speeds up the IBVS calculation of the cropped image because the removed lines do not need to be evaluated to determine whether they are one of the lines on the object (CPU fan) that needs to be tracked for the IBVS-based placement of the object.

[0048] Figure 5A is an image 500 of a workpiece being mounted to an assembly by a human demonstrator. The workpiece (CPU fan) is being lifted by the demonstrator's hand from a location indicated by box 510, which is positioned laterally from the assembly (computer case). The workpiece is then moved and installed in its designated location within the computer case, including alignment with electrical connectors and mechanical features such as pins / holes. The workpiece is shown in its final installation location within box 520. As discussed above, the human demonstration phase provides a basic retrieval-movement-installation trajectory for the gripping part and the workpiece, as well as advantageous conditions for collecting line feature data during the movement of the workpiece to identify the desired or target location of the line features when the workpiece is mounted in its final installation position.

[0049] Figure 5B is Image 550, a cropped version of the image from Figure 5A, showing line features identified for an image-based visual servo according to one embodiment of the present disclosure. Image 550 has been cropped using the moving object detection technique as described above, so that the cropped portion of Image 500 essentially contains only the performer's hand and the workpiece (CPU fan). After cropping as shown, line features associated with the performer's hand are removed as described above. This removal includes the removal (ignoring) of any line features between the index and little fingers of the hand, and the removal (ignoring) of line features within an acceptable distance of the thumb or index finger of the hand. The line features remaining in the cropped Image 550 are used for IBVS tracking as discussed above. In Image 550, lines 560, 562, 564, and 566 are identified. The lines are all geometric features of the outer frame and shroud of the CPU fan. Such lines are precisely the type of geometric features necessary for IBVS tracking.

[0050] Throughout the human demonstration phase shown in Figure 5A, images are cropped and analyzed to collect geometric feature data. The target or desired location of the geometric feature (e.g., lines 560-566) is identified when the movement of the hand and workpiece stops and the workpiece is in its final mounting position, as shown in Figure 5B. The line feature data collected during the human demonstration is then used for IBVS tracking during the robot's replay or execution phase, as discussed below.

[0051] Figure 6 shows details of a technique for human demonstration-based robot teaching using an image-based visual servo (IBVS) according to one embodiment of the present disclosure. Human demonstration stage 600 includes a human demonstration of an retrieval-transfer-placement operation in box 602. The hand posture is identified in the image in box 610 at each image frame interval. Here, the hand posture is converted to the posture of the corresponding grasping part, as discussed above with reference to Figures 1 and 2. In addition, as described above, after the demonstration is complete, the 3D retrieval-transfer-placement trajectory is determined overall in box 620. The resulting trajectory is shown in 622.

[0052] Furthermore, during the human demonstration phase 600, box 630 collects geometric features on the workpiece from the camera image for image-based visual servos. In the CPU fan example discussed above, line features were selected as the target for IBVS tracking. Other types of geometric features may also be used, based on the most appropriate features present on the workpiece. Box 640 captures a target or desired line feature that represents the line feature at the final mounting position of the workpiece. The target or desired line feature is captured when trajectory generation in box 620 determines that the workpiece has reached its "installation" location (stopped moving), as indicated by line 624. The steps performed in boxes 630 and 640 are described above in the discussion in Figures 5A and 5B. The target or desired line feature is a 2D line in coordinates on the camera's image plane.

[0053] The robot execution (or replay) stage 650 is performed using a system as shown in Figure 3. The overall retrieval-movement-placement trajectory from box 620 is provided by line 652 to the robot controller that runs the robot motion program in box 660. The robot motion program run in box 660 causes the robot to retrieve, move, and place the workpiece using a trajectory 662 that corresponds to the trajectory 622 captured during the human demonstration. The trajectory 662 used by the robot may be a simplified version of the trajectory 622 captured during the demonstration (e.g., "up, across, down").

[0054] In box 670, the robot controller determines that the robot gripping unit has reached a designated approach position and is approaching the "installation" location. This is shown in Figure 4 and discussed above. Upon reaching the approach position, the image-based vision servo is activated, as indicated by point 672 on track 662. In box 680, IBVS is performed as the robot moves the workpiece in its final approach to the "installation" location. During IBVS, the camera image of the workpiece is continuously analyzed to determine the current position of the selected line feature. The target or desired line feature from box 640 is provided to the robot controller on track 654 as shown. Robot control signals are generated based on the difference (in camera image plane coordinates) between the current position of the selected line feature and the target or desired position of the line feature. The IBVS control signals allow the robot gripping unit to install the workpiece in its final installation position (shown in box 690) with much greater precision than would be possible using track 662 alone.

[0055] The robot execution stage 650 can be performed multiple times based on a single human demonstration stage 600. For example, in the CPU fan mounting application considered herein, after the robot mounts the workpiece in box 690, a new computer enclosure (requiring a CPU fan) is positioned fixed relative to the IBVS camera, another CPU fan is grasped by the robot (whether multiple CPU fans are available in the tray for selection or only one is available), and the steps in boxes 660, 670, 680, and 690 are repeated. The robot execution stages can be repeated indefinitely for the trajectories and desired line features captured in boxes 620 and 640, respectively.

[0056] Figure 7 is a flowchart 700 of a method for human demonstration-based robot teaching using an image-based visual servo, including a demonstration phase and a robot execution phase, according to one embodiment of the present disclosure. The steps numbered 702–712 on the left side of Figure 7 represent the human demonstration phase 600 in Figure 6 and earlier. The steps numbered 720–726 on the right side of Figure 7 represent the robot execution phase 650 in Figure 6 and earlier.

[0057] Box 702 demonstrates operations such as pick-up-move-placement, which are performed by a human grasping a workpiece, moving it along a desired trajectory, and placing it in a desired location. The step of placing it in a desired location may include a step of attaching or assembling the component workpiece to a separate structure. Box 704 analyzes images of the demonstration from one or more cameras to detect key points of the hand and the workpiece in the images. The correlation between the detection of key points of the hand and the grasping position of the key points is shown in Figure 1 and has been discussed previously.

[0058] Box 706 adds a new step to the movement trajectory based on the latest image analysis of the hand and workpiece. Details of trajectory generation, including the detection of grasping actions at the retrieval point and releasing actions at the placement point, are described in the '185 application and outlined herein above. Box 708 captures the geometric features of the workpiece (such as the line features on the CPU fan discussed above) for the latest camera image. Details of capturing geometric features during a human demonstration of the operation are discussed above with reference to Figures 5 and 6, and in particular shown in Box 630 in Figure 6.

[0059] The determination rhombus 710 determines whether the installation (final) location has been reached. As discussed previously, reaching the installation location (endpoint of the trajectory) can be identified by the hand and / or workpiece coming to rest in the camera image. The installation location can also be identified by the fingertips of the hand moving away from the workpiece, as described in the '185 application. If the installation location has not been reached, the demonstration continues, and the process returns to box 704 to analyze the next camera image. If the installation location has been reached, the process moves from the determination rhombus 710 to box 712, where the trajectory is determined and the location of the target (or desired) geometric feature is captured. Both the trajectory and the location of the target geometric feature are used in the robot execution phase, as will be explained in detail with reference to Figure 6.

[0060] In Box 720, during the robotic execution of a demonstrated operation (e.g., workpiece pick-up-movement-placement), the robot controller executes the robotic program, causing the robotic gripper to lift the workpiece and move it along the trajectory generated during the human demonstration. The robotic program calculated by the controller uses inverse kinematics calculations, etc., as will be understood by those skilled in the art, to translate the position and orientation of the gripper at various points along the trajectory into robotic joint velocity commands.

[0061] Decision rhombus 722 determines whether the final approach position has been reached in the trajectory. This concept is shown in box 430 in Figure 4 and reconsidered with reference to box 670 in Figure 6. If the final approach position has not yet been reached, the process returns to box 720 and continues following the demonstrated trajectory. If the final approach position is reached in decision rhombus 722, the image-based visual servo (IBVS) is activated in box 724.

[0062] In box 726, the robot control unit uses IBVS control signals to position the robot gripping unit when maneuvering the workpiece to its final installation or mounting position. The IBVS control signals are calculated based on the difference between geometric features identified in the current camera image and the location of the target or desired feature in the image plane coordinates. This technique has been further described with reference to Figures 5 and 6. The process ends when the robot places the workpiece in its final installation or mounting position. The robot execution phase (steps numbered 720-726 in Figure 7) may be repeated multiple times based on data taken from a single instance of human demonstration.

[0063] Throughout the discussion so far, various computers and controllers have been described and implied. It should be understood that the software applications and modules for such computers and controllers run on one or more computing devices having processors and memory modules. In particular, this includes the processors in the robot controllers 220 and 340 discussed above, along with the optional separate computer discussed with respect to Figure 2. Specifically, the processor in the controller / computer is configured to perform robot teaching via human demonstration, IBVS data acquisition during the teaching phase, and IBVS control of workpiece placement during the execution phase in the manner discussed above.

[0064] As outlined above, the disclosed technique of human-demonstrated robot teaching using image-based visual servos (IBVS) makes robot motion programming faster, easier, and more intuitive than previous techniques, while providing the workpiece placement precision necessary for assembling robot components.

[0065] Having considered above many exemplary aspects and embodiments of human-demonstrated robot teaching using image-based visual servos (IBVS), those skilled in the art will recognize modifications, substitutions, additions, and partial combinations thereof. Therefore, the appended claims and the claims introduced below are intended to be interpreted as including any modifications, substitutions, additions, and partial combinations that fall within their true spirit and scope. [Aspect 1] A method for programming a robot to perform operations through human demonstration, wherein the method is A step in which the aforementioned operation on the workpiece is demonstrated by human hands, The steps include: analyzing camera images of the hand and the workpiece using a computer to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: a robot controller executing a program to cause a robot having a gripping unit to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); A method comprising the steps of: the robot controller maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature. [Aspect 2] The method according to embodiment 1, wherein the geometric features on the workpiece include one or more of points, lines, arcs, ellipses, cylinders, and irregular shapes. [Aspect 3] The method according to embodiment 1, wherein the geometric feature on the workpiece, the current location of the geometric feature, and the target location are identified by the image plane coordinates of the camera image. [Aspect 4] The method according to embodiment 1, wherein the demonstration of the operation is completed when it is determined that the movement of the workpiece, the hand, or both has stopped. [Aspect 5] The method according to embodiment 1, wherein the step of identifying geometric features on the workpiece in the demonstration data includes the step of cropping each image frame using moving object detection so that it includes only the portion of the image frame that has changed from a previous image frame. [Aspect 6] The method according to embodiment 5, wherein the moving object detection is performed using both two-dimensional (2D) color / intensity data and three-dimensional (3D) depth data, and the image frame is cropped to include pixels that have changed in both the 2D data and the 3D data. [Aspect 7] The method according to embodiment 1, wherein the step of identifying geometric features on the workpiece in the demonstration data includes the step of removing the geometric features of the human hand before identifying the geometric features of the workpiece. [Aspect 8] The method according to embodiment 7, wherein the step of removing the geometric features of the human hand includes the step of removing geometric features located between the index finger and the little finger of the hand, and the step of removing geometric features located within a predetermined distance tolerance of the index finger or thumb of the hand. [Aspect 9] The hand coordinate frame is calculated based on the principal points of two or more fingers of the hand identified in the demonstration data, wherein the principal points include finger joints, according to embodiment 1. [Aspect 10] The method according to embodiment 1, wherein the step of maneuvering the gripping portion using IBVS includes the step of generating a robot control signal based on the difference in image plane coordinates between the current location of the geometric feature and the location of the target. [Aspect 11] The method according to Embodiment 1, wherein the step of demonstrating the operation includes the step of attaching the workpiece to the assembly via the engagement of mechanical features, and the IBVS control by the robot controller causes the gripping part to attach the workpiece to the assembly via the engagement of mechanical features. [Aspect 12] The method according to embodiment 11, further comprising the step of the controller repeatedly executing the program and maneuvering the gripping portion using IBVS control for new instances of the workpiece and the assembly. [Aspect 13] A method for programming a robot to perform installation operations through human demonstration, wherein the method is: A step of demonstrating the operation of attaching the workpiece to the assembly by human hands, The steps include: analyzing camera images of the hand and the workpiece using a computer to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, Once the demonstration of the mounting operation is complete by mechanically fitting the workpiece into the assembly, the steps include determining the trajectory and capturing the target location of the geometric features, The robot controller executes a program to cause a robot having a gripping unit to perform the attachment operation on the workpiece using the trajectory, The steps include: when it is determined that the final approach position within the aforementioned trajectory has been reached, start image-based visual servo (IBVS); A method comprising the steps of: maneuvering the gripping portion until the mounting operation is completed by mechanically fitting the workpiece into the assembly using IBVS control by the robot controller, wherein the step of using IBVS control includes calculating a control signal based on the difference between the current location of the geometric feature on the workpiece and the target location of the geometric feature. [Aspect 14] A system for programming a robot to perform operations through human demonstration, wherein the system is Camera and, A robot having a gripping part, A robot controller having a processor and memory, wherein the controller communicates with the robot and receives images from the camera, and the controller The steps include: analyzing camera images of a human hand demonstrating the operation on the workpiece to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: executing a program to cause the robot having the gripping part to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); A system comprising: a robot controller configured to perform a step including: maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature. [Aspect 15] The system according to embodiment 14, wherein the geometric features on the workpiece include one or more of points, lines, arcs, ellipses, cylinders, and irregular shapes. [Aspect 16] The system according to embodiment 14, wherein the step of identifying geometric features on the workpiece in the demonstration data includes the step of cropping each image frame using moving object detection so that it includes only the portion of the image frame that has changed from a previous image frame. [Aspect 17] The system according to embodiment 14, wherein the step of identifying geometric features on the workpiece in the demonstration data includes a step of removing geometric features of the human hand before identifying the geometric features on the workpiece, the step of removing geometric features located between the index finger and little finger of the hand, and the step of removing geometric features within a predefined distance tolerance of the index finger or thumb of the hand. [Aspect 18] The system according to embodiment 14, wherein the step of maneuvering the gripping portion using IBVS includes the step of generating a robot control signal based on the difference in image plane coordinates between the current location of the geometric feature and the location of the target. [Aspect 19] The system according to embodiment 14, wherein the step of demonstrating the operation includes the step of attaching the workpiece to the assembly via the engagement of mechanical features, and the IBVS control by the robot controller causes the gripping part to attach the workpiece to the assembly via the engagement of mechanical features. [Aspect 20] The system according to embodiment 19, further comprising the step of executing the program and continuously repeating the operation of the gripping portion on new instances of the workpiece and the assembly using IBVS control.

Claims

1. A method for programming a robot to perform operations through human demonstration, wherein the method is A step in which the aforementioned operation on the workpiece is demonstrated by human hands, The steps include: analyzing camera images of the hand and the workpiece using a computer to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: a robot controller executing a program to cause a robot having a gripping unit to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); The steps include: the robot controller maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature; A method comprising the step of identifying geometric features on the workpiece in the demonstration data, the step of cropping each image frame using moving object detection so that it includes only the portion of the image frame that has changed from a previous image frame.

2. The method according to claim 1, wherein the geometric features on the workpiece include one or more of points, lines, arcs, ellipses, cylinders and irregular shapes.

3. The method according to claim 1, wherein the geometric feature on the workpiece, the current location of the geometric feature, and the target location are identified by the image plane coordinates of the camera image.

4. The method according to claim 1, wherein the demonstration of the operation is completed when it is determined that the movement of the workpiece, the hand, or both has stopped.

5. The method according to claim 1, wherein the moving object detection is performed using both two-dimensional (2D) color / intensity data and three-dimensional (3D) depth data, and the image frame is cropped to include pixels that have changed in both the 2D data and the 3D data.

6. The method according to claim 1, wherein the step of identifying geometric features on the workpiece in the demonstration data includes the step of removing the geometric features of the human hand before identifying the geometric features of the workpiece.

7. The method according to claim 6, wherein the step of removing the geometric features of the human hand includes the step of removing geometric features located between the index finger and the little finger of the hand, and the step of removing geometric features located within a predetermined distance tolerance of the index finger or thumb of the hand.

8. The method according to claim 1, wherein the hand coordinate frame is calculated based on the principal points of two or more fingers of the hand identified in the demonstration data, and the principal points include the finger joints.

9. A method for programming a robot to perform operations through human demonstration, wherein the method is: A step in which the aforementioned operation on the workpiece is demonstrated by human hands, The steps include: analyzing camera images of the hand and the workpiece using a computer to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: a robot controller executing a program to cause a robot having a gripping unit to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); The steps include: the robot controller maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature; A method for maneuvering the gripping portion using IBVS, comprising the step of generating a robot control signal based on the difference in image plane coordinates between the current location of the geometric feature and the target location.

10. The method according to claim 1, wherein the step of demonstrating the operation includes the step of attaching the workpiece to the assembly via the engagement of mechanical features, and the IBVS control by the robot controller causes the gripping portion to attach the workpiece to the assembly via the engagement of mechanical features.

11. The method according to claim 10, further comprising the step of the robot controller repeatedly executing the program and maneuvering the gripping portion using IBVS control for new instances of the workpiece and the assembly.

12. A method for programming a robot to perform installation operations through human demonstration, wherein the method is: A step of demonstrating the operation of attaching the workpiece to the assembly by human hands, The steps include: analyzing camera images of the hand and the workpiece using a computer to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, Once the demonstration of the mounting operation is complete by mechanically fitting the workpiece into the assembly, the steps include determining the trajectory and capturing the target location of the geometric features, The robot controller executes a program to cause a robot having a gripping unit to perform the attachment operation on the workpiece using the trajectory, The steps include: when it is determined that the final approach position within the aforementioned trajectory has been reached, start image-based visual servo (IBVS); A step of maneuvering the gripping portion until the mounting operation is completed, by using IBVS control by the robot controller to mechanically fit the workpiece into the assembly, wherein the step of using IBVS control includes a step of calculating a control signal based on the difference between the current location of the geometric feature on the workpiece and the target location of the geometric feature, A method comprising the step of identifying geometric features on the workpiece in the demonstration data, the step of removing the geometric features of the human hand before identifying the geometric features of the workpiece.

13. A system for programming a robot to perform operations through human demonstration, wherein the system is Camera and, A robot having a gripping part, A robot controller having a processor and memory, wherein the robot controller communicates with the robot, receives images from the camera, and the robot controller The steps include: analyzing camera images of a human hand demonstrating the aforementioned operation on a workpiece to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: executing a program to cause the robot having the gripping part to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); A robot controller configured to perform steps including: maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature; The system includes the step of identifying geometric features on the workpiece in the demonstration data, which includes the step of cropping each image frame using moving object detection so that it includes only the portion of the image frame that has changed from a previous image frame.

14. The system according to claim 13, wherein the geometric features on the workpiece include one or more of points, lines, arcs, ellipses, cylinders, and irregular shapes.

15. A system for programming a robot to perform operations through human demonstration, wherein the system is: Camera and, A robot having a gripping part, A robot controller having a processor and memory, wherein the robot controller communicates with the robot, receives images from the camera, and the robot controller The steps include: analyzing camera images of a human hand demonstrating the aforementioned operation on a workpiece to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: executing a program to cause the robot having the gripping part to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); A robot controller configured to perform steps including: maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature; A system that includes the step of identifying geometric features on the workpiece in the demonstration data, which includes the step of removing geometric features of the human hand before identifying the geometric features on the workpiece, the step of removing geometric features located between the index finger and little finger of the hand, and the step of removing geometric features within a predefined distance tolerance of the index finger or thumb of the hand.

16. A system for programming a robot to perform operations through human demonstration, wherein the system is: Camera and, A robot having a gripping part, A robot controller having a processor and memory, wherein the robot controller communicates with the robot, receives images from the camera, and the robot controller The steps include: analyzing camera images of a human hand demonstrating the aforementioned operation on a workpiece to create demonstration data; Based on the hand coordinate frame identified in the aforementioned demonstration data, the step of adding a new point to the trajectory at each image frame interval, The steps include identifying geometric features on the workpiece in the demonstration data at each image frame interval, When the demonstration of the operation described above is completed, the steps include determining the trajectory and capturing the target location of the geometric features, The steps include: executing a program to cause the robot having the gripping part to perform the operation on the workpiece using the trajectory; The steps include: when it is determined that the final approach position has been reached within the aforementioned trajectory, start an image-based visual servo (IBVS); A robot controller configured to perform steps including: maneuvering the gripping portion using IBVS control until the current location of the geometric feature on the workpiece coincides with the target location of the geometric feature; The system includes the step of maneuvering the gripping part using IBVS, which involves generating a robot control signal based on the difference in image plane coordinates between the current location of the geometric feature and the target location.

17. The system according to claim 13, wherein the step of demonstrating the operation includes the step of attaching the workpiece to the assembly via the engagement of mechanical features, and the IBVS control by the robot controller causes the gripping part to attach the workpiece to the assembly via the engagement of mechanical features.

18. The system according to claim 17, further comprising the step of executing the program and continuously repeating the operation of the gripping portion on new instances of the workpiece and the assembly using IBVS control.

Images