Method and system for training a robot

Abstract

Methods and systems for training a robot to recognize objects in a workspace of the robot are provided. Objects in the workspace are identified by a user, and the robot determines candidate objects. Feedback may be used so that the user can confirm whether the candidate objects determined by the robot system match the object intended by the user. Gripping information for the object may also be identified by the user to train the robot how to grip the object.

Description

Background Art

[0001] The present invention relates generally to robots, and more particularly to defining objects in a workspace and gripping locations on the objects.

[0002] Object interaction, such as picking and manipulation, is at the core of most robotic tasks. In operation, the robot needs to know where to find the object of interest and how to handle the object, such as where and how to grip it. Traditional robotic programming is done by experts using specialized programming languages ​​and operating systems. These tools require a deep understanding of the robot and its capabilities. Therefore, in conventional robotic programming, the expert's task is to define the location and / or gripping information of the object, usually in the form of coordinates in the robot's workspace relative to the robot.

[0003] Although object definition for the robot and programming the interaction between the robot's gripper and the object are necessary for using the robot, such tasks are time-consuming and usually require expertise. Many robotic tasks involve object gripping and some physical manipulation of the object, such as placing and / or manipulating the object. However, the robot is controlled by low-level commands, typically in a specific and specialized programming language. Such robot programming languages ​​are usually designed for expert users.

[0004] Therefore, improved methods and systems for defining workspace objects and clamping locations thereon would be useful. Summary of the invention

[0005] A method and system are described for training a robotic system to recognize objects in a workspace and gripping information for the objects. Objects can be identified by a user using a touch screen, using gestures in the workspace, or using language-based input. Feedback can be used to confirm that the robotic system has recognized the object that the user wanted to select. The user can also train the robotic system to recognize gripping information that the user wants the robot to use when gripping the recognized object. The present invention may also include any other aspects described in the following written description or in the accompanying drawings, and any combination thereof.

[0006] In addition, a system configured for training a robotic system by a user according to any of the methods described herein is provided. The system includes: a determination module configured for determining a candidate object within a workspace of the robot by the robotic system; a display module configured for displaying the candidate object to a user; and an identity learning module configured for receiving a user's identification input indicating whether the candidate object matches an expected object. In addition, a use of a system for training a robotic system by a user according to any of the methods described herein is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present invention may be more fully understood by reading the following description in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is a schematic diagram of the components of the robot training system;

[0009] Figure 2 is a schematic diagram of an object registration module; and

[0010] Figure 3 is a schematic diagram showing the clamping positions being identified. DETAILED DESCRIPTION

[0011] The improved methods and systems of this article enable intuitive human-robot interaction for defining objects and their gripping points. Defining new objects and gripping positions is crucial for flexibly configuring robots for new tasks. The described methods and systems are based on visual mechanisms and use improved interaction patterns for object teaching and gripping information definition. Therefore, the methods and systems enable non-experts to perform these parts of robot programming without requiring users to write software code in a specialized robot programming language. Instead, users can intuitively interact with collaborative robots to complete such tasks. Simplified object definitions enable robotic automation to be applied to high-mix, low-volume applications, where high customizability requires frequent reconfiguration of robot tasks. Newer systems based on fully automatic machine learning techniques (e.g., dex-net) can provide automatic information definition, but this approach does not guarantee successful identification of gripping information for objects. Therefore, an improved solution is needed to define workspace objects and gripping information.

[0012] Improved methods and systems enable non-expert users to quickly define new objects, define gripping information for unknown objects, and more generally define key positions and actions of robot grippers. The interactive mode for object information definition is based on an alternative mode of interaction between the robot's visual system and the user. The user interaction can be based in part on gestures and can include a dedicated physical device called a "wand" that is physically and visually similar to a typical gripper. By imitating a robot gripper, the wand provides an intuitive similar interface that is easy for the user to understand. If a gripper with significantly different features than the wand is installed on the robot, such as a gripper with three or more fingers and a wand with two fingers, the system can convert the gripping points identified by the wand into specific features of the robot gripper. The wand can be guided by the hands of a non-expert robot programmer to replace the gripper of an actual robot. This interactive mode allows the user to define positions, specify objects, and define required gripping information by manipulating the wand in a manner similar to that of the robot using an actual gripper attached to the robot. The rod can be tracked and its position and orientation in space evaluated by the robot's vision system or by an internal inertial measurement unit (IMU) within the rod. The rod can also be used to define additional properties, such as the gripping force applied on the object. In general, the rod can simulate any type of robotic gripper (e.g., a camera for automated quality inspection, a welding tool, or a screwdriver).

[0013] The method and system use a primarily vision-based system for object detection and location. In some embodiments, a rod can be used for object detection and location. A touch display can be used for some interaction modes that enable a tight feedback loop with the user to define objects on the screen, and / or enable rapid correction of erroneously detected objects and defined object information by simply confirming or directly canceling on the screen. Additionally, the rod can include buttons or touch sensitivity for easier interaction with the user.

[0014] The improved method and system provide interactive teaching of object identity and gripping information to a robot. Two main subsystems are provided, one subsystem includes object identity learning, and one subsystem includes object gripping information teaching. The two subsystems can share some of the same components to achieve their functions and can be supplemented by a common feedback module. The described modules can also be used separately, but it is preferred that the two modules are used together to provide optimal performance.

[0015] Depending on the specific interaction mode desired (e.g., a speech module for language-based object definitions), the system may use a number of different components. Figure 1As shown, such components may include a robot camera system 10, an object recognition unit 12, an object registration interface 14, a visual feedback system 16 (e.g., for touch interaction and feedback system), a gesture recognition unit 20 ( Figure 2 ) (e.g., for finger-based grip point definition) and interaction tools, such as a special gripper-like device lever 38 (although an alternative could be to use the user's fingers 36 to form the gripping gesture).

[0016] Object learning can be based on a two-stage visual approach. First, an object recognition module finds possible candidate objects using a general object detection algorithm. Candidate objects can be located by a bounding box and can be uniquely identifiable internally. The object recognition module is preferably substantially invariant under object translation and orientation changes, so that candidate objects can be reliably found again in the workspace. That is, the object recognition module is able to correctly identify objects regardless of their location or orientation in the workspace. Changes in lighting conditions are preferably handled to a degree that is practically relevant to the entire work area of ​​the robot. Candidate objects may have common visual attributes, such as common object classifications (e.g., "bottle"), color ("yellow"), surface finish (e.g., "glossy"), and relative size.

[0017] Object teaching then involves assigning names to candidate objects to enable full object detection and identification for further interaction in the robot task. Assigning names to candidate objects can occur through interaction with the user. If the system requires input of position, orientation, and gripping information for a specific named object (e.g., after the user has asked the robot to perform an action on a specified object), the user may be asked to enter the name of the candidate object in the form of a name, such as the user's name. Figure 2 Objects are specified in one of three ways as shown: 1) by selection on a touch screen; 2) by physical gestures using a stick or a user's finger; or 3) by verbal description. Object learning may result in single object names or in naming entire classes of objects, which may lead to the possibility of working with many such objects, such as a specific pen and all pens. In touch interaction mode 18, the system may display the robot camera image with candidate objects 30 indicated to the user. Figure 2In the gesture mode 20, the candidate objects 30 include pens and bottles, but can be any type of object. The candidate objects 30 can be pre-selected by the system (e.g., by highlighting 32 the candidate objects 30 or using a bounding box 32 around the candidate objects 30). The user can then select the intended object 30 by touching the intended object 30 on the touch screen display 18. The touch interaction can also allow multiple selections using multi-touch selections, either by selecting a mode switch or by drawing a selection box. In the gesture mode 20, the user points to the object 30 to be selected within the workspace in the robot camera field of view using a user finger or hand 36 or a rod 38. The system interprets the gesture by extending the pointing direction to the objects in the workspace and selecting the object closest to the intersection of the pointing ray. If there is no object within a predetermined maximum allowed distance from the intersection, the system can ignore the gesture. In the language-based definition mode 22, the user defines the object using one or more words 40 in a natural language. The language definition involves the user specifying one or more attributes of the object candidate using the words 40, such as "yellow bottle". These attributes may include color, object classification, surface finish, (relative) position ("front left"), orientation ("facing right"), (relative) size ("smallest"), etc. The module may provide feedback about matching candidates immediately on the screen or through text-to-speech output. The user may either explicitly accept multiple objects selected by predefined keyword triggers (e.g., "all yellow bottles"), or the system may continue to request more attributes until a unique selection is achieved. During the specification process, the system may save and use contextual information about the attributes that have been specified.

[0018] The learned objects can be fed back to the user by displaying the named object on the screen (perhaps with a colored bounding box). This can be part of the feedback system described below. The object teaching system can also be used to define positions. Positions are not tied to movable objects, but are spatially fixed locations in the workspace, such as "front right corner". This process is particularly suitable for using a stick to point to locations in the workspace.

[0019] In order to successfully pick up and manipulate objects by the robot, it is preferred that the user also teaches the robot information about object gripping. Figure 3 As shown, the object gripping information includes at least the position of the point 42 to be gripped by the robot gripping tool and the relative orientation of the gripping tool relative to the object. The gripping information can also be described in other equivalent ways, such as by specifying a gripping axis and a relative rotation of the tool around the gripping axis. The gripping information can be defined by 1) using a rod 38 as a specified tool, or 2) a user's finger 36 imitating a robot gripper.

[0020] The rod can be a dedicated tool that is constructed to physically and visually resemble a standard robot gripper. Unlike an actual robot gripper, the rod is not attached to the robot, but is a loose, unattached component. The rod can be easily held by the user's hand. The user holds the rod so that it simulates the position and orientation of the original robot gripper. For the user, the rod tool is easier to handle than guiding the actual robot gripper with the robot's arm ("lead-in"). The position and orientation of the rod in space are used to derive the desired gripping information so that the actual robot gripper approaches the object, which is in the same spatial position and orientation as the rod is held when the gripping information is defined. As described above, due to the possible different characteristics between the rod and the actual gripper, the gripping information can be adjusted to adapt to the specific actual gripper used on the robot. The position and orientation of the rod can be tracked externally by a vision system. Alternatively, the rod can derive its position and orientation by an internal tracking device, such as by an IMU (Inertial Measurement Unit), and transmit it back to the system. The user interface on the rod can also include buttons or touch-sensitive surfaces.

[0021] The user can also provide gripping information by mimicking the robot gripper with the user's hand. This may involve a gesture of the index finger and thumb forming a gripper shape. The position of the fingers and the orientation of the hand can be tracked visually using a gesture recognition system. Alternatively, the gesture can be tracked by specific body sensors, such as a glove or wristband equipped with an IMU. The gripping information can be derived and calculated so that the robot approaches the object with the gripping tool at the touch point of the fingers and the object, and replicates the relative orientation of the hand with respect to the object / gripping axis.

[0022] The rod enables the user to intuitively and naturally define the clamping information through analog gestures. Since the rod is physically similar to the clamping tool, such as similar physical dimensions, the rod allows the user to intuitively understand the physical limitations of the clamping, such as the maximum opening of the clamping arm or the physical inaccessibility of certain positions due to spatial object constraints (non-convex object shape, unreachable parts, points close to the workbench, etc.).

[0023] The system for defining the clamping information can also be used as a fallback module for fully automatic clamping information determination using modern AI solutions such as dex-net. Automatic methods occasionally fail, for example, in cases where the mass distribution in the object is not uniform, and in such cases the user is still required to manually define the clamping information.

[0024] Depending on the specific use, the rod can include additional functions that enable intuitive robot programming. For example, the rod can implement advanced measurements such as touch force, which is then wirelessly transmitted to the robot together with the clamping position. This enables the system to define more detailed clamping information, including clamping force in addition to the clamping position. In the case where the robot arm is used to position a camera in a position to take pictures (for example for inspection), the rod can project the camera's field of view onto the object to be inspected to facilitate the definition of the pose for the robot arm.

[0025] The feedback system can visually display information about the defined object and gripping information on the display. Figure 2 As shown, defined objects can be shown with their names 34 (or question marks if a name 34 has not yet been assigned), and different objects / object classes can be highlighted using different colors. Using the same color to display multiple objects belonging to the same selection allows the user to quickly identify groups of objects. Objects can be displayed overlapping on the camera image to allow the user to see the full context of all objects. Clamping information can also be displayed by drawing clamping points or axes and orientations on the display screen.

[0026] The touch screen display may allow the user to interact and redefine the necessary information in several ways. For example, a long touch press on a named object may cause the system to enter a correction mode with the following subsequent actions: touching and sliding the object may delete the association of the name and the object; and briefly touching the object again may restart the object teaching for the specified name. Thus, the correction mode removes the current association of the name and the object, and may immediately start the object teaching with the same name again.

[0027] The system can also be implemented using additional visualization channels. For example, the visualization can use the same elements described above, but can be presented in 3D space and can also be displayed using a virtual reality device. Using an augmented reality device can replace touch interaction, and the highlighted name and grip information can be directly overlaid on the object in virtual reality.

[0028] The feedback system can also be used to display situations to the user where satisfaction may be violated. For example, a defined gripping point may be unreachable for the robot, for example due to a motion conflict with the workbench or the object itself (if the 3D shape is known). In this case, the display of the gripping information can indicate this situation by, for example, a red color of the gripping point and additional text or symbol indications. These conditions can also be communicated to the user using speech-to-text output.

[0029] The display of grip point information may also include accuracy information. The uncertainty of grip points may involve inaccuracies due to camera calibration, rod tracking, robot position accuracy (if known). The system may combine all sources of uncertainty using probabilistic reasoning techniques (e.g., Markov chain Monte Carlo methods) and display the resulting grip inaccuracy as a numerical percentage or a colored semi-transparent cloud around the defined points. This information may immediately provide valuable information to the user and may lead the user to redefine the points or fix other sources of uncertainty.

[0030] As described above, the method of training a robotic system by a user may include determining, by the robotic system, a candidate object 30 within the robot's workspace. The candidate object 30 may then be displayed to the user. By viewing the display, the user may then identify to the robotic system whether the candidate object 30 matches the expected object 30.

[0031] The method may also include assigning a name 34 to the candidate object 30 by the user. The name 34 and image data of the candidate object 30 may be stored in the memory of the robotic system so that it may be accessed in the future when selecting a similar candidate object 30. Thus, for future candidate objects 30, the memory may be cross-referenced with the stored candidate objects 30 and the image data of the future candidate objects 30. In the event that a corresponding match is found between the image data, the stored name of the object 30 may be displayed on the screen. Similarly, when the user identifies an object 30 with a name stored in the memory, the system may access and display any data previously stored and associated with the name, such as different image views, gripping information, class groups, etc.

[0032] The method may also include displaying the candidate object 30 with a highlight or a boundary line 32 around the candidate object 30 on the display screen 18. This may be done at various points in the training method. For example, the robotic system may pre-select the object 30 by highlighting or drawing a boundary, or may highlight or draw a boundary for the object 30 in response to a selection made by the user through a direct selection by the user on the screen 18, in response to the user identifying the object 30 in the workspace through a gesture, or through speech recognition.

[0033] The method may also include the user identifying an area in the workspace having an expected object 30. The robotic system may then determine one or more candidate objects 30 within the area as objects 30 that may correspond to the user's intended object 30. The candidate objects 30 may then be displayed, and the user may confirm or deny that the system has selected the correct object.

[0034] The method may also include displaying the candidate objects 30 within the image of the workspace on the touch screen 18. The user may then identify which object 30 is desired by touching the touch screen 18 to select the object 30.

[0035] The method may also include identifying the intended object 30 through a user gesture 20 within the workspace. The gesture 20 may include the user positioning a hand or finger 36 within the workspace or positioning a rod 38 within the workspace. The robotic system may then determine the position and / or orientation of the hand or finger 36 or rod 38 to determine which object 30 to select. In the case where the user's hand or finger 36 is used to make a gesture within the workspace, the robotic system preferably visually tracks the position and / or orientation of the hand or finger 36.

[0036] The method may also include physically unattaching the rod 38 to the robotic system so that the rod 38 can be freely moved by the user in and out of the workspace. In one embodiment, the rod 38 may not contain any electronics and may be merely a mechanical representation of an actual robotic gripper. In this case, the robotic system may visually track the position and / or orientation of the rod 38 in the workspace. Alternatively, the rod 38 may include an IMU, and the position and / or orientation of the rod 38 may be wirelessly transmitted from the rod 38 to the robotic system.

[0037] The method may also include the user identifying the intended object 30 using one or more words 40 either verbally or through written text, preferably using natural language.

[0038] The method may also include the user identifying a gripping position 42 on the intended object 30 to the robotic system. As described above, the gripping position 42 may be identified using the touch screen 18, gestures 20, or language 22. As described above, the gripping position 42 may also be stored in memory along with the name and image data of each object 30. The gripping position 42 may also be displayed to the user on the display screen 18 for confirmation or denial of the correct gripping position 42 by the user. The user may also identify a gripping force 42 of the robotic system to be applied to the intended object 30. This is preferably accomplished using a force sensor on the rod 38 that transmits the force applied by the user to the robotic system.

[0039] Although the preferred embodiments of the present invention have been described, it should be understood that the present invention is not so limited and can be modified without departing from the present invention. Although each embodiment described herein may only relate to certain features and may not specifically relate to each feature described relative to other embodiments, it should be recognized that the features described herein are interchangeable, even without mentioning specific features, unless otherwise described. It should also be understood that the above advantages are not necessarily the only advantages of the present invention, and it is not necessarily expected that each embodiment of the present invention will achieve all of the described advantages. The scope of the present invention is defined by the appended claims, and all devices and methods falling within the meaning of the claims, whether literally or equivalent, will be included therein.

Claims

1. A method for training a robot system by a user, comprising: The robot system determines a candidate object (30) within a working area of ​​the robot; Displaying the candidate objects to the user (30); The user identifies to the robotic system whether the candidate object (30) matches the expected object (30); identifying, by the user, to the robotic system an intended gripping location (42) on the object (30); and The clamping position (42) including accuracy information is displayed to the user.

2. The method according to claim 1, further comprising: Assigning a name (34) to the candidate object (30) by the user; storing the image data and the name (34) of the candidate object (30) in a memory of the robotic system; cross-referencing future candidate objects (30) with the image data; and The name (34) and the object (30) corresponding to the future candidate of the image data are displayed.

3. The method according to claim 1, wherein the candidate object (30) is displayed on the display screen (18) with highlighting or a boundary line (32) surrounding the candidate object (30).

4. The method according to claim 1 further includes identifying, by the user, to the robotic system an area within the workspace of the robot that includes the expected object (30), wherein the robotic system determines the candidate object (30) from the area as possibly corresponding to the expected object (30).

5. The method according to claim 1, wherein the candidate objects (30) are displayed within the image of the workspace on a touch screen (18), and the user identifies the intended object (30) by touching the touch screen (18).

6. The method according to claim 1, wherein the user identifies the intended object (30) by making a gesture (20) within the workspace.

7. The method of claim 6, wherein the gesturing (20) comprises the user positioning a hand or finger (36) within the workspace, the robotic system determining a position and / or orientation of the hand or finger (36) to determine a candidate object (30).

8. The method of claim 6, wherein the gesturing (20) includes the user positioning a rod (38) within the workspace, the robotic system determining a position and / or orientation of the rod (38) to determine the candidate object (30).

9. The method of claim 8, wherein the rod (38) is not physically connected to the robotic system.

10. The method of claim 7, wherein the robotic system visually tracks the position and / or orientation of the hand or finger (36) or the rod (38).

11. The method of claim 8, wherein the rod (38) comprises an IMU and the position and / or orientation is wirelessly transmitted to the robotic system.

12. The method of claim 1, wherein the user identifies the intended object (30) using a word (40).

13. The method of claim 1, further comprising identifying, by the user, to the robotic system a gripping force (42) to be applied to the intended object (30).

Images