Teaching of a Robot System Using Hand Gesture Control and Visual Inertial Odometry

Abstract

A computer-implemented method for teaching a robotic system is provided. The method may include a first sensor system (102) of the robotic system detecting a hand gesture (103) of an operator and moving (S3) a robot arm (101) of the robotic system according to the hand gesture (103) of the operator. The method may include a second sensor system (104) of the robotic system detecting (S31) an imminent collision of the robot arm (101) with an obstacle (105) and moving the robot arm (101) along at least a portion of a collision-free path (106) around the obstacle (105). By advantageously combining hand gesture control and visual inertial odometry, the present invention provides improved teaching of robotic arm trajectories.

Description

Technical Field

[0001] This application generally relates to the field of robot control, and more specifically, to improving techniques for teaching robot systems that use hand gesture control and visual inertial odometry.

Background Art

[0002] Robots have become an essential tool for modern industrial automation. A robot system can comprise one or more robotic arms in the form of mechanical arms that can perform physical tasks, generally using human supervision and control. However, teaching a robotic arm to perform a task is often difficult, and robot experts typically need to explicitly program the robotic arm for each task. To make teaching robotic arms easier, approaches such as hand gesture control have been developed, which enable an operator to teach a robotic arm in a more intuitive and user-friendly manner.

[0003] In the field of computer vision, techniques for interpreting computer-based hand gestures have been developed as a basis for various aspects of human-machine interaction. One purpose of hand gesture recognition is to identify specific hand gestures and convey information to the robotic arm involved in the individual gesture. Based on this, dedicated commands for performing operations can be given to the robotic arm, constructing an effective communication channel between the operator and the robotic arm, and the robotic arm does not rely on conventional input devices such as keyboards and mice.

[0004] For example, Patent Document 1 titled "Remote control robot system" discloses a system that enables an operator to control a robot arm of a robot system using body gestures, hand gestures, or vocalizations. In addition to the gesture control system, the robot system may be equipped with a camera attached to the end effector of the robot arm. The intention of this additional camera is to be able to output the captured image to a monitor, enabling the operator to check the working status on the monitor while remotely operating the robot arm.

[0005] A similar technique has been proposed in Patent Document 2 titled "Tele-operation system and control method thereof", which discloses a remote operation system that enables a robot arm to move following the movement of a user's hand. Also in this system, the robot may include additional sensors for detecting the working environment and displaying information related to the surroundings of the robot on a display.

[0006] As described above, hand gesture control can be used not only to directly control the robot arm but also to teach the robot. That is, the operator can perform a desired movement or work task, and the robot arm can naturally and intuitively learn the movement trajectory and task execution from the operator. Then, the robot can autonomously repeat the learned task.

[0007] For example, Patent Document 3 and Patent Document 4 titled "Teaching device and method for robotic arm" of its family members disclose a teaching device and a method for a robotic arm. The teaching device includes a robotic arm, a control device, and a gesture recognition module. The gesture recognition module detects a control gesture signal and transmits the detected control gesture signal to the control device. Then, the control device teaches and moves the robotic arm.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0009] However, the inventors of the present application have found that there are limitations in using hand gesture control to teach a robot. One of the reasons is that there is a possibility that the operator may not be able to teach the robotic arm the most reliable and effective trajectory because the operator's perception may be limited during the teaching process. The above-described system that displays information about the surroundings of the robot on a monitor for the operator to observe can improve this situation to some extent. However, in such a system, the operator still has to constantly alternate between concentrating on the robotic arm and its control by hand gestures and observing the monitor to check the surroundings of the robot.

[0010] Furthermore, although an operator generally has sufficient knowledge of the space for task execution, there is still a risk that the robot arm trajectory cannot be properly taught. For example, due to perception limitations or other reasons, the trajectory taught by the operator may not be the most efficient movement path for the current task.

[0011] Therefore, one of the problems of the present invention is to at least partially overcome the drawbacks of the prior art by providing a method and a system for teaching a robot system that uses hand gestures that result in a particularly safe and efficient operation of the robot system.

Means for Solving the Problem

[0012] This problem is solved, in one aspect, by a computer-implemented method for teaching a robot system. The method may include a first sensor system of the robot system detecting a hand gesture of an operator and moving a robot arm of the robot system according to the hand gesture of the operator.

[0013] Therefore, this aspect of the present invention provides a remote control teaching technique for a robot arm by hand gestures, which is particularly useful, for example, when there are safety concerns. This enables the operator to teach the robot arm in a contactless and very natural manner. For example, by using different hand gestures, the operator may define various types of executions for the robot arm start position, movement trajectory, end position, and / or a specific task cycle.

[0014] However, due to the above-mentioned limitations of perception generally involved when teaching a robot from a distance using hand gestures, the operator may not be able to fully pay attention to certain obstacles, and the robot arm may hit these obstacles when following the movement of the hand gesture. In prior art techniques, the burden of defining a collision-free path falls entirely on the operator, and generally, the approach according to the prior art attempts to assist the operator in achieving the definition of a collision-free path by displaying information about the surroundings of the robot on a monitor for the operator to review during the control of the robot. This is in line with the current trend in the industrial robot industry to improve the capabilities of human operators using more information, such as augmented reality. The logical basis for this approach is the widely accepted principle in the prior art that the human operator should always be the ultimate control instance.

[0015] What the present invention deviates from this known thought pattern is that, preferably during the teaching process, the method may include that a second sensor system of the robot system detects an imminent collision of the robot arm with an obstacle, and the method may include moving the robot arm along at least a part of a collision-free path around the obstacle.

[0016] Different from the prior art, the above-mentioned aspect of the method of the present invention not only passively displays information about the surroundings of the robot on a monitor for the operator to double-check, but also advantageously combines hand gesture control and visual-inertial-odometry (VIO) to teach an improved robot arm trajectory. VIO generally refers to the localization of an object in an unknown environment by using an inertial measurement unit (IMU) and / or a monocular camera. By using VIO, information about the surrounding environment of the robot arm can be collected in real time, so that the trajectory can be autonomously optimized by the robot system.

[0017] In particular, using VIO, the robotic arm can automatically and / or autonomously deviate from the path defined by the operator and instead follow a collision-free path despite the operator's control. Such "non-compliance" with the operator's control is quite rare and may seem counterintuitive in prior art hand gesture control techniques based on the fundamental principle that the human operator always has full control. However, the inventors have found that by doing so, advantageously, the above problems are solved, it becomes possible to teach the robot system an optimal and safe path, and that it is still possible in a form widely accepted by the operator as long as they understand the advantage, namely that even if the operator's teaching is not optimal, a collision-free path will ultimately be obtained.

[0018] In one aspect of the invention, moving the robotic arm along at least a portion of a collision-free path around an obstacle may include stopping the robotic arm at a predetermined safe distance before colliding with the obstacle. When the robotic arm moves essentially vertically according to the operator's hand gesture, the method may include moving the robotic arm essentially horizontally until the path according to the operator's hand gesture no longer includes an imminent collision. When the robotic arm moves essentially horizontally according to the operator's hand gesture, the method may include moving the robotic arm essentially vertically until the path according to the operator's hand gesture no longer includes an imminent collision. In either case, the method may further include continuing to move the robotic arm according to the operator's hand gesture. Thus, obstacles within the path intended by the operator can be effectively avoided relatively easily.

[0019] In another aspect of the present invention, the method may include determining an optimal path for the robotic arm based on information regarding the environment of the robotic system detected by the collision-free path and / or the second sensor system. The optimal path may be optimized to minimize the cycle time while being collision-free. The step of determining the optimal path may be performed after the training of the robotic arm is completed. This aspect is particularly advantageous because even if the teaching trajectory is collision-free due to the above self-correction of the robotic system, the trajectory may not still be the shortest movement path for the current operation. Therefore, the optimal path can lead to a more resource-efficient operation of the robotic system.

[0020] The first sensor system may include a first camera. The first sensor system may be disposed at a fixed position of the robotic system. Thus, the first sensor system may provide means for capturing hand gestures performed by an operator in the vicinity of the robotic system.

[0021] The second sensor system may include a second camera. For example, the camera may be a tracking camera such as the Intel® RealSense™ Tracking Camera T265. The second camera may include two lens sensors. The second sensor system may include an inertial measurement device. The second sensor system may be disposed on the robotic arm. Thus, the second sensor system may provide VIO functionality configured to position an object in the vicinity of the robotic arm.

[0022] In yet another aspect of the present invention, the method may include storing one or more control instructions that, when executed, move a robotic arm along a collision-free path or an optimal path. Further, the method may include sending the one or more control instructions to a robotic system and / or causing the one or more control instructions to be executed by the robotic system. Accordingly, the operating robotic system may be directly controlled using a collision-free and optionally further optimized trajectory.

[0023] The present invention also provides an apparatus including means for performing any of the methods disclosed herein.

[0024] Further, an apparatus is provided, the apparatus comprising a processor and a memory coupled to the processor, the memory storing executable instructions that, when executed by the processor, cause the apparatus to detect an operator's hand gesture by a first sensor system of a robotic system, move a robotic arm of the robotic system according to the operator's hand gesture, detect an impending collision of the robotic arm with an obstacle by a second sensor system of the robotic system, and move the robotic arm along at least a portion of a collision-free path around the obstacle.

[0025] The memory may further store executable instructions that, when executed by the processor, cause the apparatus to perform any of the methods disclosed herein.

[0026] In any of the apparatuses disclosed herein, the apparatus may be a robotic system or the apparatus may be a separate apparatus from the robotic system.

[0027] A computer program is also provided, the computer program including instructions that, when the program is executed by a computer, cause the computer to perform any of the methods disclosed herein.

[0028] Finally, a computer-readable medium is also provided, the computer-readable medium including instructions that, when executed by a computer, cause the computer to perform any of the methods disclosed herein.

[0029] The present disclosure may be more fully understood by reference to the following drawings.

Brief Description of the Drawings

[0030]

Figure 1

Figure 2

Embodiments for Carrying Out the Invention

[0031] Embodiments of the present invention provide an improved teaching process for a robot system. In one embodiment, a first camera fixed to a station is used to detect an operator's hand gestures. A hand gesture recognition model may be used to teach a robot arm to move and perform a desired operation by translating (multiple) detected hand gesture signals into (multiple) control commands for the robot arm. A second camera, particularly a tracking camera, includes a two-lens sensor and an inertial measurement unit (IMU) and is mounted on the robot arm to provide visual-inertial odometry (VIO) functionality in one embodiment. When an operator teaches the movement of the robot arm using one or more hand gestures, the surrounding environment of the robot arm over such movement can be detected by the tracking camera, so that the teaching trajectory can be automatically corrected if a collision risk is detected. After the teaching process, the robot arm trajectory that will be used for the actual operation of the robot system can be further optimized to reduce the cycle time using the recorded initial trajectory and / or surrounding environment information.

[0032] Figure 1 shows an exemplary robot system according to an embodiment of the present invention. As can be seen from the figure, the robot system includes a robot arm 101 which is an object to be taught.

[0033] The detection camera 102 is fixed, for example, to a station above the robot arm and faces forward in order to detect the hand gesture 103 of the operator. Thereby, the operator can move the robot arm 101 and execute (a plurality of) desired operations.

[0034] Another tracking camera 104 equipped with a dual lens and an IMU is attached on the robot arm 101, preferably near the end effector, and faces downward in the example of FIG. 1. Therefore, when the operator moves the robot arm trajectory, the tracking camera 104 can inspect the surrounding environment.

[0035] When an obstacle 105 is detected on the movement path and a risk of collision is detected, the robot arm 101 will actively self - correct the movement path even before the operator's instruction, which leads to a collision - free path 106 as shown in FIG. 1.

[0036] After the teaching process is completed, the initial trajectory 106 and obstacle information learned from the operator can be sent to an optional optimization engine so that an optimal trajectory (e.g., having a reduced cycle time) can be calculated to form the final robot arm trajectory 107 for the task. Those skilled in the art will understand that there are various trajectory planning techniques that can be used in the embodiments of the present invention. For example, the optimization engine may be implemented based on the Open Motion Planning Library (OMPL), an open-source library provided by the Kavraki Lab (Department of Computer Science at Rice University). Other suitable techniques include, but are not limited to, Pilz Industrial Motion Planner, Stochastic Trajectory Optimization for Motion Planning (STOMP), Search-Based Planning Library (SBPL), and Covariant Hamiltonian Optimization for Motion Planning (CHOMP).

[0037] Figure 2 shows a teaching process according to an exemplary embodiment of the present invention. It should be understood that the order of the steps is exemplary. Other embodiments may include only a subset of the illustrated steps.

[0038] In step S1, the operator may trigger the teaching mode of the robot arm 101 using a hand gesture. In response, the robot arm 101 may move to the standby position. Alternatively, the standby mode may be triggered by an input command different from a hand gesture such as a voice command or an input via a user interface element on a graphical user interface.

[0039] In step S2, the operator may move the robotic arm 101 using hand gestures until the robotic arm 101 reaches the intended starting position for the task to be taught. The operator may indicate that the starting position has been reached by hand gestures or other suitable commands.

[0040] After these preparatory steps, step S3 includes the actual teaching process, in which the operator may define the path to be taken by the robotic arm 101 by continuously moving the robotic arm 101 using one or more hand gestures until the intended end position for the task to be taught is reached.

[0041] If the tracking camera 104 senses an obstacle 105 in the movement path during movement, the robotic arm 101 will self-correct the movement path to prevent an imminent collision.

[0042] In one embodiment, the self-correction method may be implemented to obtain a collision-free shortest movement path. For example, the self-correction method may include the following. When the robotic arm 101 moves vertically following the operator's hand gesture, if the rear-view camera 104 senses an obstacle 105 (see S31), the robotic arm 101 stops at a safe distance before hitting the obstacle 105 (see S32) and moves horizontally along the shortest path so as to have sufficient space to avoid the obstacle 105 (see S33). Thereafter, the robotic arm 101 continues to move vertically following the hand gesture (see S34), which results in the self-correction trajectory 106 shown in FIG. 2. Similarly, when the robotic arm 101 moves horizontally following the operator's hand gesture, if the rear-view camera 104 senses an obstacle 105 (see S31), the robotic arm 101 stops at a safe distance before hitting the obstacle 105 (see S32) and moves vertically along the shortest path so as to have sufficient space to avoid the obstacle 105 (see S33). Thereafter, the robotic arm 101 continues to move horizontally following the hand gesture (see S34).

[0043] Optionally, the self-correction method may first perform a preprocessing step, which may include obtaining a discretized representation of the obstacle 105, for example, in the form of a set of voxels in a three-dimensional space. By using the discretized representation of the obstacle 105, the actual shape of the obstacle 105 does not affect the above self-correction method.

[0044] In both cases, the required safe distance may be a predetermined distance, which may be set as needed. The safe distance generally depends on factors such as the dimensions / size of the robotic arm 101, the size of the obstacle, the dimensions of the components, and / or the usage scenario. Generally, the safe distance is set by the operator after evaluating all factors or at least the factors relevant to the current usage case.

[0045] After the teaching process shown in step S3, the method may proceed to step S4 and / or step S5, which may be performed in parallel or sequentially. The movement path 106 of the robot arm 101 may be stored in step S4 so that it can be used as a reference trajectory. In step S5, the tracking camera 104 attached to the robot arm 101 may also record the environmental information around the robot arm captured during the teaching process.

[0046] Using the initial reference trajectory 106 and / or the environmental information around the robot arm as input, the optimization engine may be used in step S6 to further optimize the trajectory 106 for the actual movement of the robot arm. Different from the self-correction in step S3 that focuses only on collision prevention, the optimization in step S6 may help to identify the optimal robot arm movement trajectory 107 with the shortest cycle time, and the movement trajectory 107 is still collision-free based on the reference trajectory 106 and the environmental information around the robot arm. A number of suitable algorithms for path optimization have already been described above. A particular embodiment of the present invention uses CHOMP. CHOMP is a gradient-based trajectory optimization procedure that makes the motion planning aspect of the embodiment simple and trainable.

[0047] Finally, that is, after recording the self-corrected path 106 or after optimizing the self-corrected path 106 to obtain the optimal path 107, the robot arm 101 may resume normal operation using the (optimal) trajectory for the current task. For this purpose, the self-corrected path 106 or the optimal path 107 may serve as an input for generating suitable control signals for controlling the robot arm 101 during operation (see S7).

Claims

1. A computer-implemented method for teaching a robot system, the method comprising: detecting, by a first sensor system (102) of the robot system, a hand gesture (103) of an operator; moving a robot arm (101) of the robot system according to the hand gesture (103) of the operator (S3); detecting, by a second sensor system (104) of the robot system, an impending collision of the robot arm (101) with an obstacle (105) (S31); moving the robot arm (101) along at least a part of a collision-free path (106) around the obstacle (105); determining, based on the collision-free path (106) and / or information about the environment of the robot system detected by the second sensor system (104), an optimal path (107) for the robot arm (101) (S6); wherein the optimal path (107) is optimized to minimize cycle time while being collision-free.

2. Moving the robot arm (101) along at least a part of the collision-free path (106) around the obstacle (105) comprises: stopping the robot arm (101) at a predetermined safe distance before colliding with the obstacle (105) (S32); when the robot arm (101) moves essentially vertically according to the hand gesture (103) of the operator, moving the robot arm (101) essentially horizontally until the path according to the hand gesture (103) of the operator no longer includes an impending collision (S33); when the robot arm (101) moves essentially horizontally according to the hand gesture (103) of the operator, moving the robot arm (101) essentially vertically until the path according to the hand gesture (103) of the operator no longer includes an impending collision (S33); continuing to move the robot arm (101) according to the hand gesture (103) of the operator (S34). The method according to claim 1.

3. The first sensor system (102) includes a first camera and / or The first sensor system (102) is arranged at a fixed position of the robot system, the method according to claim 1 or 2.

4. The second sensor system (104) includes a second camera, the second camera preferably includes a two-lens sensor, and / or The second sensor system (104) includes an inertial measurement device, and / or The second sensor system (104) is arranged on the robot arm (101), the method according to any one of claims 1 to 3.

5. Further including storing one or more control commands that, when executed, move the robot arm (101) along the collision-free path (106) or the optimal path (107), the method according to any one of claims 1 to 4.

6. The method according to claim 5 further includes transmitting the one or more control commands to the robot system and / or causing the robot system to execute the one or more control commands.

7. An apparatus including means for implementing the method according to any one of claims 1 to 6.

8. A computer program including instructions that, when the program is executed by a computer, cause the computer to implement the method according to any one of claims 1 to 6.

Images