Robot teaching device and robot system

Abstract

The present invention provides a robot teaching device and a robot system. The robot teaching device comprises: an image acquisition device that acquires a distance image representing distance information of a photographed object or a two-dimensional image of the photographed object as a motion image; a trajectory detection unit that uses the motion image to detect the movement trajectory of the operator's hand photographed as the photographed object in the motion image; a recognition unit that recognizes whether the detected movement trajectory represents linear movement or rotational movement; and an instruction generation unit that outputs an instruction to make a predetermined movable part of the robot translate according to the movement trajectory when it is recognized that the movement trajectory represents linear movement, and outputs an instruction to make a predetermined movable part of the robot rotate according to the movement trajectory when it is recognized that the movement trajectory represents rotational movement.

Description

Technical Field

[0001] The invention relates to a robot teaching device and a robot system. Background Art

[0002] When various tasks are performed using a system using an industrial robot, it is necessary to teach the robot so that the robot performs a desired action. A teaching device for an industrial robot generally has an operation key or an operation screen that can perform a so-called jog operation.

[0003] On the other hand, as another example of a teaching device, International Publication No. 2017 / 130389 records: "A robot teaching device is provided with an image input device for acquiring images of fingers of an operator and a work object, a finger motion detection unit for detecting finger motions of the operator based on the images acquired by the image input device, and a work content estimation unit for estimating the work content of the operator with respect to the work object based on the finger motions detected by the finger motion detection unit, and a control program generation unit for generating a control program of the robot for reproducing the work content estimated by the work content estimation unit" (Paragraph 0006).

[0004] In teaching industrial robots using a teaching device, the teaching method of operating the teaching device's jog keys or touching / sliding the teaching device's screen is mostly used. However, in teaching the operation of the jog keys, the degree of freedom of instruction is limited depending on the number of jog keys, and in teaching the touch / slide operation of the screen, the degree of freedom of instruction is limited to a two-dimensional plane. A robot teaching device and a robot system that have a higher degree of freedom and can teach a robot through intuitive operation are needed. Summary of the invention

[0005] One embodiment of the present disclosure is a robot teaching device for teaching a robot, comprising: an image acquisition device, which acquires a distance image representing distance information of a photographed object or a two-dimensional image of the photographed object as a motion image; a trajectory detection unit, which uses the motion image to detect a movement trajectory of an operator's hand photographed as the photographed object in the motion image; a recognition unit, which recognizes whether the detected movement trajectory represents a linear movement or a rotational movement; and an instruction generation unit, which outputs an instruction for causing a predetermined movable part of the robot to translate according to the movement trajectory when the recognition unit recognizes that the movement trajectory represents a linear movement, and outputs an instruction for causing the predetermined movable part of the robot to rotationally move according to the movement trajectory when the recognition unit recognizes that the movement trajectory represents a rotational movement.

[0006] Another aspect of the present disclosure is a robot system including a robot, a robot control device for controlling the robot, and the robot teaching device, wherein the robot control device controls the robot according to the command output by the command generation unit of the robot teaching device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The objects, features and advantages of the present invention will become clear from the following description of the embodiments in conjunction with the accompanying drawings. In these drawings:

[0008] Figure 1 It is a diagram showing the overall configuration of a robot system including a robot teaching device according to one embodiment.

[0009] Figure 2 This is a functional block diagram showing the functions of the robot teaching device.

[0010] Figure 3 This is a diagram showing a configuration example of the front surface of the robot teaching device when the robot teaching device is configured as a teaching operation panel.

[0011] Figure 4 This is a schematic diagram showing a state where a gesture input representing a linear movement is completed.

[0012] Figure 5 : is a schematic diagram showing a state where the gesture input representing the rotational movement is completed.

[0013] Figure 6 is a flowchart showing the gesture teaching process.

[0014] Figure 7 : is a flowchart showing the palm orientation recognition process.

[0015] Figure 8 This is a schematic diagram showing a state where the positions of fingertips are recognized by the palm direction recognition process.

[0016] Fig. 9 This is a schematic diagram showing the arrangement relationship among the imaging plane, lens, reference plane, and hand when acquiring the three-dimensional position information of the hand using a two-dimensional image. DETAILED DESCRIPTION

[0017] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In all the drawings, corresponding structural elements are marked with common reference marks. For ease of understanding, the scales of these drawings are appropriately changed. In addition, the method shown in the drawings is an example for implementing the present invention, and the present invention is not limited to the method shown in the drawings.

[0018] Figure 1 1 is a diagram showing the overall configuration of a robot system 100 including a robot teaching device 30 according to an embodiment. Figure 2 is a functional block diagram showing the functions of the robot teaching device 30. Figure 1 As shown, the robot system 100 includes a robot 10, a robot control device 20 for controlling the robot 10, and a robot teaching device 30 for teaching the robot 10. The robot 10 is, for example, a vertical multi-joint robot, but other types of robots may also be used. The robot control device 20 controls the movement of the robot 10 by controlling the motors of the axes of the robot 10. The robot control device 20 has a structure as a general computer, which has a CPU, ROM, RAM, a storage device, a display unit, an operation unit, a communication unit, and the like.

[0019] The robot teaching device 30 is, for example, a teaching operating panel or a tablet terminal type teaching device, and has a structure as a general computer, which has a CPU 31 as a hardware component, a memory 32 such as ROM and RAM, and a communication interface 33. The robot teaching device 30 is connected to the robot control device 20 via the communication interface 33. In addition, the robot teaching device 30 includes an operation unit 35 including various operation keys, and a display unit 36 ​​(see Figure 3 ) as a user interface for teaching. The robot teaching device 30 of this embodiment further includes an image acquisition device 34 for photographing the hand of the operator performing gesture input.

[0020] The image acquisition device 34 is a distance image camera that acquires a distance image representing distance information depending on the spatial position of the photographed object as a moving image, or a camera that acquires a two-dimensional image of the photographed object as a moving image. As an example, the structure of the image acquisition device 34 as a distance image camera is that it has a light source, a projector that projects pattern light, and two cameras arranged on both sides of the projector with the projector sandwiched therebetween. The object projecting the pattern light is photographed from the two cameras arranged at different positions, and the three-dimensional position information of the object is acquired by a stereo method. Alternatively, the image acquisition device 34 as a distance image camera may also be a TOF (Time of Flight) type camera that has a light source and a photographing element and measures the time it takes for light to irradiate the object and return. Alternatively, the image acquisition device 34 as a distance image camera may be a stereo camera that acquires the three-dimensional position information of the object by a stereo method based on the position of the object being identified in each of the images taken by the two cameras by image recognition. In addition, other types of devices may also be used as distance image cameras. Figure 1-3 In the illustrated configuration example, the image acquisition device 34 is configured as a device of a type built into the robot teaching device 30 , but the image acquisition device 34 may be configured as an external device connected to the robot teaching device 30 by wire or wirelessly.

[0021] Figure 3 FIG. 2 shows a configuration example of the front surface of the robot teaching device 30 when the robot teaching device 30 is configured as a teaching operation panel. Figure 3 As shown in FIG. 1 , the image acquisition device 34 is arranged on the front surface of the robot teaching device 30 where the operation unit 35 and the display unit 36 ​​are arranged, and the object to be photographed is photographed from the front side. In this structure, the operator can hold the robot teaching device 30 with one hand and move the other hand to input gestures. As described in detail below, the robot teaching device 30 causes the predetermined movable part of the robot 10 to move horizontally (see FIG. 1 ) when the operator's hand moves in a straight line. Figure 4 ), when the operator's hand movement is rotational movement, a predetermined movable part of the robot 10 is caused to rotate and move ( Figure 5 ). Hereinafter, as an example, the predetermined movable portion is the arm tip 11 of the robot 10.

[0022] like Figure 2 As shown, the robot teaching device 30 includes, in addition to the image acquisition device 34, a trajectory detection unit 311, a recognition unit 312, and an instruction generation unit 313. These functions are implemented, for example, by the CPU 31 executing software stored in the memory 32. The trajectory detection unit 311 uses the motion image acquired by the image acquisition device 34 to detect the movement trajectory of the operator's hand that is photographed as a subject in the motion image. The recognition unit 312 recognizes whether the detected movement trajectory of the hand represents linear movement or rotational movement. When the instruction generation unit 313 recognizes that the movement trajectory represents linear movement, it generates an instruction to cause the front end 11 of the arm of the robot 10 to translate according to the movement trajectory (see Figure 4 ), when the moving trajectory is identified as indicating rotational movement, an instruction is output to cause the front end 11 of the arm of the robot 10 to rotate and move according to the moving trajectory (refer to Figure 5 ). The robot control device 20 controls the robot 10 according to the instruction from the robot teaching device 30.

[0023] Figure 6 This is a flowchart showing a teaching process of a gesture (hereinafter referred to as a gesture teaching process). Figure 6 The gesture teaching process is started, for example, by completing a predetermined operation on the robot teaching device 30 via the operation unit 35. The gesture teaching process is executed under the control of the CPU 31 of the robot teaching device 30. When the gesture teaching process is started, the image acquisition device 34 takes pictures. When inputting gestures, the operator projects only the operator's hand in the image of the image acquisition device 34, and performs the operation in order not to project other parts such as the face.

[0024] First, the robot teaching device 30 (trajectory detection unit 311) recognizes the hand in each frame of the motion image acquired by the image acquisition device 34 (step S11). When the image acquisition device 34 is a range image camera, the recognition of the hand in frames is described separately for the case where it is a two-dimensional camera.

[0025] (Step S11: In the case of a distance image camera)

[0026] In each frame of the image captured by the range image camera, the distance to the captured object is expressed as, for example, shading. The operator's hand can be considered as pixels that are gathered as a block within a certain distance range in the range image (i.e., a collection of pixels where the distance between adjacent pixels does not vary much). Therefore, by marking pixels within a certain distance range in the range image, the collection of marked pixels can be identified as a hand. In addition, under the condition that the operator moves his hand at a certain distance (e.g., approximately 15 cm) from the image acquisition device 34, the recognition accuracy can be improved by performing image recognition using this method.

[0027] (Step S11: Case of a two-dimensional camera)

[0028] When the image acquisition device 34 is a camera that acquires a moving image of a two-dimensional image, the trajectory detection unit 311 may perform hand recognition by any of the following methods.

[0029] (A1) Skin color pixels are grouped in the image and recognized as hands. At this time, the skin color area is extracted by converting the color information of the image from RGB to HSV (hue, color, brightness), for example, so that the skin color area can be extracted with high accuracy.

[0030] (A2) Image recognition is performed by a deep training method. As an example, multiple hand images are pre-learned in a convolutional neural network, and the hand recognition in each frame of the image is performed. In this case, for example, under the condition that the operator inputs a hand gesture in a standard posture, images of the hands of multiple people in standard postures can be prepared and learned as training data.

[0031] (A3) Hand recognition by template matching At this time, the operator may perform template matching under the condition that the operator performs gesture input with a predetermined gesture (for example, a standard posture).

[0032] Note that the palm orientation recognition in step S11 will be described later.

[0033] When the hand is recognized for each frame in this way, the trajectory detection unit 311 then tracks the movement of the hand between frames and obtains the trajectory of the hand (step S12). In step S12, the trajectory detection unit 311 obtains the center of gravity of the hand recognized for each frame. Here, various methods for obtaining the geometric center of gravity of the hand based on the two-dimensional position information of the hand recognized in step S11 can be used. The trajectory detection unit 311 obtains the movement trajectory of the hand by connecting the center of gravity positions of the hand in each frame obtained. The acquisition of the movement trajectory of each hand when the image acquisition device 34 is a distance image camera and when it is a two-dimensional camera is described.

[0034] (S12: When it is a distance image camera)

[0035] When the distance image is used, the pixels of the hand area identified in step S11 have distance information, so the three-dimensional position of the center of gravity of the hand can be obtained. Therefore, at this time, the trajectory detection unit 311 connects the center of gravity of the hand identified in each frame to obtain the three-dimensional position information of the movement trajectory of the hand.

[0036] (S12: When it is a two-dimensional camera)

[0037] By connecting the positions of the center of gravity of the hand recognized in the two-dimensional images of each frame, two-dimensional information of the movement trajectory of the hand can be obtained. In order to obtain three-dimensional information of the movement trajectory of the hand using a two-dimensional camera, as an example, gesture input is performed under the following conditions.

[0038] (r1) The operator holds the robot teaching device 30 at a known position in the robot coordinate system, and holds it in a posture such that the optical axis of the lens 34b of the image acquisition device 34 substantially coincides with the X-axis of the robot coordinate system.

[0039] (r2) The operator will recognize that the reference plane S0 (i.e., the plane that is the same position in the X-axis direction relative to the image acquisition device 34) that is perpendicular to the optical axis at a predetermined distance from the robot teaching device 30 (e.g., about 15 cm) is the plane that serves as the reference for configuring the hand when gesture input begins.

[0040] Fig. 9The positional relationship between the imaging surface 34a, lens 34b and reference plane S0 of the imaging element of the image acquisition device 34 under the above conditions (r1) and (r2) is shown. According to the above conditions (r1) and (r2), the respective positions P0 and P1 of the imaging surface 34a and the reference plane S0 in the X-axis direction of the robot coordinate system are known. The size (image magnification) of the image of the hand in the imaging surface 34a when the hand is placed at the position of the reference plane S0 in the configuration relationship is pre-stored. When the operator's hand moves in the X-axis direction relative to the reference plane S0 with the gesture action, and the size of the image of the hand on the imaging surface 34a changes, it is compared with the size of the image of the hand when the hand is on the reference plane S0, thereby the position (P2) of the hand in the X-axis direction can be calculated based on the focal length of the lens 34b and the imaging formula of the lens. As described above, in the case of a two-dimensional camera, three-dimensional position information of the movement trajectory of the hand can be obtained.

[0041] The three-dimensional position of the hand movement trajectory obtained above is the position information in the coordinate system (hereinafter referred to as the camera coordinate system) fixed on the image acquisition device 34, so the trajectory detection unit 311 converts the coordinate value of the movement trajectory into the coordinate value in the robot coordinate system based on the relative position relationship between the robot 10 and the image acquisition device 34 (the robot coordinate system and the camera coordinate system). The information representing the relative position relationship between the robot 10 and the image acquisition device 34 can also be obtained by the following method. For example, when the operator holds the robot teaching device 30 at a known position / gesture in the robot coordinate system (for example, a posture in which the optical axis of the lens of the image acquisition device 34 is roughly consistent with the X-axis of the robot coordinate system at a known position in the robot coordinate system) and inputs a gesture, the coordinate value of the movement trajectory can be converted from the camera coordinate system to the robot coordinate system under such conditions. Alternatively, a sensor (gyro sensor, acceleration sensor, etc.) for detecting the position / posture can also be installed on the robot teaching device 30. At this time, for example, the relative positional relationship between the robot coordinate system and the camera coordinate system is calibrated in advance when the robot teaching device 30 is started, and the position / posture of the robot teaching device 30 in the robot coordinate system is always calculated by the above-mentioned sensor. Other methods can also be used to obtain the relative positional relationship between the robot coordinate system and the camera coordinate system.

[0042] Next, the robot teaching device 30 (recognition unit 312) uses the hand movement trajectory obtained in step S12 to identify whether the hand movement trajectory represents linear movement or rotational movement (step S13). The recognition here is performed as an example using two-dimensional position information of the hand movement trajectory (for example, position information when the movement trajectory is projected onto a certain plane). Hough transformation is used to identify whether the hand movement trajectory is a linear movement. Specifically, the following steps are used to determine whether the movement trajectory obtained in step S12 is a straight line.

[0043] (E1) For each point (x0, y0) on the plane representing the hand's movement trajectory (x, y), the relationship r=x0cosθ+y0sinθ is used to draw the entire set of straight lines passing through the point on the (r, θ) plane.

[0044] (E2) The point where multiple curves intersect in the (r, θ) plane represents a straight line in the (x, y) plane. Here, for example, when it is determined that a predetermined number or more of pixels on the trajectory of the (x, y) plane are on the same straight line, the movement trajectory of the hand can also be determined to be a straight line.

[0045] Regarding the identification of whether the hand movement trajectory is a rotational movement, the generalized Hough transform is used. Specifically, the generalized Hough transform can be used to determine whether the hand movement trajectory is a circle, an ellipse, or a quadratic curve. As an example, whether the movement trajectory is a circle can be identified by the following steps.

[0046] (F1) For each point (x0, y0) on the plane representing the hand's movement trajectory (x, y), use (x0-A) 2 +(y0-B) 2 =R 2 The relationship describes the set of entire circles passing through the point in (A, B, R) space.

[0047] (F2) A point shared by multiple surfaces in the (A, B, R) space represents a circle in the (x, y) plane. Here, for example, when it is determined that a predetermined number or more of pixels on the movement trajectory in the (x, y) plane are located on the same circle, it can also be determined that the movement trajectory of the hand is a rotation. By a method similar to the case of a circle, it is also possible to determine whether the movement trajectory is a rotational movement by determining whether the pixels on the trajectory in the (x, y) plane are located on the same ellipse or on a quadratic curve.

[0048] When it is determined that linear movement is recognized through the above steps (E1) and (E2) (S13: linear movement), the robot teaching device 30 (command generation unit 313) generates a command to make the arm front end 11 move horizontally, and sends it to the robot control device 20 (step S14). The robot control device 20 that receives the linear movement command makes the arm front end 11 of the robot 10 move horizontally. Figure 4 As shown, the translation path A2 of the front end 11 of the arm is parallel to the movement path A1 of the hand movement trajectory detected in step S12 in the robot coordinate system. The movement direction is the direction corresponding to the direction of the palm identified by the method described later. Figure 4 As shown, when the palm is facing upward, the front end 11 of the arm is moved upward. The amount of movement of the front end 11 of the arm can be, for example, (1) by moving a preset distance, or (2) by moving a distance corresponding to the length of the movement trajectory of the hand. The moving speed can also be a value pre-set for the robot 10.

[0049] When it is determined that rotational movement is recognized through the above steps (F1) and (F2) (S13: rotational movement), the robot teaching device 30 (command generation unit 313) generates a command for rotational movement of the arm tip 11 and sends it to the robot control device 20 (step S14). The robot controller 20 that receives the rotational movement command causes the arm tip 11 of the robot 10 to rotate. Figure 5 As shown, the rotation axis R1 when the arm tip 11 is rotated is parallel to the direction of the rotation axis R0 of the hand in the robot coordinate system. The rotation direction C2 of the arm tip 11 can be consistent with the rotation direction C1 in the hand movement trajectory. As an example, the recognition unit 312 can obtain the rotation axis R0 of the hand movement trajectory in the robot coordinate system based on the three-dimensional movement trajectory of the hand obtained in step S12.

[0050] When neither the linear movement nor the rotational movement is detected ( S13 : unrecognizable), the present process ends without causing the robot 10 to operate.

[0051] Next, the recognition of the palm orientation performed in step S11 will be described. Figure 7It is a flowchart representing the process of identifying the direction of the palm (hereinafter referred to as the palm direction identification process). When performing the palm direction identification process, whether the hand used in the gesture is the right hand or the left hand and the hand posture during the gesture are pre-registered in the robot teaching device 30. The hand posture used in the gesture is a standard gesture as an example. First, the trajectory detection unit 311 uses the pixel group of the part of the hand detected in step S11 to extract the contour of the hand (step S21). Next, the trajectory detection unit 311 puts dots on the contour line of the hand at equal intervals (step S22). Next, the trajectory detection unit 311 sets a line segment connecting adjacent points among the points set in step S22, and extracts a point where the angle formed by two straight lines connecting a certain point and the adjacent points on both sides becomes an acute angle less than a fixed value (step S23). In this way, a sharp part like a fingertip is extracted in the contour line.

[0052] Next, the trajectory detection unit 311 selects five points at a certain distance from the center of gravity of the hand from the points extracted in step S23 and regards them as fingertip positions (step S24). Figure 8 The left side of the right hand indicates that when the palm side of the right hand faces the image acquisition device 34, Figure 7 The positions of the fingertip positions P11-P15 are detected by the processing of steps S21-S24. Figure 8 On the right side, when the back of the right hand faces the image acquisition device 34, it means that Figure 7 The positions of the fingertip positions P21-P25 detected by the processing of steps S21-S24. By predetermining the left and right hands used for gesture input, according to the positional relationship of the fingertips on the palm side and the back side, as shown in FIG. Figure 8 The difference shown in the figure can determine the direction of the palm displayed in the image (step S25). For example, the palm or back of the hand can be determined based on the position of the fingertips of the index finger, middle finger, ring finger, and little finger located at a relatively close position and which side the thumb is located on.

[0053] Next, the timing of starting and ending the gesture recognition will be described. In order to avoid recognizing the hand movement that the operator does not intend to do as a gesture instruction, for example, the following method can be adopted: the hand movement trajectory is detected at a certain time interval, and when the movement amount is below a certain value, the recognition of whether it is a linear movement or a rotational movement is not performed. For example, the hand movement can be repeatedly started at a certain time interval. Figure 6 In the gesture teaching process, if the movement amount of the hand movement trajectory identified as linear movement or rotational movement by Hough transformation in step S13 is less than a certain amount, it is determined that it is unrecognizable. In this way, it is possible to avoid recognizing the hand movement unintentionally performed by the operator as a gesture instruction. In addition, in the palm direction recognition in step S11, if, for example, the fingertips cannot be recognized, it can also be determined as unrecognizable in step S13.

[0054] According to the present embodiment described above, it is possible to teach the robot with a higher degree of freedom and by intuitive operation.

[0055] The embodiments of the present disclosure are described above, but those skilled in the art should understand that various modifications and changes can be made without departing from the disclosed scope of the technical solution.

[0056] In the above embodiment, the functional blocks of the trajectory detection unit 311 , the recognition unit 312 , and the command generation unit 313 are arranged in the robot teaching device 30 , but at least part of these functions may be arranged on the robot control device 20 side.

[0057] The robot teaching device 30 may include both a distance image camera and a two-dimensional camera as the image acquisition device 34. In this case, the distance image camera and the two-dimensional camera are configured so that the positional relationship between them is known (i.e., the relationship between the pixel configurations of their images). In such a configuration, the various recognition methods when the distance image camera described in the above embodiment and the various recognition methods when the two-dimensional camera is used can be used simultaneously.

[0058] The program for executing the gesture teaching processing or palm orientation recognition processing shown in the above embodiment can be recorded in various computer-readable recording media (such as semiconductor memories such as ROM, EEPROM, flash memory, magnetic recording media, CD-ROM, DVD-ROM, etc.).

Claims

1. A robot teaching device, the robot teaching device teaching a robot, It is characterized in that The robot teaching device has: An image acquisition device that acquires a distance image representing distance information of a photographed object or a two-dimensional image of the photographed object as a moving image; a trajectory detection unit that detects a movement trajectory of a hand of the operator captured as the captured object in the motion image using the motion image; a recognition unit for recognizing whether the detected movement trajectory represents linear movement or rotational movement; as well as a command generating unit that outputs a command for causing a predetermined movable part of the robot to translate according to the movement trajectory when the recognition unit recognizes that the movement trajectory indicates linear movement, and outputs a command for causing the predetermined movable part of the robot to rotate according to the movement trajectory when the recognition unit recognizes that the movement trajectory indicates rotational movement, The trajectory detection unit detects the movement trajectory based on the operator's hand being in a predetermined posture in the motion image. When the recognition unit recognizes that the movement trajectory indicates rotational movement, the recognition unit further detects a rotation axis of the rotational movement of the hand as a coordinate value of the robot coordinate system. The command generation unit generates a command for rotationally moving the predetermined movable part based on the movement trajectory and the detected rotation axis of the rotational movement of the hand.

2. The robot teaching device according to claim 1, characterized in that: The trajectory detection unit detects the movement trajectory as a coordinate value of the robot coordinate system based on information indicating a relative positional relationship between the robot and the image acquisition device. The command generation unit generates the command to move the predetermined movable portion parallel to the movement trajectory when the recognition unit recognizes that the movement trajectory indicates linear movement.

3. The robot teaching device according to claim 1 or 2, characterized in that: The trajectory detection unit further detects the direction of the palm based on the motion image. The command generation unit sets a moving direction when the predetermined movable part is translated, according to the detected orientation of the palm.

4. The robot teaching device according to claim 1 or 2, characterized in that: The command generation unit sets a rotation axis when the predetermined movable part is rotationally moved so that the rotation axis when the predetermined movable part is rotationally moved is parallel to the rotation axis of the detected rotational movement of the hand.

5. A robot system, characterized in that: The robot system comprises a robot, a robot control device for controlling the robot, and the robot teaching device according to any one of claims 1 to 4. The robot control device controls the robot according to the command output by the command generation unit of the robot teaching device.

Images