Robot image display method, computer program, and robot image display system

Abstract

To provide a technique for teaching a robot without actually installing and operating a robot.SOLUTION: A method for displaying a robot image includes: (a) a step of recognizing a position and an attitude of a base of a robot from a base part image of a base part for teaching; (b) a step of recognizing a position and an attitude of a fingertip part of the robot from a fingertip part image of the fingertip part for teaching; (c) a step of calculating an angle of a joint that the robot has from the position and the attitude of the base and the position and the attitude of the fingertip part; and (d) a step of displaying a three-dimensional image of the robot represented by the angle of the joint calculated in the step (c), in a virtual space.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a robot image display method, a computer program, and a robot image display system. [Background technology]

[0002] Patent Document 1 discloses a robot system that can perform robot teaching work by assuming that an end effector or robot peripheral devices are present, even if they are not. In this system, a virtual image of the robot's end effector or robot peripheral devices is displayed superimposed on an actual image of the robot captured by a camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-104944 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that in order to teach a robot, it is necessary to actually install the robot and actually operate the robot arm. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a method for displaying a robot image, which includes the steps of: (a) acquiring a base image created by photographing a teaching base for teaching the position and posture of a base of a robot, and recognizing the position and posture of the base from the base image; (b) acquiring an end-user image created by photographing a teaching end-user for teaching the position and posture of an end-user of the robot, and recognizing the position and posture of the end-user from the end-user image; (c) calculating angles of joints of the robot from the position and posture of the base recognized in the step (a) and the position and posture of the end-user recognized in the step (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated in the step (c) in a virtual space.

[0006] According to a second aspect of the present disclosure, there is provided a computer program for performing display processing of a robot image. This computer program causes a processor to execute the following processes: (a) acquiring a base image created by photographing a teaching base for teaching the position and posture of a base of a robot, and recognizing the position and posture of the base from the base image; (b) acquiring an end-of-hand image created by photographing a teaching end-of-hand for teaching the position and posture of an end-of-hand of the robot, and recognizing the position and posture of the end-of-hand from the end-of-hand image; (c) calculating angles of joints of the robot from the position and posture of the base recognized in the process (a) and the position and posture of the end-of-hand recognized in the process (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated in the process (c) in a virtual space.

[0007] According to a third aspect of the present disclosure, there is provided a robot image display system, comprising: a teaching base unit for teaching a position and a posture of a base of a robot, a teaching hand unit for teaching a position and a posture of a hand unit of the robot, an imaging unit for imaging the teaching base unit and the teaching hand unit, and a control unit connected to the imaging unit. The control unit executes the following processes: (a) acquiring a base unit image created by photographing the teaching base unit using the imaging unit, and recognizing the position and posture of the base from the base unit image; (b) acquiring a hand unit image created by photographing the teaching hand unit using the imaging unit, and recognizing the position and posture of the hand unit from the hand unit image; (c) calculating the angles of the joints of the robot from the position and posture of the base recognized in process (a) and the position and posture of the hand unit recognized in process (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated in process (c) in a virtual space. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing a robot teaching system according to an embodiment. [Diagram 2] FIG. 4 is an explanatory diagram showing a comparison between the teaching base unit and the teaching end unit and the robot. [Diagram 3] FIG. [Figure 4] FIG. 2 is an explanatory diagram showing the robot in the state shown in FIG. 1 displayed in augmented reality. [Diagram 5] FIG. 11 is an explanatory diagram showing another example of teaching using the teaching system. [Figure 6] FIG. 6 is an explanatory diagram showing the robot displayed in augmented reality in the state shown in FIG. 5; [Figure 7] 4 is a flowchart showing a procedure of a teaching process in the embodiment. [Figure 8] FIG. 13 is an explanatory diagram showing a robot model selection screen. [Figure 9] FIG. 11 is an explanatory diagram showing how to change the model of a robot. [Figure 10] FIG. 13 is an explanatory diagram showing a screen for changing the joint angle of the robot. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] FIG. 1 is an explanatory diagram showing a robot teaching system in an embodiment. This teaching system includes a teaching base unit 110, a teaching hand unit 130, an imaging unit 200, a control device 300, and AR (Augmented Reality) glasses 400. In this example, the teaching base unit 110 is installed on a table 500. The teaching hand unit 130 is held by the hand PH of the worker PS, and the AR glasses 400 are worn on the head of the worker PS. For convenience of illustration, the worker PS is drawn with a dashed line. The teaching hand unit 130, the imaging unit 200, and the AR glasses 400 are connected to the control device 300 by wire or wirelessly. The control device 300 corresponds to the "control unit" in this disclosure.

[0010] The teaching base unit 110 is a member for teaching the position and posture of the base of the robot. A base coordinate system Σr1 is set in the teaching base unit 110, with a predetermined reference point of the teaching base unit 110 as the origin. The teaching hand unit 130 is a member for teaching the position and posture of the hand unit of the robot. A control point TCP (Tool Center Point) is set in the teaching hand unit 130. A hand coordinate system Σt1 is set in the teaching hand unit 130, with a predetermined reference point as the origin. In this example, the origin of the hand coordinate system Σt1 is the control point TCP. One feature of this teaching system is that it teaches the movement of the robot using the teaching base unit 110 and the teaching hand unit 130 without using an actual robot. More specifically, first, the positions of the teaching base unit 110 and the teaching hand unit 130 are set, and then their images are taken by the imaging unit 200. Using these images, the positions and postures of the robot's base and hand unit are recognized, and the angles of each joint of the robot are calculated. The details of this process will be described later.

[0011] The imaging unit 200 captures images of the teaching base unit 110 and the teaching hand unit 130. As the imaging unit 200, a stereo camera, an optical distance measuring device such as Lidar (Laser Imaging Detection and Ranging), a monocular camera, etc. can be used. When using an optical distance measuring device, it is preferable to use a monocular camera together with the optical distance measuring device. This has the advantage that the distance at each position in the two-dimensional image can be accurately obtained from the distance image obtained by the optical distance measuring device and the two-dimensional image taken by the monocular camera. A camera coordinate system Σc is set in the imaging unit 200, with a predetermined reference point of the imaging unit 200 as the origin. As shown in FIG. 1, the imaging unit 200 is placed on the table 500, but it may also be attached to the top of the AR glasses 400.

[0012] FIG. 2 is an explanatory diagram showing a comparison between the teaching base unit 110 and the teaching hand unit 130 and an actual robot 600. The robot 600 includes a base 610 and a robot arm 620, and has a hand unit 630 at the tip of the robot arm 620. The hand unit 630 may include an end effector, or may be a part provided with a tool flange for attaching an end effector such as a hand without including an end effector. The end effector may be called a "tool." In this embodiment, the robot arm 620 is connected in sequence by a plurality of joints J1 to J6. As the robot 600, a robot having any arm mechanism having two or more joints can be used. In addition, the robot 600 in this embodiment is a vertical multi-joint robot, but a horizontal multi-joint robot may also be used.

[0013] A robot coordinate system Σr0 is set in the base 610 of the robot 600, with a predetermined reference point of the base 610 as the origin. This robot coordinate system Σr0 is associated with a base coordinate system Σr1 set in the teaching base 110. A control point TCP (Tool Center Point) is set in the hand 630. A hand coordinate system Σt0 is set in the hand 630 of the robot 600, with a predetermined reference point as the origin. In the example of FIG. 2, the origin of the hand coordinate system Σt0 is the control point TCP. The hand coordinate system Σt0 is associated with the hand coordinate system Σt1 set in the teaching hand 130. Note that the hand coordinate system Σt0 may be set with a position other than the control point TCP as the origin. For example, the hand coordinate system Σt0 may be set with a reference point of a tool flange at the tip of the robot arm 620 as the origin. The same applies to the hand coordinate system Σt1 set in the teaching hand 130.

[0014] The teaching base unit 110 is a member for recognizing the position and posture of the base 610 of the robot 600. In this example, the teaching base unit 110 has substantially the same shape as the base 610 of the robot 600, but a base unit of any shape can be used. The teaching base unit 110 can be formed using any material, such as a metal or plastic plate, or paper.

[0015] A first mark 112 consisting of a two-dimensional code is provided on the surface of the teaching base unit 110. This first mark 112 is used to recognize the position and orientation of the teaching base unit 110 from an image of the teaching base unit 110. The position and orientation of the teaching base unit 110 means the position and orientation of the base unit coordinate system Σr1 in the camera coordinate system Σc. For example, the first mark 112 can be configured as a black and white pattern including data representing the position and orientation of the base unit coordinate system Σr1 in the camera coordinate system Σc. As described above, the base unit coordinate system Σr1 set on the teaching base unit 110 corresponds to the robot coordinate system Σr0 set on the base 610 of the robot 600. Therefore, the position and orientation of the teaching base unit 110 recognized from the image of the teaching base unit 110 can be considered to be the same as the position and orientation of the base 610 of the robot 600. Generally, a position is expressed by three coordinate values, and an orientation is expressed by a 3×3 rotation matrix, a quaternion, etc. In addition, a position and orientation can also be expressed by a 4×4 homogeneous transformation matrix.

[0016] The first mark 112 is not limited to a two-dimensional code, and may be, for example, a convex portion, a concave portion, a light-emitting portion, or a printed pattern. In this way, the position and posture of the teaching base unit 110 can be easily recognized from the image of the first mark 112. In addition, it is preferable that the first marks 112 are provided on a plurality of surfaces of the teaching base unit 110. However, if the position and posture of the teaching base unit 110 can be recognized from the shape of the teaching base unit 110 itself, the first marks 112 can be omitted. In this case, it is preferable to adopt a shape that is not symmetrical as the shape of the teaching base unit 110. Note that the base 610 of the actual robot 600 may be used as the teaching base unit 110.

[0017] The teaching end portion 130 is a member for recognizing the position and posture of the end portion 630 of the robot 600. In this example, the teaching end portion 130 has a shape similar to that of the end portion 630 of the robot 600, but any shape can be used. The teaching end portion 130 can be formed using any material, such as a metal or plastic plate or paper.

[0018] A second mark 132 consisting of a two-dimensional mark is provided on the surface of the teaching hand 130. This second mark 132 is used to recognize the position and posture of the teaching hand 130 from the image of the teaching hand 130. The position and posture of the teaching hand 130 means the position and posture of the hand coordinate system Σt1 in the camera coordinate system Σc. For example, the second mark 132 can be configured as a black and white pattern including data representing the position and posture of the hand coordinate system Σt1 in the camera coordinate system Σc. As described above, the hand coordinate system Σt1 set in the teaching hand 130 corresponds to the hand coordinate system Σt0 set in the hand 630 of the robot 600. Therefore, the position and posture of the teaching hand 130 recognized from the image of the teaching hand 130 can be regarded as the same as the position and posture of the hand 630 of the robot 600.

[0019] The second mark 132 is not limited to a two-dimensional code, and may be, for example, a convex portion, a concave portion, a light-emitting portion, or a printed pattern. In this way, the position and posture of the teaching hand 130 can be easily recognized from the image of the second mark 132. In addition, it is preferable that the second mark 132 is provided on multiple surfaces of the teaching hand 130. However, if the position and posture of the teaching hand 130 can be recognized from the shape of the teaching hand 130 itself, the second mark 132 can be omitted. In this case, it is preferable to adopt a shape without symmetry as the shape of the teaching hand 130. Note that the hand 630 of the actual robot 600 may be used as the teaching hand 130.

[0020] The teaching hand 130 further has a button 134 and a force detection unit 136. In this embodiment, the buttons 134 include two buttons, an A button 134a and a B button 134b. These buttons 134 can be used to set teaching points and paths, change the model of the robot, and change the joint angles of the robot. For example, when setting a teaching point, pressing one of the buttons 134 registers the position and posture of the teaching hand 130 at that time as a teaching point. When setting a path, pressing one of the buttons 134 while moving the teaching hand 130 registers the path. The change of the joint angle will be described later. It is preferable to provide one or more buttons on the teaching hand 130, but the button 134 may be omitted. In this case, the operator's instructions may be given to the control device 300 by using gestures such as the movement of the operator's fingers. In this case, the gestures of the worker are captured by the AR glasses 400 or the imaging unit 200, and the instructions of the worker can be recognized by the control device 300 in accordance with the movements.

[0021] The force detection unit 136 is a sensor that measures an external force applied to the teaching hand 130. For example, a six-axis force sensor can be used as the force detection unit 136. The force detection unit 136 may be omitted.

[0022] The teaching hand 130 may be provided with other components such as a display unit for displaying the teaching state, a switch for generating an enable signal, and a sensor such as a gyroscope for improving the recognition accuracy of the position and posture of the teaching hand 130. The teaching hand 130 may be the same hand as that of the actual robot. Alternatively, the teaching hand 130 may be a wrist of the actual robot detached from the robot. In the latter case, the teaching hand 130 does not have a part corresponding to an end effector.

[0023] 3 is a block diagram showing the functions of the control device 300. The control device 300 can be realized as an information processing device such as a personal computer. The control device 300 has a processor 310, a memory 320, an interface circuit 330, and an input device 340 and a display unit 350 connected to the interface circuit 330. The interface circuit 330 is further connected to the teaching hand unit 130, the imaging unit 200, and the AR glasses 400 by wire or wirelessly. However, if the teaching hand unit 130 does not have electrical components such as the button 134 and the force detection unit 136, the teaching hand unit 130 does not need to be connected to the control device 300.

[0024] The processor 310 has the functions of a robot selection unit 312, a position and orientation recognition unit 314, a joint angle calculation unit 316, and a control program creation unit 318. The robot selection unit 312 is used when selecting one model to be a target for teaching processing from a plurality of models of the robot 600. The position and orientation recognition unit 314 recognizes the position and orientation of the base 610 and the position and orientation of the hand unit 630 of the robot 600 from an image obtained by photographing the teaching base unit 110 and the teaching hand unit 130 by the imaging unit 200. The joint angle calculation unit 316 calculates the angle of each joint of the robot 600 from the position and orientation of the base 610 and the position and orientation of the hand unit 630 recognized by the position and orientation recognition unit 314. The control program creation unit 318 creates a control program for the robot 600 using the joint angles calculated by the joint angle calculation unit 316. The functions of these units 312, 314, 316, and 318 are realized by the processor 310 executing a computer program stored in the memory 320. However, some or all of the functions of these units may be realized by hardware circuits.

[0025] The memory 320 stores robot attribute data RD, peripheral object attribute data PD, and a robot control program RP. The robot attribute data RD includes various robot characteristics such as the configuration and movable range of the robot arm 620 for multiple models of the robot 600. In addition, the robot attribute data RD preferably includes three-dimensional data representing the three-dimensional shape of the robot 600 in order to display a three-dimensional image of the robot 600 in augmented reality using the AR glasses 400. The peripheral object attribute data PD includes three-dimensional data representing the three-dimensional shape of peripheral objects present around the robot 600. Examples of peripheral objects that can be used include a stand, a shelf, a wall, a parts feeder, and the like. In addition, the peripheral object attribute data PD may include data representing the shape and weight of a workpiece handled by the robot 600. The robot control program RP is composed of multiple commands that operate the robot 600. This robot control program RP is created by a teaching process described later.

[0026] FIG. 4 is an explanatory diagram showing a state in which the robot 600 is displayed as an augmented reality in the state of FIG. 1. The 3D image of the robot 600 is superimposed and displayed on the teaching base unit 110 and the teaching end unit 130 by the AR glasses 400. More specifically, the 3D image of the robot 600 is displayed such that the image of the base 610 is arranged according to the position and posture of the base 610 recognized using the teaching base unit 110, and the image of the end unit 630 is arranged according to the position and posture of the end unit 630 recognized using the teaching end unit 130. In the example of FIG. 4, for convenience of illustration, the display positions of the base 610 and the end unit 630 of the robot 600 are drawn as positions slightly shifted from the positions of the teaching base unit 110 and the teaching end unit 130. In addition, when the shape of a wrist part without an end effector is used as the shape of the teaching end unit 130, it is preferable to display the end effector as a virtual reality. It is also preferable that the end effector be configured to change its form by moving a movable part of the end effector in a virtual space using one of the buttons 134. Note that "virtual space" refers to an artificial environment created by a computer.

[0027] FIG. 5 is an explanatory diagram showing another example of teaching using a teaching system, and FIG. 6 is an explanatory diagram showing a state in which the robot 600 is displayed in augmented reality in the state of FIG. 5. In these examples, the tip of the teaching hand 130 is pressed against the workpiece WK. In this state, when the worker presses one of the buttons 134, the teaching base 110 and the teaching hand 130 are photographed by the imaging unit 200, and the force detected by the force detection unit 136 is supplied to the control device 300. In this way, it is possible to set a preferred force at the same time as setting a teaching point. At this time, since the teaching hand 130 is held by the worker's hand, it is possible to set the preferred force more easily than when the force is input as a numerical value in Newton units. Such a preferred force is set, for example, when the workpiece WK has a button and an inspection is performed by pressing the button of the workpiece WK using the robot 600. Alternatively, the force is set even when a polishing member is provided on the hand 630 and the workpiece WK is polished.

[0028] 7 is a flowchart showing the procedure of the teaching process in one embodiment. In step S10, the model of the robot 600 is selected using the function of the robot selection unit 312.

[0029] FIG. 8 is an explanatory diagram showing a robot model selection screen WS1. In this embodiment, this selection screen WS1 is displayed as an augmented reality according to image data supplied from the robot selection unit 312 to the AR glasses 400. In the selection screen WS1, multiple robot models are arranged as options. In addition to the model name, the number of axes, maximum reach, and payload capacity are shown as attributes of the robot for each model. However, some or all of the robot attributes may be omitted. Also, an image of each model may be displayed. The worker can select a model using the button 134 provided on the teaching hand unit 130. That is, it is possible to select one of multiple models by pressing the A button 134a and to confirm by pressing the B button 134b. In the state of FIG. 8, a robot with a model name of "C1" is selected. The selection result by the worker is received by the robot selection unit 312. Note that instead of using the selection screen WS1, the worker may select a model by directly specifying the model name.

[0030] In step S20 of FIG. 7, the position and orientation recognition unit 314 captures the teaching base unit 110 and the teaching hand unit 130 using the imaging unit 200 to generate a base unit image and a hand unit image. This step S20 is started by the operator pressing one of the buttons 134 of the teaching hand unit 130, for example, the A button 134a. It is preferable that the imaging unit 200 has a sufficiently wide angle of view so that the teaching base unit 110 and the teaching hand unit 130 can be captured simultaneously. The base unit image and the hand unit image may be the same image. Alternatively, from the image captured by the imaging unit 200, an image portion including the teaching base unit 110 may be extracted as the base unit image, and the other image portion including the teaching hand unit 130 may be extracted as the hand unit image. The base unit image and the hand unit image may be captured separately. The base unit image and the hand unit image are temporarily stored in the memory 320. Moreover, a moving image may be captured by the imaging unit 200. When it is desired to set a path rather than to register a teaching point, in step S20, a moving image of the teaching hand 130 is captured.

[0031] In step S30, the position and orientation recognition unit 314 acquires the base image obtained in step S20 from the memory 320, and recognizes the position of the base 610 of the robot 600 from the base image. In this embodiment, as shown in Fig. 2, the position and orientation of the base coordinate system Σr1 in the camera coordinate system Σc are recognized from the image of the first mark 112 provided on the teaching base 110. The position and orientation of this base coordinate system Σr1 can be regarded as the position and orientation of the base 610 of the robot 600, i.e., the position and orientation of the robot coordinate system Σr0.

[0032] In step S40, the position and orientation recognition unit 314 acquires the hand image obtained in step S20 from the memory 320, and recognizes the position of the hand 630 of the robot 600 from the hand image. In this embodiment, as shown in Fig. 2, the position and orientation of the hand coordinate system Σt1 in the camera coordinate system Σc are recognized from the image of the second mark 132 provided on the teaching hand 130. The position and orientation of this hand coordinate system Σt1 can be regarded as the position and orientation of the hand 630 of the robot 600, i.e., the position and orientation of the hand coordinate system Σt0.

[0033] In step S50, the position and orientation recognition unit 314 calculates the robot coordinates of the control point TCP of the hand end unit 630 from the position and orientation of the base 610 of the robot 600 and the position and orientation of the hand end unit 630. The robot coordinates of the control point TCP are expressed by the position and orientation of the hand end unit coordinate system Σt0 in the robot coordinate system Σr0.

[0034] In step S60, the robot selection unit 312 determines whether or not the robot model needs to be changed. Specifically, if the robot coordinates of the control point TCP calculated in step S50 are within the movable range of the currently selected robot model, it is determined that the model does not need to be changed. On the other hand, if the robot coordinates of the control point TCP are outside the movable range, it is determined that the model needs to be changed. If the model does not need to be changed, the process proceeds to step S80, which will be described later, and if the model needs to be changed, the process proceeds to step S70. In step S70, the model of the robot 600 is changed using the functions of the robot selection unit 312.

[0035] FIG. 9 is an explanatory diagram showing a state in which the robot model is changed using the robot model selection screen WS1. Here, among the multiple models, "C1" and "D1" are models in which the robot coordinates of the control point TCP are outside the movable range. These models "C1" and "D1" are displayed so that the worker can visually confirm that they are invalid options that cannot be selected. In the example of FIG. 9, the models "C1" and "D1" are shaded, and the selection box is also in a state in which it cannot be selected. The other models are presented as a list of valid options that can be selected. The worker can change the robot model by selecting one of the valid options. In the example of FIG. 9, the model "C2" is selected as the model after the change. In this way, when the control point TCP of the hand part 630 is outside the movable range of the robot, the control point TCP can be brought within the movable range of the robot by changing the robot model. Note that it is not necessary to display multiple valid options, but it is preferable to display one or more valid options. In addition, the control point TCP that is determined to be outside the movable range may not be only the control point TCP calculated in step S50. For example, the control points TCP stored before step S50, that is, the control points TCP stored in the past teaching process, may be added. In this case, among the multiple models, any model in which any one of the above-mentioned multiple control points TCP is outside the movable range may be shaded as an invalid option that cannot be selected.

[0036] In this way, in step S70, one or more robot models whose position of the control point TCP in the robot coordinate system Σr0 falls within the movable range of the robot are presented to the worker, and the model selected by the worker from the one or more models is adopted. In this way, the robot model can be easily changed. Note that instead of using the selection screen WS1, the model may be changed by the worker directly specifying the model name.

[0037] In step S80, the joint angle calculation unit 316 calculates the angle of each joint of the robot 600 from the position and posture of the control point TCP in the robot coordinate system Σr0. This calculation is performed according to inverse kinematics. Generally, there are many cases where a plurality of combinations of angles of each joint calculated by inverse kinematics are possible. In this case, in step S80, one of the plurality of combinations is selected according to a predetermined rule.

[0038] In step S90, the joint angle calculation unit 316 determines whether or not a change in the joint angle is required. In this embodiment, when the shape of the robot arm 620 represented by the angles of each joint in the virtual space is in an interference state that may cause interference with a peripheral object, it is determined that a change in the joint angle is required. The outer shape of the peripheral object is represented by three-dimensional data included in the peripheral object attribute data PD stored in the memory 320. The joint angle calculation unit 316 calculates the distance between the outer shape of the peripheral object represented by the three-dimensional data of the peripheral object and the shape of the robot arm 620 represented by the joint angle calculated in step S80, and when the distance is equal to or less than a predetermined threshold, it is determined that there is an interference state, and it is possible to determine that a change in the joint angle is required. For example, a value between 0 and 10 cm is set as the threshold. When the shape of the peripheral object and the shape of the robot arm 620 are not in an interference state in the virtual space, it is determined that a change in the joint angle is not required. Note that when the shape of the robot arm 620 is in an interference state that may cause interference with a peripheral object, the joint angle calculation unit 316 may notify the operator of an alert indicating that there is an interference state. If the joint angle does not need to be changed, the process proceeds to step S110, which will be described later, and if the joint angle needs to be changed, the process proceeds to step S100. In step S100, the joint angle of the robot 600 is changed using the function of the joint angle calculation unit 316.

[0039] FIG. 10 is an explanatory diagram showing a screen WS2 for changing the joint angle of the robot. In this embodiment, this screen WS2 is displayed as an augmented reality according to the image data supplied from the joint angle calculation unit 316 to the AR glasses 400. In this example, two options are displayed as a combination of joint angles. The angles A1 and A2 are different combinations of the angles of the three joints J2, J3, and J5, but the position and posture of the hand 630 are the same. At the angle A1, the robot arm 620 may interfere with the surrounding object PB, and at the angle A2, the robot arm 620 may not interfere with the surrounding object PB. Therefore, in this example, the angle A2 is selected as the combination of joint angles after the change. In this way, the joint angle can be changed by selecting one from among multiple options regarding the combination of joint angles. Such a change in the joint angle is performed so as to maintain the position and posture of the hand 630. In addition, the change in the joint angle is performed according to an instruction from the operator. As a result, when the robot arm 620 is in an interference state in which there is a possibility of interference with the surrounding object PB in the virtual space, the interference state can be resolved by changing the joint angles. Note that, instead of the operator selecting one of the options, the joint angle calculation unit 316 may automatically determine a combination of joint angles that resolves the interference itself.

[0040] The determination in step S90 does not need to be automatically performed by the joint angle calculation unit 316. Instead, when an operator specifies that the joint angle needs to be changed, the joint angle may be determined to need to be changed. That is, the operator may view images of the robot arm 620 and the peripheral object PB and determine whether the robot arm 620 and the peripheral object PB are in an interference state. In this case, too, it is preferable that the combination of joint angles is selected by the operator. In this way, the operator can freely select a preferred state of the robot arm 620.

[0041] In step S110, the joint angle calculation unit 316 displays an image of the robot 600 in the virtual space. In this embodiment, the robot 600 is displayed as an augmented reality according to the image data supplied from the joint angle calculation unit 316 to the AR glasses 400. For example, as shown in Figs. 4 and 6 described above, the three-dimensional image of the robot 600 is displayed with an image of the base 610 of the robot 600 superimposed on the teaching base unit 110, and with an image of the hand unit 630 of the robot 600 superimposed on the teaching hand unit 130.

[0042] In step S120, the operator determines whether the state of the robot 600 displayed in step S110 is suitable as a teaching point. If it is not suitable as a teaching point, the process returns to step S20, where the operator changes the position of the teaching hand 130 and captures an image again. On the other hand, if it is suitable as a teaching point, the process proceeds to step S130.

[0043] In step S130, the control program creation unit 318 registers the state in which the image was captured in step S20 as a teaching point. This step S130 is started by the operator pressing one of the buttons 134 of the teaching hand unit 130, for example, the B button 134b. The teaching point is registered in the robot control program RP in the memory 320. Also, when it is desired to set a path rather than registering a teaching point, that is, when a video is captured, the control program creation unit 318 generates multiple still images in chronological order from the video, and records the state of the still images as a teaching point. The control program creation unit 318 generates a path from the multiple teaching points recorded in this way.

[0044] In step S140, the operator determines whether the teaching process has been completed. If the teaching process has not been completed, the process returns to step S20, and the operator changes the position of the teaching hand 130 and captures an image again. On the other hand, if the teaching process has been completed, the process in FIG. 7 is terminated.

[0045] After the process of FIG. 7, a process for setting other setting items for the robot control program RP may be executed as a part of the teaching process. After the teaching process is completed, the robot control program RP created by the teaching process may be used to reproduce the operation of the robot 600 in a virtual space. For example, when the robot 600 is used to transport a workpiece, the shape and weight of the workpiece may be stored in the memory 320 in advance, and the robot 600 may transport the workpiece in a virtual space. In this way, the worker can determine whether the workpiece transport operation is appropriate without actually operating the robot 600. After the joint angle of the robot 600 is changed in step S100, an image of the robot 600 is displayed in the virtual space in step S110. However, after step S80, an image of the robot 600 may be displayed in the virtual space, and a determination may be made to change the joint angle of the robot 600 after that. In such a case, the display and change of a plurality of joint angles may be repeated. In particular, in a robot with a high degree of redundant freedom such as a vertical articulated robot, an optimal joint angle can be selected from a plurality of joint angles.

[0046] As described above, in the above embodiment, the position and posture of the base 610 of the robot 600 are recognized from the image of the teaching base unit 110, the position and posture of the hand unit 630 of the robot 600 are recognized from the image of the teaching hand unit 130, the angles of the joints of the robot 600 are calculated using these positions and postures, and a three-dimensional image of the robot 600 represented by the calculated joint angles is displayed in a virtual space. Therefore, the operator can easily understand what state the robot arm will be in and can teach the robot without actually installing or operating the robot.

[0047] In the above-described embodiment, the AR glasses 400 are used to display three-dimensional images of the robot 600 and the peripheral object PB as virtual reality. Instead, the display unit 350 that displays two-dimensional images may be used to display images of the robot 600 and the peripheral object PB in a virtual space. This case is also common to the case where the AR glasses 400 are used in that the image of the robot 600 is displayed in a virtual space. However, if the AR glasses 400 are used to display a three-dimensional image of the robot 600 in augmented reality, there is an advantage that the worker can easily understand the posture of the robot 600.

[0048] Other embodiments: The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various forms without departing from the spirit of the present disclosure. For example, the present disclosure can be realized in the following aspects. The technical features in the above-mentioned embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined in order to solve some or all of the problems of the present disclosure, or to achieve some or all of the effects of the present disclosure. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0049] (1) According to a first aspect of the present disclosure, there is provided a method for displaying a robot image, the method including the steps of: (a) acquiring a base image created by photographing a teaching base for teaching the position and posture of a base of a robot, and recognizing the position and posture of the base from the base image; (b) acquiring a hand image created by photographing a teaching end part for teaching the position and posture of an end part of the robot, and recognizing the position and posture of the end part from the hand image; (c) calculating angles of joints of the robot from the position and posture of the base recognized in step (a) and the position and posture of the end part recognized in step (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated in step (c) in a virtual space. According to this display method, an operator can easily understand what state the robot arm will be in and can teach the robot without actually installing or operating the robot.

[0050] (2) In the above display method, the step (c) may include the steps of: (i) calculating, in a robot coordinate system, the position of a control point set in the endoscope unit from the position and posture of the base recognized in the step (a) and the position and posture of the endoscope unit recognized in the step (b); and (ii) changing the model of the robot when the position of the control point in the robot coordinate system is outside the movable range of the robot. According to this display method, when the control point of the end effector is outside the movable range of the robot, the control point of the end effector can be brought within the movable range of the robot by changing the model of the robot.

[0051] (3) In the above display method, the step (ii) may include a step of presenting to an operator one or more models of the robot, the position of the control point in the robot coordinate system being within a movable range of the robot, and a step of adopting the model selected by the operator from the one or more models. This display method allows the robot model to be easily changed.

[0052] (4) In the above display method, the step (c) may include an angle changing step of selecting and changing one of a plurality of possible combinations of angles of the joints. This display method allows a user to select a desirable state for the robot arm.

[0053] (5) In the above display method, the combination of joint angles may be selected by an operator. This display method allows the operator to freely select a desired state for the robot arm.

[0054] (6) In the above display method, the angle changing step may include an interference resolution step of, when the shape of the robot arm represented by the joint angles calculated in step (c) is in an interference state in which there is a possibility of interference with a surrounding object, changing the combination of the angles of the joints while maintaining the position and posture of the hand unit to resolve the interference state. This display method can reduce the possibility of interference between the robot arm and surrounding objects.

[0055] (7) In the above display method, the interference resolution step includes a step of calculating a distance between the peripheral object represented by the three-dimensional data of the peripheral object, the peripheral object represented by the three-dimensional data of the peripheral object, and a shape of the robot arm represented by the angles of the joints calculated in step (c); and a step of determining that the interference state exists when the distance is equal to or smaller than a predetermined threshold, and automatically determining a combination of angles of the joints that resolves the interference state while maintaining the position and posture of the hand. It may also include. This display method makes it possible to automatically eliminate interference between the robot arm and surrounding objects.

[0056] (8) In the above display method, the three-dimensional image may be superimposed and displayed as an augmented reality on the teaching base and the teaching hand in the step (d). According to this display method, the robot arm is displayed as augmented reality, so that the worker can easily understand what state the robot arm will be in.

[0057] (9) In the above display method, the teaching base unit may be provided with a first mark used to recognize the position and posture of the base of the robot, and the teaching hand unit may be provided with a second mark used to recognize the position and posture of the hand unit of the robot. According to this display method, the position and posture of the base and the position and posture of the hand tip can be easily recognized using the first mark and the second mark.

[0058] (10) In the above display method, each of the first mark and the second mark may include a two-dimensional code, a convex portion, a concave portion, a light-emitting portion, or a printed pattern. According to this display method, the position and posture of the base and the position and posture of the hand tip can be easily recognized from the first mark and the second mark.

[0059] (11) According to a second aspect of the present disclosure, there is provided a computer program for performing display processing of a robot image. This computer program causes a processor to execute the following processes: (a) acquiring a base image created by photographing a teaching base for teaching the position and posture of a base of a robot, and recognizing the position and posture of the base from the base image; (b) acquiring an end effector image created by photographing a teaching hand for teaching the position and posture of an end effector of the robot, and recognizing the position and posture of the hand from the end effector image; (c) calculating angles of joints of the robot from the position and posture of the base recognized in the process (a) and the position and posture of the hand recognized in the process (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated in the process (c) in a virtual space. This computer program allows an operator to easily understand what state the robot arm will be in and to teach the robot, without actually installing or operating the robot.

[0060] (12) According to a third aspect of the present disclosure, there is provided a robot image display system, comprising: a teaching base unit for teaching a position and posture of a base of a robot, a teaching hand unit for teaching a position and posture of a hand unit of the robot, an imaging unit for imaging the teaching base unit and the teaching hand unit, and a control unit connected to the imaging unit. The control unit executes the following processes: (a) acquiring a base unit image created by photographing the teaching base unit using the imaging unit, and recognizing the position and posture of the base from the base unit image; (b) acquiring a hand unit image created by photographing the teaching hand unit using the imaging unit, and recognizing the position and posture of the hand unit from the hand unit image; (c) calculating the angles of the joints of the robot from the position and posture of the base recognized in process (a) and the position and posture of the hand unit recognized in process (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated in process (c) in a virtual space. This teaching system allows an operator to easily understand what state the robot arm will be in and teach the robot, without actually installing or operating the robot.

[0061] The present disclosure may be realized in various forms other than those described above, such as a robot system including a robot and a control device, a computer program for implementing the functions of the control device of the robot, a non-transitory storage medium on which the computer program is recorded, and the like. [Explanation of symbols]

[0062] 110...teaching base unit, 112...first mark, 130...teaching hand unit, 132...second mark, 134...button, 134a...A button, 134b...B button, 136...force detection unit, 200...imaging unit, 300...control device, 310...processor, 312...robot selection unit, 314...position and orientation recognition unit, 316...joint angle calculation unit, 318...control program creation unit, 320...memory, 330...interface circuit, 350...display unit, 400...AR glasses, 500...table, 600...robot, 610...base, 620...robot arm, 630...hand unit

Claims

1. (a) By photographing a teaching base unit for teaching the position and posture of a base of a robot A base part image created by the above is acquired, and the position and orientation of the base are recognized from the base part image. And a process of (b) taking an image of a teaching end point for teaching the position and posture of the end point of the robot; and acquiring a hand image created by the above-mentioned method, and determining the position and posture of the hand from the hand image. Recognizing the momentum; (c) The position and orientation of the base recognized in the step (a) and the position and orientation of the base recognized in the step (b) calculating angles of joints of the robot from the position and posture of the end effector; 、 (d) a three-dimensional representation of the robot represented by the angles of the joints calculated in (c); Displaying the image in a virtual space; (e) The position and orientation of the base recognized in the step (a) and the position and orientation of the base recognized in the step (b) The position of the control point set in the end effector is calculated based on the position and posture of the end effector. calculating in a standard system; (f) the position of the control point in the robot coordinate system is outside the movable range of the robot In this case, the position of the control point in the robot coordinate system is presenting to an operator one or more models that are within a range of motion of the robot; (g) changing the model to a model selected by the operator from among the one or more models; , including display methods.

2. The display method according to claim 1, further comprising: The step (c) When a plurality of combinations of angles of the joints are possible, an angle changing step of selecting and changing one of the combinations; Including, how to display.

3. The display method according to claim 2, A display method in which the combination of joint angles is selected by an operator.

4. The display method according to claim 2, The angle changing step includes: The shape of the robot arm represented by the joint angles calculated in the step (c) is compared with the surrounding objects. When the hand is in an interference state where there is a possibility of interference with the body, the hand is maintained in position and posture. and an interference solution that resolves the interference state by changing the combination of angles of the joints. Extinguishing process, Including, how to display.

5. The display method according to claim 4, The interference cancellation step includes: the surrounding object represented by the three-dimensional data of the surrounding object; calculating a distance between the shape of the robot arm represented by the joint angle calculated by the and, When the distance is equal to or smaller than a predetermined threshold, it is determined that the interference state exists, and A combination of angles of the joints that eliminates the interference state while maintaining the position and orientation of the joints is automatically determined. and Including, how to display.

6. The display method according to any one of claims 1 to 5, further comprising: In the step (d), the three-dimensional image is displayed as an augmented reality on the teaching base unit and the teaching hand. A display method in which the information is superimposed on the front.

7. The display method according to any one of claims 1 to 6, The teaching base unit includes a display unit for displaying a display used to recognize the position and posture of the base of the robot. a first mark is provided, The teaching end effector unit includes a teaching device for detecting the position and posture of the end effector unit of the robot. A second mark is provided for use in the Display method.

8. The display method according to claim 7, Each of the first mark and the second mark may include a two-dimensional code, a convex portion, a concave portion, a light-emitting portion, Or a display method including a printed pattern.

9. (a) By photographing a teaching base unit for teaching the position and posture of a base of a robot A base part image created by the above is acquired, and the position and orientation of the base are recognized from the base part image. and (b) taking an image of a teaching end point for teaching the position and posture of the end point of the robot; and acquiring a hand image created by the above-mentioned method, and determining the position and posture of the hand from the hand image. A process of recognizing the momentum; (c) The position and orientation of the base recognized in the process (a) and the position and orientation of the base recognized in the process (b) and calculating angles of the joints of the robot from the position and posture of the end effector. 、 (d) a three-dimensional image of the robot represented by the angles of the joints calculated in the process (c); A process of displaying the image in a virtual space; (e) The position and orientation of the base recognized in the process (a) and the position and orientation of the base recognized in the process (b) The position of the control point set in the end effector is calculated based on the position and posture of the end effector. A process of calculating in the standard system; (f) the position of the control point in the robot coordinate system is outside the movable range of the robot In this case, the position of the control point in the robot coordinate system is A process of presenting one or more models that are within the movable range of the robot to the worker; (g) changing the model to a model selected by the operator from the one or more models; A computer program that causes a processor to execute.

10. a teaching base unit for teaching the position and orientation of the base of the robot; A teaching end portion for teaching a position and a posture of the end portion of the robot; an imaging unit for imaging the teaching base unit and the teaching tip unit; A control unit connected to the imaging unit; Equipped with The control unit is (a) a base image created by photographing the teaching base using the photographing unit; and recognizing the position and orientation of the base from the base portion image. (b) a hand image created by photographing the teaching hand using the photographing unit; a process of acquiring an image of the hand and recognizing a position and a posture of the hand from the image of the hand; (c) The position and orientation of the base recognized in the process (a) and the position and orientation of the base recognized in the process (b) and calculating angles of the joints of the robot from the position and posture of the end effector. 、 (d) a three-dimensional image of the robot represented by the angles of the joints calculated in the process (c); A process of displaying the image in a virtual space; (e) The position and orientation of the base recognized in the process (a) and the position and orientation of the base recognized in the process (b) The position of the control point set in the end effector is calculated based on the position and posture of the end effector. A process of calculating in the standard system; (f) the position of the control point in the robot coordinate system is outside the movable range of the robot In this case, the position of the control point in the robot coordinate system is A process of presenting one or more models that are within the movable range of the robot to the worker; (g) changing the model to a model selected by the operator from the one or more models; , which executes the display system.

Images