Robot image display method, display system, and recording medium

Abstract

The present disclosure relates to a method, display system, and recording medium for displaying robot images, providing a technology that enables robot teaching without actually setting up the robot and operating it. The method includes: (a) recognizing the position and posture of the robot's base based on a base image for teaching; (b) recognizing the position and posture of the robot's fingertips based on fingertip images for teaching; (c) calculating the angles of one or more joints of the robot based on the position and posture of the base and the position and posture of the fingertips; and (d) displaying a three-dimensional image of the robot in a virtual space, with the joints at the angles calculated in step (c).

Description

Technical Field

[0001] The present disclosure relates to a method for displaying a robot image, a recording medium, and a system for displaying a robot image. Background Art

[0002] Patent Document 1 discloses a robot system that allows robot teaching to be performed even when an end effector or robot peripherals are not present, by assuming these devices are present. In this system, a virtual image of the robot's end effector or peripherals is displayed superimposed on an actual image of the robot captured by a camera.

[0003] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-104944

[0004] However, in the conventional technology, there is a problem that in order to teach the robot, it is necessary to actually install the robot and actually operate the robot arm. Summary of the Invention

[0005] According to a first aspect of the present disclosure, a method for displaying a robot image is provided. The method includes the following steps: (a) acquiring a base portion image created by photographing a teaching base portion for teaching a robot a position and posture, and recognizing the position and posture of the base portion based on the base portion image; (b) acquiring a fingertip portion image created by photographing a teaching fingertip portion for teaching a robot a position and posture, and recognizing the position and posture of the fingertip portion based on the fingertip portion image; (c) calculating angles of joints of the robot based on the position and posture of the base portion recognized in step (a) and the position and posture of the fingertip portion recognized in step (b); and (d) displaying a three-dimensional image of the robot in a virtual space, the angles of the joints calculated in step (c).

[0006] According to a second aspect of the present disclosure, a computer program for performing robot image display processing is provided. The computer program causes a processor to execute the following processing: (a) acquiring a base portion image created by photographing a teaching base portion for teaching a robot a position and posture of a base portion, and recognizing the position and posture of the base portion based on the base portion image; (b) acquiring a fingertip portion image created by photographing a teaching fingertip portion for teaching a robot a position and posture of a fingertip portion, and recognizing the position and posture of the fingertip portion based on the fingertip portion image; (c) calculating the angles of the joints of the robot based on the position and posture of the base portion recognized by the processing (a) and the position and posture of the fingertip portion recognized by the processing (b); and (d) displaying a three-dimensional image of the robot in a virtual space, represented by the angles of the joints calculated by the processing (c).

[0007] According to a third aspect of the present disclosure, a robot image display system is provided. The teaching system includes: a teaching base unit for teaching the position and posture of a robot base; a teaching fingertip unit for teaching the position and posture of the robot's fingertips; a camera unit for capturing images of the teaching base unit and the teaching fingertip unit; and a control unit connected to the camera unit. The control unit performs the following processing: (a) obtaining a base portion image created by photographing the teaching base portion using the photographic unit, and identifying the position and posture of the base based on the base portion image; (b) obtaining a fingertip portion image created by photographing the teaching fingertip portion using the photographic unit, and identifying the position and posture of the fingertip portion based on the fingertip portion image; (c) calculating the angles of the joints of the robot based on the position and posture of the base identified by the processing (a) and the position and posture of the fingertip portion identified by the processing (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated by the processing (c) in a virtual space. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is an explanatory diagram showing a teaching system for a robot in the embodiment.

[0009] Figure 2 This is an explanatory diagram showing a comparison between the teaching base unit and the teaching fingertip unit and the robot.

[0010] Figure 3 This is a functional block diagram of the control device.

[0011] Figure 4 It is shown in Figure 1 An illustration showing the robot's status in augmented reality.

[0012] Figure 5 This is an explanatory diagram showing another example of teaching using the teaching system.

[0013] Figure 6 It is shown in Figure 5 An illustration showing the robot's status in augmented reality.

[0014] Figure 7 This is a flowchart showing the procedure of the teaching process in the embodiment.

[0015] Figure 8 This is an explanatory diagram showing a robot model selection screen.

[0016] Figure 9 This is an explanatory diagram showing a case where the model of a robot is changed.

[0017] Figure 10 This is an explanatory diagram showing a screen for changing the joint angles of a robot.

[0018] Description of Reference Numerals

[0019] 110…teaching base unit, 112…first mark, 130…teaching fingertip unit, 132…second mark, 134…button, 134a…button A, 134b…button B, 136…force detection unit, 200…camera unit, 300…control device, 310…processor, 312…robot selection unit, 314…position and posture recognition unit, 316…joint angle calculation unit, 318…control program creation unit, 320…memory, 330…interface circuit, 350…display unit, 400…AR glasses, 500…workbench, 600…robot, 610…base, 620…robotic arm, 630…fingertip unit. DETAILED DESCRIPTION

[0020] Figure 1 : This is an explanatory diagram of the teaching system of the robot in the embodiment. The teaching system includes a teaching base portion 110, a teaching fingertip portion 130, a camera portion 200, a control device 300, and AR (Augmented Reality) glasses 400. In this example, the teaching base portion 110 is set on a workbench 500. In addition, the teaching fingertip portion 130 is held by the hand PH of the operator PS, and the AR glasses 400 are assembled on the head of the operator PS. It should be noted that for the convenience of illustration, the operator PS is drawn by dotted lines. The teaching fingertip portion 130, the camera portion 200, and the AR glasses 400 are connected to the control device 300 by wire or wirelessly. The control device 300 is equivalent to the "control unit" in the present disclosure.

[0021] The teaching base unit 110 is a component for teaching the position and posture of the base of the robot. A base unit coordinate system Σr1 with a predetermined reference point of the teaching base unit 110 as the origin is set in the teaching base unit 110. The teaching fingertip unit 130 is a component for teaching the position and posture of the fingertip of the robot. A control point TCP (Tool Center Point) is set in the teaching fingertip unit 130. A fingertip unit coordinate system Σt1 with a predetermined reference point as the origin is set in the teaching fingertip unit 130. In this example, the origin of the fingertip unit coordinate system Σt1 is the control point TCP. A feature of this teaching system is that the actual robot is not used, but the teaching base unit 110 and the teaching fingertip unit 130 are used to teach the robot's movements. More specifically, after the positions of the teaching base unit 110 and the teaching fingertip unit 130 are set, images of these units are captured by the imaging unit 200. These images are used to identify the positions and postures of the robot's base unit and fingertip units, and to calculate the angles of each joint of the robot. This process will be described in detail later.

[0022] The camera unit 200 captures images of the teaching base unit 110 and the teaching fingertip unit 130. As the camera unit 200, a stereo camera, an optical distance measuring device such as Lidar (Laser Imaging Detection and Ranging), and a monocular camera can be used. When using an optical distance measuring device, it is preferable to use a monocular camera together with the optical distance measuring device. This has the advantage of being able to correctly calculate the distance of each position in the two-dimensional image based on the distance image obtained by the optical distance measuring device and the two-dimensional image captured by the monocular camera. A camera coordinate system Σc with a predetermined reference point of the camera unit 200 as the origin is set in the camera unit 200. It should be noted that, as Figure 1 As shown, the imaging unit 200 is disposed on the table 500 , but may also be mounted on the upper portion of the AR glasses 400 .

[0023] Figure 2This is an explanatory diagram showing a comparison between the teaching base portion 110 and the teaching fingertip portion 130 and the real robot 600. The robot 600 includes a base 610 and a robotic arm 620, and a fingertip portion 630 at the front end of the robotic arm 620. The fingertip portion 630 may also include an end effector, or may be a portion that does not include an end effector such as a hand and is provided with a tool flange for mounting an end effector. The end effector is sometimes referred to as a "tool". In the present embodiment, a plurality of joints J1 to J6 are connected in sequence in the robotic arm 620. As the robot 600, a robot having an arbitrary arm structure having two or more joints can be used. In addition, the robot 600 of the present embodiment is a vertical multi-joint robot, but a horizontal multi-joint robot may also be used.

[0024] A robot coordinate system Σr0 is set in the base 610 of the robot 600, with a predetermined reference point of the base 610 as the origin. The robot coordinate system Σr0 corresponds to the base coordinate system Σr1 set in the teaching base 110. A control point TCP (Tool Center Point) is set in the fingertip 630. A fingertip coordinate system Σt0 is set in the fingertip 630 of the robot 600, with a predetermined reference point as the origin. Figure 2 In the example, the origin of the fingertip coordinate system Σt0 is the control point TCP. The fingertip coordinate system Σt0 corresponds to the fingertip coordinate system Σt1 set on the teaching fingertip unit 130. It should be noted that the fingertip coordinate system Σt0 can also be set at a location other than the control point TCP as its origin. For example, the fingertip coordinate system Σt0 can also be set using the reference point of the tool flange at the front end of the robot arm 620 as its origin. The same applies to the fingertip coordinate system Σt1 set on the teaching fingertip unit 130.

[0025] The teaching base 110 is a component used to identify the position and posture of the base 610 of the robot 600. In this example, the teaching base 110 has a shape substantially identical to the base 610 of the robot 600. However, a base having any shape may be used. Furthermore, the teaching base 110 may be formed of any material, such as a metal or plastic plate, or paper.

[0026] A first mark 112, consisting of a QR code, is provided on the surface of the teaching base 110. This first mark 112 is used to identify the position and posture of the teaching base 110 based on an image of the teaching base 110. The position and posture of the teaching base 110 refer to the position and posture of the base coordinate system Σr1 within the camera coordinate system Σc. For example, the first mark 112 can be configured as a black and white pattern containing data representing the position and posture of the base coordinate system Σr1 within the camera coordinate system Σc. As described above, the base coordinate system Σr1 set on the teaching base 110 corresponds to the robot coordinate system Σr0 set on the base 610 of the robot 600. Therefore, the position and posture of the teaching base 110 identified from the image of the teaching base 110 can be considered to be the same as the position and posture of the base 610 of the robot 600. It should be noted that, in general, position is represented by three coordinate values, and posture is represented by a 3×3 rotation matrix, quaternion, or the like. In addition, the position and orientation can also be represented by a 4×4 homogeneous transformation matrix.

[0027] The first mark 112 is not limited to a QR code, and for example, a convex portion, a concave portion, a light-emitting portion, or a printed pattern can be used. Thus, the position and posture of the teaching base portion 110 can be easily identified based on the image of the first mark 112. In addition, it is preferred that the first mark 112 is provided on multiple surfaces of the teaching base portion 110. However, in a case where the position and posture of the teaching base portion 110 can be identified based on the shape of the teaching base portion 110 itself, the first mark 112 can be omitted. In this case, as the shape of the teaching base portion 110, a shape without symmetry is preferably adopted. It should be noted that as the teaching base portion 110, the base 610 of the robot 600 of a real machine can also be used.

[0028] Teaching fingertip 130 is a component used to identify the position and posture of fingertip 630 of robot 600. In this example, teaching fingertip 130 has a shape similar to fingertip 630 of robot 600, but fingertips of any shape may be used. Furthermore, teaching fingertip 130 may be formed of any material, such as a metal or plastic plate, or paper.

[0029] A second marker 132, consisting of a two-dimensional marker, is provided on the surface of the teaching fingertip 130. This second marker 132 is used to identify the position and posture of the teaching fingertip 130 based on the image of the teaching fingertip 130. The position and posture of the teaching fingertip 130 refers to the position and posture of the fingertip coordinate system Σt1 in the camera coordinate system Σc. For example, the second marker 132 can be configured as a black and white pattern including data representing the position and posture of the fingertip coordinate system Σt1 in the camera coordinate system Σc. As described above, the fingertip coordinate system Σt1 set for the teaching fingertip 130 corresponds to the fingertip coordinate system Σt0 set for the fingertip 630 of the robot 600. Therefore, the position and posture of the teaching fingertip 130 identified based on the image of the teaching fingertip 130 can be considered to be the same as the position and posture of the fingertip 630 of the robot 600.

[0030] The second mark 132 is not limited to a QR code, and for example, a convex portion, a concave portion, a luminous portion, or a printed pattern can be used. Thus, the position and posture of the teaching fingertip portion 130 can be easily identified based on the image of the second mark 132. In addition, it is preferred that the second mark 132 is provided on multiple surfaces of the teaching fingertip portion 130. However, in a case where the position and posture of the teaching fingertip portion 130 can be identified based on the shape of the teaching fingertip portion 130 itself, the second mark 132 can be omitted. In this case, as the shape of the teaching fingertip portion 130, a shape without symmetry is preferably adopted. It should be noted that as the teaching fingertip portion 130, the fingertip portion 630 of the robot 600 of a real machine can also be used.

[0031] The teaching fingertip 130 also includes buttons 134 and a force detector 136. In this embodiment, the buttons 134 include two buttons: A button 134a and B button 134b. These buttons 134 can be used to set teaching points, paths, change the robot model, or change the robot's joint angles. For example, when setting a teaching point, pressing one of the buttons 134 registers the position and posture of the teaching fingertip 130 at that point in time as the teaching point. Furthermore, when setting a path, pressing one of the buttons 134 and moving the teaching fingertip 130 registers the path. Changing joint angles will be described later. While it is preferable to have one or more buttons on the teaching fingertip 130, buttons 134 may be omitted. In this case, the operator's instructions can be provided to the control device 300 using gestures, such as finger movements. In this case, the AR glasses 400 and the camera unit 200 capture the operator's gestures, and the control device 300 can recognize the operator's instructions based on their movements.

[0032] The force detection unit 136 is a sensor for measuring the external force applied to the teaching fingertip 130. For example, a six-axis force sensor can be used as the force detection unit 136. The force detection unit 136 may also be omitted.

[0033] It should be noted that the teaching fingertip 130 may also include other components such as a display unit that shows the teaching status, a switch that generates an enable signal, and a sensor such as a gyroscope that improves the accuracy of recognizing the position and posture of the teaching fingertip 130. Furthermore, the teaching fingertip 130 may be the same hand as that of a real robot. Alternatively, the teaching fingertip 130 may be the fingertip of a robot wrist that has been removed from the real robot. In the latter case, the teaching fingertip 130 does not have a portion corresponding to the end effector.

[0034] Figure 3 3 is a block diagram showing the functions of the control device 300. The control device 300 can be implemented as an information processing device such as a personal computer. The control device 300 has a processor 310, a memory 320, an interface circuit (IF) 330, an input device 340 connected to the interface circuit 330, and a display unit 350. The interface circuit 330 is also connected to the teaching fingertip 130, the camera unit 200, and the AR glasses 400 in a wired or wireless manner. However, in the case where electrical components such as the button 134 and the force detection unit 136 are not provided in the teaching fingertip 130, the teaching fingertip 130 does not need to be connected to the control device 300.

[0035] Processor 310 includes the functions of a robot selection unit 312, a position and posture recognition unit 314, a joint angle calculation unit 316, and a control program creation unit 318. Robot selection unit 312 is used to select a model from multiple models of robot 600 to be used for teaching. Position and posture recognition unit 314 recognizes the position and posture of base 610 and the position and posture of fingertips 630 of robot 600 based on images captured by imaging unit 200 of teaching base 110 and teaching fingertips 130. Joint angle calculation unit 316 calculates the angles of each joint of robot 600 based on the position and posture of base 610 and fingertips 630 recognized by position and posture recognition unit 314. Control program creation unit 318 creates a control program for robot 600 using the joint angles calculated by joint angle calculation unit 316. The functions of these units 312, 314, 316, and 318 are implemented by processor 310 executing a computer program stored in memory 320. However, part or all of the functions of these components may be realized by hardware circuits.

[0036] The robot attribute data RD, the surrounding object attribute data PD, and the robot control program RP are stored in the memory 320. The robot attribute data RD includes various robot characteristics such as the structure of the robot arm 620 and the movable range for multiple models of the robot 600. In addition, in order to use the AR glasses 400 to display a three-dimensional image of the robot 600 in an augmented reality manner, the robot attribute data RD preferably includes three-dimensional data representing the three-dimensional shape of the robot 600. The surrounding object attribute data PD includes three-dimensional data representing the three-dimensional shape of the surrounding objects existing around the robot 600. As the surrounding objects, for example, a stand, a shelf, a wall, a parts feeder, etc. can be used. In addition, the surrounding object attribute data PD may also include data representing the shape and weight of the workpiece handled by the robot 600. The robot control program RP is composed of a plurality of commands for causing the robot 600 to move. The robot control program RP is created through the teaching process described later.

[0037] Figure 4 It is shown in Figure 1 3D image of the robot 600 is displayed as an augmented reality in a state of being displayed as an augmented reality. The three-dimensional image of the robot 600 is displayed overlapping with the teaching base unit 110 and the teaching fingertip unit 130 through the AR glasses 400. More specifically, the three-dimensional image of the robot 600 is displayed as an image of the base 610 configured according to the position and posture of the base 610 identified using the teaching base unit 110, and further, an image of the fingertip unit 630 is displayed according to the position and posture of the fingertip unit 630 identified using the teaching fingertip unit 130. It should be noted that in Figure 4 In the example, for ease of illustration, the display positions of the robot 600's base 610 and fingertip 630 are drawn slightly offset from the positions of the teaching base 110 and teaching fingertip 130. It should be noted that when the shape of the teaching fingertip 130 is a wrist without an end effector, it is preferable that the end effector also be displayed as a virtual reality. Furthermore, it is preferable that a single button 134 be used to move the movable portion of the end effector in virtual space, thereby switching its form. It should be noted that "virtual space" refers to an artificial environment created by a computer.

[0038] Figure 5 is an explanatory diagram showing another example of teaching using the teaching system, Figure 6 It is shown in Figure 5, which is an illustration of the robot 600 being displayed in an augmented reality manner in a state in which the robot 600 is in a state of being held. In these examples, the tip of the teaching fingertip 130 is pressed against the workpiece WK. In this state, when the operator presses one of the buttons 134, the teaching base 110 and the teaching fingertip 130 are photographed by the camera 200, and the force detected by the force detection unit 136 is supplied to the control device 300. Thus, when setting the teaching point, the preferred force can be set at the same time. In this case, the teaching fingertip 130 is held by the operator's hand, so the preferred force can be set more easily than when the force is input in numerical values in Newtons. Such preferred force setting is performed, for example, when the workpiece WK has a button and the robot 600 is used to perform an inspection of pressing the button of the workpiece WK. Alternatively, the force setting is also performed when a grinding component is set on the fingertip 630 and the workpiece WK is ground.

[0039] Figure 7 1 is a flowchart showing the procedure of the teaching process in one embodiment. In step S10 , the model of the robot 600 is selected using the function of the robot selection unit 312 .

[0040] Figure 8 : is an explanatory diagram showing a selection screen WS1 for the model of the robot. In the present embodiment, the selection screen WS1 is displayed as augmented reality based on the image data supplied from the robot selection unit 312 to the AR glasses 400. In the selection screen WS1, multiple models of robots are arranged as options. In each model, in addition to the model name, the number of axes, maximum reach, and carryable weight are also shown as attributes of the robot. However, some or all of the attributes of the robot may be omitted. In addition, an image of each model may also be displayed. The operator can select a model using the button 134 provided on the teaching fingertip portion 130. That is, one of the multiple models can be selected by pressing the A button 134a, and confirmed by pressing the B button 134b. In Figure 8 In the state of , the robot with the model name "C1" is selected. The operator's selection result is received by the robot selection unit 312. It should be noted that the operator can also select the model by directly specifying the model name instead of using the selection screen WS1.

[0041] exist Figure 7In step S20, the position and posture recognition unit 314 uses the camera unit 200 to photograph the teaching base unit 110 and the teaching fingertip unit 130, and generates a base unit image and a fingertip unit image. Step S20 is started by the operator pressing one of the buttons 134 of the teaching fingertip unit 130, such as the A button 134a. Preferably, the camera unit 200 has a sufficiently wide viewing angle so that the teaching base unit 110 and the teaching fingertip unit 130 can be photographed at the same time. It should be noted that the base unit image and the fingertip unit image can also be the same image. Alternatively, the image portion including the teaching base unit 110 can be extracted from the image captured by the camera unit 200 as the base unit image, and the other image portion including the teaching fingertip unit 130 can be extracted as the fingertip unit image. In addition, the base unit image and the fingertip unit image can be photographed separately. The base unit image and the fingertip unit image are temporarily stored in the memory 320. Furthermore, the image captured by the image capture unit 200 may be a moving image. When it is desired to set a route instead of registering a teaching point, the moving teaching fingertip 130 is captured as a moving image in step S20.

[0042] In step S30, the position and posture recognition unit 314 obtains the base image obtained in step S20 from the memory 320, and recognizes the position of the base 610 of the robot 600 based on the base image. Figure 2 As shown, the position and posture of the base unit coordinate system Σr1 in the camera coordinate system Σc are recognized based on the image of the first marker 112 provided on the teaching base unit 110. The position and posture of the base unit coordinate system Σr1 can be regarded as the position and posture of the base 610 of the robot 600, that is, the position and posture of the robot coordinate system Σr0.

[0043] In step S40, the position and posture recognition unit 314 obtains the fingertip image obtained in step S20 from the memory 320, and recognizes the position of the fingertip 630 of the robot 600 based on the fingertip image. Figure 2 As shown, the position and posture of the fingertip coordinate system Σt1 in the camera coordinate system Σc are recognized based on the image of the second marker 132 provided on the teaching fingertip 130. The position and posture of the fingertip coordinate system Σt1 can be regarded as the position and posture of the fingertip 630 of the robot 600, that is, the position and posture of the fingertip coordinate system Σt0.

[0044] In step S50, the position and posture recognition unit 314 calculates the robot coordinates of the control point TCP of the fingertip 630 based on the position and posture of the base 610 of the robot 600 and the position and posture of the fingertip 630. The robot coordinates of the control point TCP are represented by the position and posture of the fingertip coordinate system Σt0 in the robot coordinate system Σr0.

[0045] In step S60, the robot selection unit 312 determines whether the robot model needs to be changed. Specifically, if the robot coordinates of the control point TCP calculated in step S50 are within the movable range of the currently selected robot model, it is determined that a model change is not necessary. On the other hand, if the robot coordinates of the control point TCP are outside the movable range, it is determined that a model change is necessary. If a model change is not necessary, the process proceeds to step S80, described below. If a model change is required, the process proceeds to step S70. In step S70, the model of the robot 600 is changed using the function of the robot selection unit 312.

[0046] Figure 9 This is an explanatory diagram showing how to change the robot model using the robot model selection screen WS1. Here, among multiple models, "C1" and "D1" are models where the robot coordinates of the control point TCP are outside the movable range. These models "C1" and "D1" are displayed so that the operator can visually confirm that they are invalid options that cannot be selected. Figure 9 In the example, the models "C1" and "D1" are shaded, and the selection boxes for selection are not selectable. The other models are presented as a list of valid options that can be selected. The operator can change the robot model by selecting one of the valid options. Figure 9 In the example, model "C2" is selected as the changed model. Thus, when the control point TCP of the fingertip portion 630 is outside the movable range of the robot, the control point TCP can be placed within the movable range of the robot by changing the model of the robot. It should be noted that, although it is not necessary to display multiple valid options, it is preferred to display more than one valid option. In addition, the control point TCP that is judged to be outside the movable range may not only be the control point TCP calculated by step S50. For example, the control point TCP stored before step S50, that is, the control point TCP stored in the past teaching process, may also be added. In this case, among the multiple models, a model in which any one of the multiple control point TCPs is outside the movable range may be shaded as an invalid option that cannot be selected.

[0047] Thus, in step S70, the operator is presented with one or more robot models whose control point TCP in the robot coordinate system Σr0 is within the robot's movable range. The model selected by the operator from these one or more models is then used. This makes it easy to change the robot model. Note that the operator can also change the model by directly specifying the model name, rather than using the selection screen WS1.

[0048] In step S80, the joint angle calculation unit 316 calculates the angles of each joint of the robot 600 based on the position and posture of the control point TCP in the robot coordinate system Σr0. This calculation is performed based on inverse kinematics. Generally, multiple combinations of joint angles calculated based on inverse kinematics are often used. In such cases, in step S80, one combination is selected from the multiple combinations based on a predetermined rule.

[0049] In step S90, the joint angle calculation unit 316 determines whether the joint angle needs to be changed. In this embodiment, in the virtual space, when the shape of the robot arm 620 represented by the angle of each joint is in an interference state where there is a possibility of interference with the surrounding objects, it is determined that the joint angle needs to be changed. The outer shape of the surrounding objects is represented by the three-dimensional data contained in the surrounding object attribute data PD stored in the memory 320. The joint angle calculation unit 316 calculates the outer shape of the surrounding objects represented by the three-dimensional data of the surrounding objects and the distance between the shape of the robot arm 620 represented by the joint angle calculated in step S80. When the distance is below a predetermined threshold, it can be determined that it is in an interference state and that the joint angle needs to be changed. As the threshold, for example, a value greater than 0 and less than 10 cm is set. In the virtual space, when the shapes of the surrounding objects and the robot arm 620 are not in an interference state, it is determined that the joint angle does not need to be changed. It should be noted that when the shape of robot arm 620 is in an interference state, potentially causing interference with surrounding objects, joint angle calculation unit 316 may also issue an interference alarm to the operator. If the joint angles do not need to be changed, the process proceeds to step S110, described below. If the joint angles do need to be changed, the process proceeds to step S100. In step S100, the joint angles of robot 600 are changed using the function of joint angle calculation unit 316.

[0050] Figure 10This is an explanatory diagram showing a screen WS2 for changing the robot's joint angles. In this embodiment, this screen WS2 is displayed in an augmented reality format based on image data supplied to the AR glasses 400 by the joint angle calculation unit 316. In this example, two options are displayed for joint angle combinations. In the case of angles A1 and A2, the angle combinations of the three joints J2, J3, and J5 are different, but the position and posture of the fingertip 630 remain the same. In the case of angle A1, there is a possibility that the robot arm 620 will interfere with the surrounding object PB, while in the case of angle A2, there is no possibility of interference. Therefore, in this example, angle A2 is selected as the changed joint angle combination. Thus, the joint angles can be changed by selecting one of multiple options related to joint angle combinations. Such joint angle changes are performed to maintain the position and posture of the fingertip 630. Furthermore, joint angle changes are performed based on instructions from the operator. As a result, in virtual space, when the robot arm 620 is in an interference state with a surrounding object PB, the interference state can be eliminated by changing the joint angles. Note that, instead of the operator selecting one of the options, the joint angle calculation unit 316 may automatically determine a combination of joint angles that eliminates the interference state.

[0051] It should be noted that the determination in step S90 does not need to be performed automatically by the joint angle calculation unit 316. Instead, the determination that a joint angle change is necessary can be made when the operator specifies that a joint angle change is necessary. In other words, the operator can observe the image of the robot arm 620 and the surrounding object PB to determine whether the robot arm 620 and the surrounding object PB are in an interfering state. In this case, the selection of the joint angle combination is also preferably performed by the operator. Thus, the operator can freely select the preferred state of the robot arm 620.

[0052] In step S110, the joint angle calculation unit 316 displays the image of the robot 600 in the virtual space. In this embodiment, the robot 600 is displayed in an augmented reality manner based on the image data supplied from the joint angle calculation unit 316 to the AR glasses 400. For example, as described above Figure 4 、 Figure 6 As shown, in the three-dimensional image of the robot 600 , the image of the base 610 of the robot 600 is displayed superimposed on the teaching base unit 110 , and the image of the fingertip 630 of the robot 600 is displayed superimposed on the teaching fingertip unit 130 .

[0053] In step S120, the operator determines whether the state of robot 600 displayed in step S110 is suitable as a teaching point. If not, the process returns to step S20, where the operator changes the position of teaching fingertip 130 and captures the image again. On the other hand, if it is suitable, the process proceeds to step S130.

[0054] In step S130, the control program creation unit 318 registers the state of the image captured in step S20 as a teaching point. Step S130 is initiated by the operator pressing one of the buttons 134 on the teaching fingertip unit 130, for example, button B 134b. The teaching point is registered in the robot control program RP in the memory 320. Furthermore, if a path is to be set rather than registering teaching points, that is, if a moving image is to be captured, the control program creation unit 318 generates multiple still images in chronological order based on the moving image and records the state of these still images as the teaching points. The control program creation unit 318 generates a path based on the multiple teaching points thus recorded.

[0055] In step S140, the operator determines whether the teaching process is finished. If the teaching process is not finished, the process returns to step S20, and the operator changes the position of the teaching fingertip 130, and the image is captured again. On the other hand, if the teaching process is finished, the process ends. Figure 7 processing.

[0056] It should be pointed out that it is also possible to Figure 7 After the teaching process, other settings for the robot control program RP are set as part of the teaching process. Furthermore, after the teaching process is completed, the robot control program RP created through the teaching process can be used to recreate the movements of the robot 600 in virtual space. For example, when using the robot 600 to transport a workpiece, the shape and weight of the workpiece can be pre-stored in memory 320, and the robot 600 can be recreated in virtual space to transport the workpiece. This allows the operator to determine whether the workpiece transport operation is appropriate without actually moving the robot 600. Furthermore, after the joint angles of the robot 600 are changed in step S100, the image of the robot 600 is displayed in virtual space in step S110. However, the image of the robot 600 can also be displayed in virtual space after step S80, and then the determination of whether to change the joint angles of the robot 600 is made. In this case, the display and change of multiple joint angles can be repeated. In particular, in robots with a high degree of redundant freedom, such as vertical multi-joint robots, the most appropriate joint angle can be selected from multiple joint angles.

[0057] As described above, in the above embodiment, the position and posture of robot 600's base 610 are recognized based on images from teaching base unit 110, and the position and posture of robot 600's fingertips 630 are recognized based on images from teaching fingertip unit 130. These positions and postures are used to calculate the angles of robot 600's joints, and a three-dimensional image of robot 600 represented by the calculated joint angles is displayed in virtual space. Therefore, even without actually setting up the robot and operating it, the operator can easily understand the state of the robot arm and perform robot teaching.

[0058] It should be noted that in the above embodiment, the three-dimensional images of the robot 600 and the surrounding objects PB are displayed as virtual reality using the AR glasses 400. However, the display unit 350, which displays two-dimensional images, can alternatively be used to display the images of the robot 600 and the surrounding objects PB in a virtual space. In this case, the image of the robot 600 is displayed in a virtual space, which is the same as when the AR glasses 400 are used. However, using the AR glasses 400 to display the three-dimensional image of the robot 600 in an augmented reality manner has the advantage of allowing the operator to easily understand the posture of the robot 600.

[0059] Other implementation methods:

[0060] The present disclosure is not limited to the above-mentioned embodiments and can be implemented in various ways without departing from its purpose. For example, the present disclosure can also be implemented in the following aspects. In order to solve part or all of the technical problems of the present disclosure, or to achieve part or all of the effects of the present disclosure, the technical features in the above-mentioned embodiments corresponding to the technical features in the various methods described below can be appropriately replaced or combined. In addition, if the technical feature is not described as a necessary technical feature in this specification, it can be appropriately deleted.

[0061] (1) According to a first aspect of the present disclosure, a method for displaying a robot image is provided. The method includes the following steps: (a) acquiring a base portion image created by photographing a teaching base portion for teaching a robot a position and posture of a base portion, and recognizing the position and posture of the base portion based on the base portion image; (b) acquiring a fingertip portion image created by photographing a teaching fingertip portion for teaching a robot a position and posture of a fingertip portion, and recognizing the position and posture of the fingertip portion based on the fingertip portion image; (c) calculating angles of joints of the robot based on the position and posture of the base portion recognized in step (a) and the position and posture of the fingertip portion recognized in step (b); and (d) displaying a three-dimensional image of the robot in a virtual space, the angles of the joints calculated in step (c).

[0062] According to this display method, without actually installing the robot and operating it, the operator can easily understand the status of the robot arm and can perform robot teaching.

[0063] (2) In the above-mentioned display method, the step (c) may also include the following steps: (i) calculating the position of the control point set at the fingertip portion in the robot coordinate system based on the position and posture of the base identified by the step (a) and the position and posture of the fingertip portion identified by the step (b); and (ii) changing the model of the robot when the position of the control point in the robot coordinate system is outside the movable range of the robot.

[0064] According to this display method, when the control point of the fingertip is outside the movable range of the robot, the control point of the fingertip can be brought within the movable range of the robot by changing the model of the robot.

[0065] (3) In the above-mentioned display method, the step (ii) may also include the following steps: prompting the operator with one or more models as the model of the robot in which the position of the control point in the robot coordinate system is within the movable range of the robot; and adopting the model selected by the operator from the one or more models.

[0066] According to this display method, the model of the robot can be easily changed.

[0067] (4) In the above-mentioned display method, the step (c) may also include: an angle changing step of selecting a combination from the plurality of combinations and changing the combination when there are a plurality of combinations of angles of the joints.

[0068] According to this display method, a preferred state can be selected as the state of the robot arm.

[0069] (5) In the above-mentioned display method, the operator may select the combination of the joint angles.

[0070] According to this display method, the operator can freely select the preferred state of the robot arm.

[0071] (6) In the above-mentioned display method, the angle changing process may also include: an interference elimination process, when the shape of the robotic arm represented by the angle of the joint calculated by the process (c) is in an interference state in which there is a possibility of interference with surrounding objects, the interference state is eliminated by changing the combination of the angles of the joint while maintaining the position and posture of the fingertip.

[0072] According to this display method, the possibility of interference between the robot arm and surrounding objects can be reduced.

[0073] (7) In the above-mentioned display method, the interference elimination process may also include the following process: calculating the distance between the surrounding object represented by the three-dimensional data of the surrounding object and the shape of the robotic arm represented by the angle of the joint calculated by the process (c); and when the distance is below a predetermined threshold value, determining that it is in the interference state, while maintaining the position and posture of the fingertip, automatically determining the combination of the angles of the joint that eliminates the interference state.

[0074] According to this display method, interference between the robot arm and surrounding objects can be automatically eliminated.

[0075] (8) The display method may further include displaying the three-dimensional image in the step (d) by overlapping the teaching base unit and the teaching fingertip unit in an augmented reality manner.

[0076] According to this display method, the robot arm is displayed in an augmented reality manner, so the operator can easily understand the status of the robot arm.

[0077] (9) In the above-mentioned display method, a first mark for identifying the position and posture of the base of the robot may be provided on the teaching base portion, and a second mark for identifying the position and posture of the fingertip portion of the robot may be provided on the teaching fingertip portion.

[0078] According to this display method, the position and posture of the base and the position and posture of the fingertip can be easily recognized using the first mark and the second mark.

[0079] (10) In the above display method, the first mark and the second mark may respectively include a two-dimensional code, a convex portion, a concave portion, a luminous portion, or a printed pattern.

[0080] According to this display method, the position and posture of the base and the position and posture of the fingertip can be easily recognized based on the first mark and the second mark.

[0081] (11) According to a second aspect of the present disclosure, a computer program for performing a display process of a robot image is provided. The computer program causes a processor to execute the following processes: (a) acquiring a base portion image created by photographing a teaching base portion for teaching the position and posture of a base of a robot, and recognizing the position and posture of the base based on the base portion image; (b) acquiring a fingertip portion image created by photographing a teaching fingertip portion for teaching the position and posture of a fingertip portion of the robot, and recognizing the position and posture of the fingertip portion based on the fingertip portion image; (c) calculating the angles of the joints of the robot based on the position and posture of the base portion recognized by the process (a) and the position and posture of the fingertip portion recognized by the process (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated by the process (c) in a virtual space.

[0082] According to this computer program, without actually installing the robot and operating it, the operator can easily understand the status of the robot arm and perform robot teaching.

[0083] (12) According to a third aspect of the present disclosure, a system for displaying robot images is provided. The teaching system includes: a teaching base unit for teaching the position and posture of a robot base; a teaching fingertip unit for teaching the position and posture of the robot's fingertips; a camera unit for photographing the teaching base unit and the teaching fingertip unit; and a control unit connected to the camera unit. The control unit performs the following processing: (a) obtaining a base unit image created by photographing the teaching base unit using the photographic unit, and identifying the position and posture of the base unit based on the base unit image; (b) obtaining a fingertip unit image created by photographing the teaching fingertip unit using the photographic unit, and identifying the position and posture of the fingertip unit based on the fingertip unit image; (c) calculating the angles of the joints of the robot based on the position and posture of the base unit identified by the processing (a) and the position and posture of the fingertip unit identified by the processing (b); and (d) displaying a three-dimensional image of the robot represented by the angles of the joints calculated by the processing (c) in a virtual space.

[0084] According to this teaching system, without actually installing the robot and operating it, the operator can easily understand the status of the robot arm and perform teaching of the robot.

[0085] The present disclosure may also be implemented in various ways other than the above-described ways. For example, it may be implemented by a robot system including a robot and a control device, a computer program for implementing the functions of the robot control device, and a non-transitory storage medium recording the computer program.

Claims

1. A display method, characterized in that: The process includes the following steps: (a) acquiring a teaching base portion image created by photographing a teaching base portion for teaching a position and posture of a base of a robot, and recognizing the position and posture of the base portion based on the base portion image; (b) acquiring a fingertip image created by photographing a teaching fingertip for teaching the position and posture of the fingertip of the robot, and recognizing the position and posture of the fingertip based on the fingertip image; (c) calculating angles of one or more joints of the robot based on the position and posture of the base recognized in step (a) and the position and posture of the fingertip recognized in step (b); as well as (d) displaying a three-dimensional image of the robot in a virtual space in which the joint is at the angle calculated in step (c), The step (c) includes the following steps: (i) calculating the position of a control point set at the fingertip in a robot coordinate system based on the position and posture of the base recognized in step (a) and the position and posture of the fingertip recognized in step (b); and (ii) when the position of the control point in the robot coordinate system is outside the movable range of the robot, changing the model of the robot; The step (ii) includes the following steps: presenting, as the model of the robot, one or more models in which the position of the control point in the robot coordinate system is within the movable range of the robot to the operator; and The model selected by the operator from the one or more models is adopted.

2. The display method according to claim 1, wherein: Further, the step (c) comprises: The angle changing step is to select and change a combination of angles of the joints when a plurality of combinations are possible.

3. The display method according to claim 2, wherein: The operator selects the combination of the angles of the joints.

4. The display method according to claim 2, wherein: The angle changing process includes: The interference elimination step includes, when the shape of the robot arm represented by the angles of the joints calculated in the step (c) is in an interference state that may interfere with a surrounding object, eliminating the interference state by changing a combination of the angles of the joints while maintaining the position and posture of the fingertips.

5. The display method according to claim 4, wherein: The interference elimination process includes the following steps: calculating a distance between the surrounding object represented by the three-dimensional data of the surrounding object and the shape of the robot arm whose joints have the angles calculated in the step (c); and When the distance is equal to or smaller than a predetermined threshold, it is determined that the interference state exists, and a combination of the angles of the joints that eliminates the interference state is automatically determined while maintaining the position and posture of the fingertip.

6. The display method according to claim 1, wherein: Furthermore, in the step (d), the three-dimensional image is displayed superimposed on the teaching base unit and the teaching fingertip unit in an augmented reality manner.

7. The display method according to claim 1, wherein: A first mark for identifying the position and posture of the base of the robot is provided on the teaching base portion. A second mark for recognizing the position and posture of the fingertip of the robot is provided on the teaching fingertip.

8. The display method according to claim 7, wherein: The first mark and the second mark respectively include a QR code, a convex portion, a concave portion, a luminous portion or a printed pattern.

9. A non-transitory recording medium having a computer program recorded thereon, characterized in that: The computer program causes the processor to perform the following processing: (a) acquiring a teaching base portion image created by photographing a teaching base portion for teaching a position and posture of a base of a robot, and recognizing the position and posture of the base portion based on the base portion image; (b) acquiring a fingertip image created by photographing a teaching fingertip for teaching the position and posture of the fingertip of the robot, and recognizing the position and posture of the fingertip based on the fingertip image; (c) calculating angles of one or more joints of the robot based on the position and posture of the base recognized in the process (a) and the position and posture of the fingertip recognized in the process (b); as well as (d) displaying a three-dimensional image of the robot in a virtual space in which the joint is at the angle calculated by the process (c), The process (c) includes the following processes: (i) calculating the position of the control point set at the fingertip in the robot coordinate system based on the position and posture of the base recognized by the process (a) and the position and posture of the fingertip recognized by the process (b); and (ii) when the position of the control point in the robot coordinate system is outside the movable range of the robot, changing the model of the robot; The treatment (ii) includes the following treatments: presenting, as the model of the robot, one or more models in which the position of the control point in the robot coordinate system is within the movable range of the robot to the operator; and The model selected by the operator from the one or more models is adopted.

10. A display system, characterized in that: have: The teaching base unit is used to teach the position and posture of the robot base; a teaching fingertip portion, used to teach the position and posture of the fingertip portion of the robot; a photographing unit configured to photograph the teaching base unit and the teaching fingertip unit; and A control unit connected to the photographing unit, The control unit performs the following processing: (a) acquiring a base portion image created by photographing the teaching base portion using the imaging unit, and recognizing the position and posture of the base portion based on the base portion image; (b) acquiring a fingertip image created by photographing the teaching fingertip using the photographing unit, and recognizing the position and posture of the fingertip based on the fingertip image; (c) calculating angles of one or more joints of the robot based on the position and posture of the base recognized in the process (a) and the position and posture of the fingertip recognized in the process (b); as well as (d) displaying a three-dimensional image of the robot in a virtual space in which the joint is at the angle calculated by the process (c), The process (c) includes the following processes: (i) calculating the position of the control point set at the fingertip in the robot coordinate system based on the position and posture of the base recognized by the process (a) and the position and posture of the fingertip recognized by the process (b); and (ii) when the position of the control point in the robot coordinate system is outside the movable range of the robot, changing the model of the robot; The treatment (ii) includes the following treatments: presenting, as the model of the robot, one or more models in which the position of the control point in the robot coordinate system is within the movable range of the robot to the operator; and The model selected by the operator from the one or more models is adopted.

Images