Terminal device

Abstract

The invention provides a terminal device which can create a work program without reducing the operation rate of a robot. A terminal device (1) captures a first image including a work object and a work object mark, sets a user coordinate system on the basis of the work object mark included in the first image, and sets an operation path of an industrial robot in the user coordinate system on the basis of a work site of the work object included in the first image. A work program for operating the industrial robot is created on the basis of the set operation path of the industrial robot, and a second image including a work object, a work object mark, and a robot mark attached to the industrial robot, the work object and the work object mark maintaining the positional relationship in the first image is captured. A robot coordinate system is estimated on the basis of the robot mark included in the second image and the position / attitude information of the industrial robot, and the operation path of the industrial robot is converted from the user coordinate system to the robot coordinate system to correct the work program.

Description

Technical Field

[0001] The present invention relates to a terminal device capable of creating an operating program for a robot. Background Art

[0002] In the past, there is a technology that uses images captured by a camera when creating a program for operating a robot. For example, in the technology of Patent Document 1, an image of a robot and a welding object including a mounting mark captured by a camera is used to create a program for operating a robot. In the technology of Patent Document 2, a virtual image of a robot's manipulator and the like is superimposed on an image of a real space including the robot captured by a camera to create a program for operating a robot.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: JP Patent No. 7190552

[0006] Patent Document 2: JP Patent No. 6420229

[0007] However, in the techniques of Patent Documents 1 and 2, it is necessary to capture an image including the robot when creating a program. Therefore, the robot is occupied during the program creation, and the operation rate of the robot decreases. Summary of the invention

[0008] Therefore, an object of the present disclosure is to provide a terminal device that can create a work program without reducing the operating rate of a robot.

[0009] A terminal device involved in one method of the present disclosure comprises: a first shooting unit, which shoots a first image including a work object and a mark for the work object; a user coordinate system setting unit, which sets the user coordinate system based on the mark for the work object contained in the first image; a motion path setting unit, which sets the motion path of an industrial robot in the user coordinate system based on the work part of the work object contained in the first image; a program creation unit, which creates a work program for moving the industrial robot based on the set motion path of the industrial robot; a second shooting unit, which shoots a second image including, in addition to the work object and the mark for the work object that maintain the positional relationship in the first image, a robot mark installed on the industrial robot; a robot coordinate system estimation unit, which estimates the robot coordinate system based on the robot mark contained in the second image and the position and posture information of the industrial robot; and a program correction unit, which transforms the coordinate system of the motion path of the industrial robot from the user coordinate system to the robot coordinate system to correct the work program.

[0010] According to this method, by shooting a first image including a work object and a work object mark to set a user coordinate system based on a work object mark, a work program of the industrial robot can be created based on a motion path of the industrial robot in the user coordinate system, wherein the motion path is set based on a work part of the work object. Then, by shooting a second image including the work object and the work object mark and a robot mark installed on the industrial robot that maintain the positional relationship in the first image, a robot coordinate system based on the robot mark and the position and posture information of the industrial robot is estimated, and the coordinate system of the motion path of the industrial robot can be converted from the user coordinate system to the robot coordinate system to correct the work program.

[0011] Thus, an operating program for the industrial robot in the user coordinate system is created based on the first image that does not include the industrial robot, and the coordinate system representing the motion path of the industrial robot is transformed from the user coordinate system to the robot coordinate system based on the second image taken later and in which a robot marker installed on the industrial robot is added. The operating program based on the user coordinate system can be corrected into an operating program based on the robot coordinate system.

[0012] In the above aspect, the program correction unit may transform the coordinate system of the motion path of the industrial robot based on a difference between the origin of the user coordinate system and the origin of the robot coordinate system.

[0013] According to this aspect, the coordinate system of the motion path of the industrial robot can be transformed based on the difference between the origin of the user coordinate system and the origin of the robot coordinate system.

[0014] In the above aspect, the user coordinate system setting unit may calculate the origin and coordinate axes of the user coordinate system based on the work object mark.

[0015] According to this aspect, a work program for the industrial robot can be created based on the origin and coordinate axes of the user coordinate system estimated based on the work object marker.

[0016] In the above aspect, the robot coordinate system estimating unit may estimate the origin and coordinate axes of the robot coordinate system based on the robot marker and the position and posture information of the industrial robot.

[0017] According to this aspect, the work program of the industrial robot can be corrected using the origin of the robot coordinate system estimated based on the robot marker and the coordinate axes of the robot coordinate system estimated based on the position and posture information of the industrial robot.

[0018] In the above aspect, the position and posture information of the industrial robot may include angle information of each axis of the industrial robot acquired from a robot control device that controls the operation of the industrial robot.

[0019] According to this aspect, by using the angle information of each axis of the industrial robot acquired from the robot control device, it is possible to improve the accuracy of estimating the coordinate axes of the robot coordinate system.

[0020] Effects of the Invention

[0021] According to the present disclosure, it is possible to provide a terminal device that can create a work program without reducing the operating rate of a robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a diagram illustrating a system configuration of a welding robot system including a terminal device according to an embodiment.

[0023] Figure 2 is an example Figure 1 A diagram showing the physical structure of the terminal device and the robot control device.

[0024] Figure 3 It is a schematic diagram for explaining the first image captured by the imaging unit.

[0025] Figure 4 It is a schematic diagram for explaining the second image captured by the imaging unit.

[0026] Figure 5 is an example Figure 2 A diagram of the functional structure of the control unit of the terminal device shown.

[0027] Figure 6 This is a flowchart for explaining an example of the operation of the terminal device.

[0028] Explanation of symbols

[0029] 1…terminal device, 2…robot control device, 3…manipulator, 11…control unit, 12…storage unit, 13…communication unit, 14…shooting unit, 15…distance measuring unit, 16…display unit, 21…control unit, 22…storage unit, 23…communication unit, 100…welding robot system, 111…user coordinate system setting unit, 112…coordinate assigning unit, 113…motion path setting unit, 114…program creation unit, 115…robot coordinate system estimation unit, 116…program correction unit, Mr…robot mark, Mw…workpiece mark, W…workpiece. DETAILED DESCRIPTION

[0030] Suitable embodiments of the present disclosure are described with reference to the accompanying drawings. In addition, in each figure, the components marked with the same symbols have the same or similar structures. In addition, since the drawings are schematic, the dimensions and ratios of the various structural elements are different from the actual ones.

[0031] Figure 11 is a diagram illustrating a system configuration of a welding robot system 100 including a terminal device 1 according to an embodiment. The welding robot system 100 includes, for example, a terminal device 1, a robot control device 2, and a manipulator (industrial robot) 3. The terminal device 1 and the robot control device 2, and the robot control device 2 and the manipulator 3 are connected via a network. The terminal device 1 and the robot control device 2 are connected via wireless communication such as WiFi (Wireless Fidelity), and the robot control device 2 and the manipulator 3 are connected via a communication cable. In addition, the network may be either wired (including a communication cable) or wireless.

[0032] Furthermore, a teaching pendant may be included in the welding robot system 100. The teaching pendant can be connected to the robot control device 2 and is an operating device used when an operator teaches the operation of the robot hand 3.

[0033] The manipulator 3 is a welding robot that performs arc welding on a workpiece W that is a welding target according to the operation conditions set in the robot control device 2. The manipulator 3 includes, for example, a multi-joint arm provided on a base member fixed to the floor of a factory, and a welding torch (tool) connected to the front end of the multi-joint arm.

[0034] A robot mark Mr is attached to the front end portion of the multi-joint arm of the robot 3 , and a workpiece mark Mw is arranged on the workpiece W.

[0035] The robot mark Mr and the workpiece mark Mw may be any identifier that can be recognized by the terminal device 1, and preferably an AR mark is used. By using the AR mark, when the AR mark is recognized, the user coordinate system with the AR mark as the origin can be easily displayed superimposed on the actual image.

[0036] refer to Figure 2 , the structures of the robot control device 2 and the terminal device 1 are described in sequence below.

[0037] The robot control device 2 is a control unit that controls the operation of the manipulator 3 , and includes, for example, a control unit 21 , a storage unit 22 , and a communication unit 23 .

[0038] The control unit 21 is a processor, and controls the robot 3 by executing a welding program or the like stored in the storage unit 22 .

[0039] The communication unit 23 is a communication interface, and controls communication with the terminal device 1 and the robot 3 connected via a network.

[0040] In addition, the robot control device 2 may further include a welding power supply unit. The welding power supply unit supplies welding current and welding voltage to the robot hand 3 according to predetermined welding conditions, for example, in order to generate an arc between the tip of the welding wire and the workpiece W. The welding power supply unit may be provided separately from the robot control device 2.

[0041] The terminal device 1 is, for example, a portable terminal with a digital camera. Portable terminals include, for example, tablet terminals, smart phones, portable information terminals (PDAs), notebook PCs (personal computers), and other portable terminals. The terminal device 1 includes, for example, a control unit 11, a storage unit 12, a communication unit 13, a photographing unit 14, a distance measuring unit 15, and a display unit 16.

[0042] The control unit 11 is a processor, and controls each unit of the terminal device 1 by executing a program stored in the storage unit 12 .

[0043] The storage unit 12 is a computer-readable recording medium, and stores programs for realizing various functions of the terminal device 1 and various data used in the programs.

[0044] The communication unit 13 is a communication interface, and controls communication with the robot control device 2 connected via a network.

[0045] The imaging unit (first imaging unit, second imaging unit) 14 is, for example, a 2D camera including a lens and an image sensor (imaging element), and converts light of a subject received by the lens into an electrical signal (digital image data).

[0046] Here, in the present embodiment, the imaging unit 14 captures two types of images, namely, a first image and a second image. Figure 3 as well as Figure 4 The first image and the second image will be described.

[0047] like Figure 3 In this way, the first image captured by the imaging unit 14 includes the workpiece W, which is the welding target of the robot 3, and the workpiece marker (welding target marker) Mw arranged on the workpiece W. The robot 3 does not need to be included in the first image. Therefore, when the first image is captured, the robot 3 can perform welding work at a work place (e.g., a robot cell) located at a place different from the place where the workpiece W exists, for example.

[0048] like Figure 4As shown in FIG. 1 , the second image captured by the imaging unit 14 includes the workpiece W and the workpiece mark Mw, which maintain the positional relationship in the first image, and the robot mark Mr attached to the manipulator 3. When capturing the second image, the workpiece W and the workpiece mark Mw included in the first image are preferably moved to the work site of the manipulator 3. Alternatively, the manipulator 3 with a carriage may be moved to the work site where the workpiece W and the workpiece mark Mw exist.

[0049] Figure 2 The distance measuring unit 15 is, for example, a 3D camera equipped with a distance measuring sensor. The distance measuring sensor is a sensor that can measure the distance to an object. As the distance measuring sensor, for example, a LiDAR (Light Detection and Ranging) sensor, a millimeter wave sensor, an ultrasonic sensor, etc. can be used.

[0050] In addition, the distance measuring unit 15 may be only one of a 3D camera and a distance measuring sensor. In the case of only a 3D camera, it is preferred to calculate the three-dimensional coordinate data corresponding to the object based on a plurality of images obtained by photographing the object from a plurality of different positions. In this case, a three-dimensional measurement technique based on a known stereo method can be used.

[0051] Here, by including a distance measuring sensor in the terminal device 1, the positional relationship between the image sensor and the distance measuring sensor can be fixed, and the timing of acquiring data in each sensor can be made consistent. This can improve the processing efficiency of setting a specific position of a workpiece mark Mw described later on the point group data.

[0052] The display unit 16 is, for example, a display having a touch panel, and displays the image of the subject captured by the image capturing unit 14 and receives inputs such as operation instructions from an operator. The display unit 16 may be provided separately from the terminal device 1 as a display device having a touch panel.

[0053] Figure 5 This is a diagram illustrating a functional configuration of the control unit 11 of the terminal device 1 according to the embodiment. The control unit 11 of the terminal device 1 includes, for example, a user coordinate system setting unit 111, a coordinate assigning unit 112, an action path setting unit 113, a program creating unit 114, a robot coordinate system estimating unit 115, and a program correcting unit 116 as a functional configuration.

[0054] The user coordinate system setting unit 111 sets a three-dimensional user coordinate system based on the workpiece mark Mw included in the first image captured by the imaging unit 14. Figure 3 As shown, the first image includes the workpiece W and the workpiece mark Mw.

[0055] Specifically, the user coordinate system setting unit 111 estimates three-dimensional coordinate axes (orientations) with a specific position of the workpiece mark Mw (for example, a corner of the mark, the center of the mark, etc.) as the origin, and sets the user coordinate system.

[0056] In addition, the origin of the user coordinate system is not limited to the specific position of the workpiece mark Mw, and for example, it may be the origin of a temporary robot coordinate system determined based on the specific position of the workpiece mark Mw. For example, a virtual model of a robot can be placed next to the workpiece W, and the temporary robot coordinate system can be set as the coordinate system of the virtual model. By fixing the position of the workpiece mark Mw relative to the virtual model of the robot, the specific position of the workpiece mark Mw can be determined on the temporary robot coordinate system. The user coordinate system in this case can be treated as a temporary robot coordinate system.

[0057] For example, the operator can arbitrarily specify the position where the virtual model of the robot is to be placed by touching the image of the work space according to guidance such as voice or text messages.

[0058] Figure 5 The coordinate assigning unit 112 assigns coordinates of the user coordinate system to the point cloud data obtained by the distance measuring unit 15 that measures the distance to the object included in the image captured by the imaging unit 14. This will be described in detail below.

[0059] The coordinate assigning unit 112 detects the specific position of the workpiece mark Mw based on the first image, and sets the detected specific position of the workpiece mark Mw on the point group data obtained by the distance measuring unit 15. The coordinate assigning unit 112 assigns the point group data to the coordinates of the user coordinate system having the specific position of the workpiece mark Mw set on the point group data as the origin (reference).

[0060] The specific position of the workpiece mark Mw set on the point group data can be automatically identified by data analysis, or the specific position of the workpiece mark Mw on the point group data can be specified by the operator pointing it out.

[0061] The motion path setting unit 113 sets the motion path of the robot 3 in the user coordinate system based on the welding location of the workpiece W included in the first image.

[0062] For example, the motion path setting unit 113 sets the motion path of the virtual model of the robot in the first image so that the welding torch connected to the front end of the virtual model of the robot moves along the welding part of the workpiece W. The motion path can be set by manual selection by a user, or can be automatically set by calculating the distance between the welding part of the workpiece W and the welding torch.

[0063] Alternatively, candidates for welding locations may be detected based on the point group data, and the user may select a welding location of the workpiece W from the detected candidates for welding locations, or candidates for welding locations may be detected by performing image processing on the first image, and the user may select a welding location of the workpiece W from the detected candidates for welding locations. When selecting a welding location, the user may select or specify a welding order, a welding direction, etc. of the welding location.

[0064] The program creation unit 114 creates a welding program for welding a welding portion of the workpiece W included in the first image based on the motion path of the robot 3 in the user coordinate system set by the motion path setting unit 113 .

[0065] The robot coordinate system estimation unit 115 estimates the robot coordinate system based on the robot marker Mr and the position and posture information of the manipulator 3 included in the second image captured by the imaging unit 14. Figure 4 As shown, the second image includes the workpiece W and the workpiece mark Mw, which maintain the positional relationship in the first image, and the robot mark Mr attached to the manipulator 3 .

[0066] Specifically, the robot coordinate system estimation unit 115 estimates the origin of the robot coordinate system based on the robot mark Mr and estimates the coordinate axes (orientation) of the robot coordinate system based on the position and posture information of the manipulator 3 , thereby estimating the robot coordinate system.

[0067] Here, the robot controller 2 holds the coordinates of the robot coordinate system corresponding to the attachment position of the robot marker Mr and the position and posture information of the manipulator 3 controlled by the robot controller 2. The position and posture information of the manipulator 3 includes, for example, angle information of each axis of the manipulator 3.

[0068] Since the content of the position and posture information of the manipulator 3 may also change depending on the type of the manipulator 3, when the position and posture information of the manipulator 3 is obtained from the robot control device 2, the type of the manipulator 3 may also be obtained. In this case, the robot coordinate system estimation unit 115 preferably estimates the coordinate axis (orientation) of the robot coordinate system based on the position and posture information of the manipulator 3 and the type of the manipulator 3.

[0069] Figure 5 The program correction unit 116 converts the coordinate system of the motion path of the manipulator 3 from the user coordinate system to the robot coordinate system to correct the welding program.

[0070] By way of example, the program correction unit 116 calculates the difference between the origin of the user coordinate system and the origin of the robot coordinate system, and moves the coordinates of the motion path of the manipulator 3 in the user coordinate system based on the calculated difference.

[0071] Thereby, the coordinates of the motion path can be converted from the user coordinate system to the robot coordinate system. The program correction unit 116 corrects the welding program created by the program creation unit 114 so that the program is expressed by the coordinates of the robot coordinate system after the coordinate conversion.

[0072] The program correction unit 116 sends the corrected welding program to the robot controller 2 and stores it in the storage unit 22 of the robot controller 2. Thus, the control unit 21 of the robot controller 2 can control the robot 3 according to the corrected welding program so that the robot 3 welds the welding part of the workpiece W included in the second image.

[0073] refer to Figure 6 An example of the operation of the terminal device 1 will be described.

[0074] Initially, the imaging unit 14 of the terminal device 1 captures a first image including a workpiece W that is a welding target of the robot 3 and a workpiece mark Mw disposed on the workpiece W (step S101 ).

[0075] Next, the user coordinate system setting unit 111 of the terminal device 1 sets a three-dimensional user coordinate system based on the workpiece mark Mw included in the first image captured in the above-mentioned step S101 (step S102 ).

[0076] Next, the motion path setting unit 113 of the terminal device 1 sets the motion path of the robot 3 in the user coordinate system based on the welding location of the workpiece W included in the first image (step S103 ).

[0077] Next, the program creation unit 114 of the terminal device 1 creates a welding program for welding the welding portion of the workpiece W included in the first image based on the motion path of the robot 3 in the user coordinate system set in step S103 (step S104 ).

[0078] Next, the imaging unit 14 of the terminal device 1 captures a second image including the workpiece W and the workpiece mark Mw, and the robot mark Mr attached to the manipulator 3 , maintaining the positional relationship in the first image (step S105 ).

[0079] Next, the robot coordinate system estimation unit 115 of the terminal device 1 estimates the robot coordinate system based on the robot marker Mr and the position and posture information of the manipulator 3 included in the second image captured in the above-mentioned step S105 (step S106 ).

[0080] Next, the program correction unit 116 of the terminal device 1 converts the coordinate system of the motion path of the robot 3 set in step S103 from the user coordinate system to the robot coordinate system, and corrects the welding program created in step S104 (step S107). Then, this operation ends.

[0081] As described above, according to the terminal device 1 involved in the embodiment, by capturing the first image including the workpiece W and the workpiece mark Mw, a user coordinate system based on the workpiece mark Mw is set, and a welding program of the robot 3 can be prepared based on the motion path of the robot 3 in the user coordinate system, wherein the motion path is set based on the welding part of the workpiece W. Then, by capturing the second image including the workpiece W and the workpiece mark Mw and the robot mark Mr attached to the robot 3, which maintain the positional relationship in the first image, a robot coordinate system based on the robot mark Mr and the position and posture information of the robot 3 is estimated, and the coordinate system of the motion path of the robot 3 can be converted from the user coordinate system to the robot coordinate system to correct the welding program.

[0082] Thus, a welding program for the manipulator 3 in the user coordinate system is created based on the first image that does not include the manipulator 3, and the coordinate system representing the movement path of the manipulator 3 is transformed from the user coordinate system to the robot coordinate system based on the second image taken later and in which the robot marker Mr installed on the manipulator 3 is added. The welding program based on the user coordinate system can be corrected into a welding program based on the robot coordinate system.

[0083] Therefore, according to the terminal device 1 according to the embodiment, since the occupancy rate of the robot 3 when creating a welding program can be reduced, the welding program can be created without reducing the operation rate of the robot 3.

[0084] In addition, the present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various other forms within the scope not departing from the gist of the present disclosure. Therefore, the above-mentioned embodiments are merely illustrative and should not be interpreted in a limiting sense.

[0085] For example, in the above-mentioned embodiments, the welding robot is used for description, but the present invention is not limited thereto. For example, the present disclosure can be applied to an industrial robot that also includes an operating robot for picking up, etc. In this case, the welding program, welding object, welding part, and welding action used in the above-mentioned embodiments can be replaced with an operation program, an operation object, an operation part, and an operation, respectively.

Claims

1. A terminal device comprising: A first imaging unit that captures a first image including a work object and a mark for the work object; A user coordinate system setting unit that sets a user coordinate system based on the work object mark included in the first image; a motion path setting unit that sets a motion path of the industrial robot in the user coordinate system based on a working portion of the working object included in the first image; a program creation unit that creates a work program for causing the industrial robot to move based on the set motion path of the industrial robot; a second imaging unit that captures a second image including, in addition to the work object and the work object marker maintaining the positional relationship in the first image, a robot marker mounted on the industrial robot; a robot coordinate system estimating unit that estimates a robot coordinate system based on the robot marker and position and posture information of the industrial robot included in the second image; The program correction unit converts the coordinate system of the motion path of the industrial robot from the user coordinate system to the robot coordinate system to correct the operation program.

2. The terminal device according to claim 1, wherein: The program correction unit transforms a coordinate system of a motion path of the industrial robot based on a difference between an origin of the user coordinate system and an origin of the robot coordinate system.

3. The terminal device according to claim 1, wherein: The user coordinate system setting unit calculates the origin and coordinate axes of the user coordinate system based on the work object mark.

4. The terminal device according to claim 1, wherein: The robot coordinate system estimation unit estimates the origin and coordinate axes of the robot coordinate system based on the robot marker and position and posture information of the industrial robot.

5. The terminal device according to claim 1, wherein: The position and posture information of the industrial robot includes angle information of each axis of the industrial robot acquired from a robot control device that controls the operation of the industrial robot.

Images