Terminal device

Abstract

A terminal device capable of increasing the precision of specifying an interference area is provided. The terminal device comprises a coordinate system setting unit (111) for setting a user coordinate system based on a marker included in an image including an industrial robot and the workspace of the industrial robot captured by a shooting unit; a coordinate assignment unit (112) for assigning coordinates of the user coordinate system to point cloud data obtained by a distance measuring unit that measures the distance to an object included in the image; an area specification unit (113) for specifying a robot area on the user coordinate system corresponding to the industrial robot based on shape size information of the industrial robot corresponding to the type of industrial robot and attitude information of the industrial robot; and an avoidance point cloud generation unit (114) for generating point cloud data for interference avoidance by removing point cloud data included in the robot area from the point cloud data obtained by the distance measuring unit.

Description

Technology Field

[0001] The present invention relates to a terminal device. Background Technology

[0002] Conventionally, when performing work using a robot, in order to avoid interference between the robot and surrounding obstacles, an interference area where obstacles exist is set around the robot, and the robot is controlled so that it does not enter the interference area. As a method for setting an interference area around a robot, for example, Patent Document 1 discloses a method of setting an interference area by making the robot and other objects different colors, as shown below. In this method, a 3D map is created based on images from a 3D camera that captures the robot and the surroundings of the robot, the rows of points corresponding to the robot's color are removed from each row of points in the 3D map, and the interference area is set using the position data of the rows of points remaining in the 3D map. Prior art literature

[0003] Patent Document 1: Japanese Patent Publication No. 5980873 The problem to be solved

[0004] However, the method of Patent Document 1 imposes a constraint that the robot and the workpiece must be of different colors, and there is a risk that the color of the robot and other colors may become difficult to distinguish due to, for example, the attachment of contamination to the robot or workpiece, or changes in the color or brightness of the lighting, and thus the interference area may not be correctly identified.

[0005] Accordingly, the present disclosure aims to provide a terminal device capable of increasing the precision of specifying an interference region. means of solving the problem

[0006] A terminal device according to one aspect of the present disclosure comprises: a coordinate system setting unit that sets a user coordinate system based on a marker included in an image including an industrial robot and a workspace of said industrial robot captured by a shooting unit; a coordinate assignment unit that assigns coordinates of the user coordinate system to point cloud data obtained by a distance measuring unit that measures the distance to an object included in the image; an area specification unit that specifies a robot area on the user coordinate system corresponding to an industrial robot based on shape size information of an industrial robot corresponding to a type of industrial robot and attitude information of an industrial robot; and an avoidance point cloud generation unit that generates point cloud data for interference avoidance by removing point cloud data included in the robot area from the point cloud data obtained by the distance measuring unit.

[0007] According to this sun, a user coordinate system is established based on markers included in an image capturing an industrial robot and its workspace, coordinates of the user coordinate system are assigned to point cloud data corresponding to objects included in the image, a robot region on the user coordinate system corresponding to the industrial robot is specified, and point cloud data for interference avoidance is created by removing the point cloud data included in the robot region from the point cloud data corresponding to objects included in the image. By doing so, it becomes possible to specify as an interference region the region formed based on the point cloud data obtained by removing the point cloud data corresponding to the industrial robot from the point cloud data corresponding to the captured image.

[0008] In the above-mentioned sun, a program writing unit may be further provided to write a work program so that an industrial robot performs a work while avoiding interference with point cloud data for interference avoidance.

[0009] According to this sun, it becomes possible to have industrial robots perform tasks while avoiding interference with objects other than industrial robots.

[0010] In the above-mentioned sun, when specifying the robot area, the area specifying part may further utilize shape and size information of a tool corresponding to a type of tool mountable on an industrial robot to specify the robot area in the user coordinate system corresponding to the industrial robot equipped with the tool.

[0011] According to this sun, since the area of ​​a tool mounted on an industrial robot can be included in the robot area, it becomes possible to create point cloud data for interference avoidance based on a different robot area for each mounted tool.

[0012] In the above example, the type of industrial robot and the attitude information of the industrial robot may be obtained from a control device that controls the industrial robot.

[0013] According to this sun, since the robot area can be specified based on information acquired from a control device that controls the posture of an industrial robot based on the type of industrial robot, it becomes possible to increase the precision of specifying the robot area.

[0014] In the above-mentioned sun, the attitude information of the industrial robot may be angle information of each axis of the industrial robot.

[0015] According to this sun, since the robot area can be specified based on the angle of each axis of the industrial robot, it becomes possible to specify a robot area that corresponds to the angle of each axis changing in accordance with the robot's movement.

[0016] In the above-mentioned sun, the robot region may be a region larger than the size of the industrial robot determined based on the shape size information and attitude information of the industrial robot.

[0017] According to this sun, since point cloud data contained within an area larger than the size of an industrial robot can be removed, it becomes possible to increase the accuracy of removing point cloud data corresponding to the industrial robot. Effects of the invention

[0018] According to the present disclosure, a terminal device capable of increasing the precision of specifying an interference region can be provided. Brief explanation of the drawing

[0019] FIG. 1 is a drawing illustrating the system configuration of a welding robot system including a terminal device according to an embodiment. FIG. 2 is a drawing illustrating the physical configuration of the terminal device and robot control device shown in FIG. 1. FIG. 3 is a diagram illustrating the functional configuration of the control unit of the terminal device shown in FIG. 2. Figure 4 is a schematic diagram illustrating the robot area. FIG. 5 is a flowchart for explaining an example of the operation of a terminal device. Specific details for implementing the invention

[0020] Preferred embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in each drawing, elements denoted by the same reference numeral have the same or similar configuration. Furthermore, since the drawings are schematic, the dimensions or proportions of each component may differ from the actual ones.

[0021] FIG. 1 is a diagram illustrating the system configuration of a welding robot system including a terminal device according to an embodiment. The welding robot system (100) comprises, for example, a terminal device (1), a robot control device (2), and a manipulator (industrial robot) (3). The terminal device (1) and the robot control device (2), and the robot control device (2) and the manipulator (3) are each connected via a network. The terminal device (1) and the robot control device (2) are connected via wireless communication, for example, WiFi (Wireless Fidelity), and the robot control device (2) and the manipulator (3) are connected via, for example, a communication cable. Additionally, the network may be wired (including a communication cable) or wireless.

[0022] Additionally, the welding robot system (100) may include a teach pendant. The teach pendant is a control device that can be connected to a robot control device (2) and is used by an operator to teach the operation of the manipulator (3).

[0023] The manipulator (3) is a welding robot that performs arc welding on a workpiece (W) to be welded according to construction conditions set by the robot control device (2). The manipulator (3) has a multi-joint arm installed on a base member fixed to, for example, the floor of a factory, and a welding torch (tool) connected to the tip of the multi-joint arm. Here, the welding wire supplied to the welding torch is not included in the configuration of the manipulator (3).

[0024] A marker (M) is mounted on the welding torch portion of the manipulator (3). The marker (M) can be an identifier that can be recognized by the terminal device (1) as existing within the workspace, but preferably, an AR marker is used. By using an AR marker, when an AR marker existing within the workspace is recognized, it is possible to easily realize a user coordinate system with the AR marker as the origin superimposed on the actual image and display it.

[0025] As shown in FIG. 2, the robot control device (2) is a control unit that controls the operation of the manipulator (3), and is equipped with, for example, a control unit (21), a memory unit (22) and a communication unit (23).

[0026] The control unit (21) is a processor and controls the manipulator (3) by executing a welding program, etc., stored in the memory unit (22).

[0027] The communication unit (23) is a communication interface and controls communication with a terminal device (1) or manipulator (3) connected through a network.

[0028] Additionally, the robot control device (2) may further be equipped with a welding power supply unit. The welding power supply unit supplies welding current and welding voltage, etc., to the manipulator (3) according to predetermined welding conditions in order to generate an arc between the tip of the welding wire and the workpiece (W), for example. The welding power supply unit is the robot control device (2) and It can be provided separately.

[0029] The terminal device (1) is, for example, a portable terminal equipped with a digital camera. The portable terminal includes, for example, a tablet terminal, a smartphone, a portable information terminal (PDA), a notebook PC (personal computer), and other portable terminals. The terminal device (1) includes, for example, a control unit (11), a memory unit (12), a communication unit (13), a shooting unit (14), a distance measuring unit (15), and a display unit (16).

[0030] The control unit (11) is a processor and controls each part of the terminal device (1) by executing a program stored in the memory unit (12).

[0031] The memory unit (12) is a computer-readable recording medium and stores a program for realizing various functions of the terminal device (1) and various data used in the program. The various data includes, for example, shape and size information for each type of manipulator (3) described later, and shape and size information for each type of tool mounted on the manipulator (3).

[0032] The communication unit (13) is a communication interface and controls communication with a robot control device (2) connected through a network.

[0033] The shooting unit (14) is, for example, a 2D camera including a lens and an image sensor (imaging element), and converts light from a subject received from the lens into an electrical signal (digital image data).

[0034] The distance measuring unit (15) is, for example, a 3D camera equipped with a distance measuring sensor. The distance measuring sensor is a sensor capable of measuring the distance to an object. As a distance measuring sensor, for example, a LiDAR (Light Detection and Ranging) sensor, a millimeter wave sensor, an ultrasonic sensor, etc. can be used.

[0035] Additionally, the distance measuring unit (15) may be either a 3D camera or a distance measuring sensor. In the case of a 3D camera only, it is preferable to calculate three-dimensional coordinate data corresponding to an object based on multiple images of the object taken from multiple different positions. In this case, a known three-dimensional measurement method based on the stereo method may be used.

[0036] Here, by including a distance measuring sensor in the terminal device (1), the positional relationship between the image sensor and the distance measuring sensor can be fixed, and it becomes possible to synchronize the timing of acquiring data with each sensor. Therefore, it becomes possible to improve the processing efficiency of setting a specific position of the marker described later on the point cloud data. In addition, by equipping the image sensor and the distance measuring sensor in the terminal device (1), the operator operating the terminal device (1) can freely move to any position where the manipulator (3) and the work (W) can be photographed simultaneously, thereby increasing work efficiency.

[0037] In addition, the terminal device (1) may include a sensor that combines the function of an image sensor that captures and acquires an image and the function of a distance measuring sensor that measures and acquires a distance. By doing so, since an image including a welding target and the distance to the welding target can be acquired from the same location at the same timing, it becomes possible to further increase the processing efficiency of setting a specific position of a marker, which will be described later, on the point cloud data.

[0038] The display unit (16) is, for example, a display having a touch panel, and displays an image of a subject captured by the shooting unit (14) and simultaneously receives input such as operation instructions from an operator. The display unit (16) may be, for example, a display device having a touch panel and may be provided separately from the terminal device (1).

[0039] FIG. 3 is a diagram illustrating the functional configuration of the control unit (11) of the terminal device (1) according to the present invention. The control unit (11) of the terminal device (1) has, for example, a coordinate system setting unit (111), a coordinate assignment unit (112), an area specification unit (113), a point cloud creation unit for avoidance (114), and a program creation unit (115) as functional configurations.

[0040] The coordinate system setting unit (111) sets a three-dimensional user coordinate system based on a marker (M) included in an image captured by the shooting unit (14). The image captured by the shooting unit (14) includes, for example, a manipulator (3) and a work (W) existing in the space where the manipulator (3) works, as shown in FIG. 1. A marker (M) is mounted on the tip of the arm of the manipulator (3).

[0041] Specifically, the coordinate system setting unit (111) sets a three-dimensional user coordinate system with a specific location of the marker (M) (e.g., a corner of the marker or the center of the marker) as the origin.

[0042] In addition, the origin of the user coordinate system is not limited to a specific position of the marker (M), but may also be the origin of the robot coordinate system determined based on the specific position of the marker (M). Specifically, by fixing the position of the marker (M) relative to the manipulator (3), the specific position of the marker (M) can be determined in the robot coordinate system, so it becomes possible to determine the origin of the robot coordinate system based on the specific position of the marker (M). In this case, the user coordinate system becomes the robot coordinate system.

[0043] The coordinate assignment unit (112) assigns coordinates of the user coordinate system to point cloud data obtained by the distance measuring unit (15), which measures the distance to an object included in an image captured by the shooting unit (14).

[0044] The following is explained in detail. The coordinate assignment unit (112) detects a specific location of a marker (M) based on an image captured by the shooting unit (14), and sets the detected specific location of the marker on the point cloud data obtained by the distance measuring unit (15). The coordinate assignment unit (112) assigns coordinates of a user coordinate system, with the specific location of the marker set on the point cloud data as the origin (reference), to the point cloud data. By doing so, the point cloud data is drawn in the user coordinate system. The specific location of the marker set on the point cloud data may be specified, for example, by automatically recognizing the specific location of the marker on the point cloud data through data analysis, or by the operator pointing to and indicating the specific location of the marker on the point cloud data.

[0045] The area specification part (113) specifies a robot area in the user coordinate system corresponding to the manipulator (3) based on shape size information of the manipulator (3) corresponding to the type of manipulator (3) and posture information of the manipulator (3).

[0046] It is preferable to obtain the type of manipulator (3) and the posture information of the manipulator (3) from a robot control device (2) that controls the manipulator (3). The posture information of the manipulator (3) may be, for example, angle information of each axis of the manipulator (3).

[0047] The area specification part (113) may specify the robot area in the user coordinate system corresponding to the manipulator (3) when the tool is mounted by further utilizing shape and size information of the tool corresponding to the type of tool, such as a welding torch, when specifying the robot area.

[0048] It is preferable that the robot area be an area larger than the size of the manipulator (3) when the tool is mounted, which is determined based on the shape size information of the manipulator (3), the posture information of the manipulator (3), and the shape size information of the tool. With reference to FIG. 4, the robot area will be described in detail.

[0049] FIG. 4 illustrates, for example, a workspace of a manipulator (3) equipped with a welding torch, which is a tool, and a jig (J) for placing a workpiece. A robot area (R) is installed to surround the entire manipulator (3) equipped with the welding torch, and no robot area is installed in the workspace other than that containing the jig (J). The robot area (R) is installed over a larger area than the manipulator (3) equipped with the welding torch.

[0050] A range larger than the manipulator (3) equipped with a welding torch is, for example, a range capable of absorbing errors that may occur in the point cloud data obtained by the distance measuring unit (15), and it is preferable to have a size that provides a margin for error with respect to the size of the manipulator (3) when the welding torch is equipped. By doing so, the robot area (R) can be specified by eliminating the influence of errors that may occur in the point cloud data, thereby making it possible to improve the quality of the point cloud data for interference avoidance.

[0051] The point cloud generation unit (114) generates point cloud data for interference avoidance by removing point cloud data included in the robot area (R) from the point cloud data obtained by the distance measurement unit (15). In FIG. 4, among the point cloud data obtained by the distance measurement unit (15), the point cloud data included in the robot area (R) is removed, and the point cloud data corresponding to the remaining jig (J) is generated as point cloud data for interference avoidance.

[0052] The program creation unit (115) creates a welding program so that the manipulator (3) performs a welding operation while avoiding interference with the interference avoidance point cloud data.

[0053] Here, interference in this embodiment refers to a case where, when the manipulator (3) performs a welding operation, the welding operation of the manipulator (3) is hindered (normal welding operation is impeded) by contacting an object corresponding to the interference avoidance point cloud data, etc.

[0054] Specifically, the program creation unit (115) creates a welding program so that a manipulator (3) virtually placed in a user coordinate system avoids interference with interference avoidance point cloud data drawn in the user coordinate system and also finds the welding line of the welding target.

[0055] More specifically, when the program creation unit (115) determines that there is a point in the movement trajectory of the manipulator (3) in the user coordinate system where the manipulator (3) interferes with the interference avoidance point cloud data, it changes a part of the movement trajectory of the manipulator (3) in a direction that can avoid interference with the interference avoidance point cloud data. On the other hand, when the program creation unit (115) determines that there is no point in the movement trajectory where there is interference with the interference avoidance point cloud data, it converts the movement trajectory in the user coordinate system into a movement trajectory in the robot coordinate system and creates a welding program.

[0056] The virtual location where the manipulator (3) is placed can be arbitrarily designated by the operator touching the image of the workspace, for example, according to a guide such as voice or text message.

[0057] In addition, the virtually placed manipulator (3) may be a model connected to, for example, a wire feeder or various sensors.

[0058] The program creation unit (115) transmits the created welding program to the robot control device (2) and stores it in the memory unit (22) of the robot control device (2). By doing so, when the manipulator (3) performs arc welding, the control unit (21) of the robot control device (2) reads the welding program and makes it possible to control the manipulator (3) to perform arc welding while avoiding interference with an object according to the welding sequence specified in the welding program.

[0059] Referring to FIG. 5, an example of the operation of the terminal device (1) will be described.

[0060] First, the coordinate system setting unit (111) of the terminal device (1) sets a three-dimensional user coordinate system based on a marker (M) included in an image captured by the shooting unit (14) (step S101).

[0061] Next, the coordinate assignment unit (112) of the terminal device (1) assigns coordinates of the user coordinate system to the point cloud data obtained by the distance measurement unit (15) (step S102).

[0062] Next, the area specification part (113) of the terminal device (1) specifies a robot area in the user coordinate system corresponding to the manipulator (3) when the welding torch is mounted, based on the shape size information of the manipulator (3), the attitude information of the manipulator (3), and the shape size information of the tool (step S103).

[0063] Next, the point cloud generation unit (114) for avoidance of the terminal device (1) generates point cloud data for interference avoidance by removing point cloud data included in the robot area specified in step S103 from the point cloud data obtained by the distance measurement unit (15) (step S104).

[0064] Next, the program creation unit (115) of the terminal device (1) creates a welding program (step S105) so that the manipulator (3) performs a welding operation while avoiding interference with the interference avoidance point cloud data created in step S104. Then, the operation is terminated.

[0065] As described above, according to the terminal device (1) of the embodiment, a user coordinate system is established based on a marker (M) included in an image of a manipulator (3) and its workspace, coordinates of the user coordinate system are assigned to point cloud data corresponding to an object included in the image, a robot area (R) on the user coordinate system corresponding to the manipulator (3) is specified, and point cloud data included in the robot area (R) is removed from the point cloud data corresponding to an object included in the image to create point cloud data for interference avoidance. By doing so, it becomes possible to specify an area formed based on point cloud data by removing point cloud data corresponding to the manipulator (3) from the point cloud data corresponding to the captured image as an interference area.

[0066] Therefore, according to the terminal device (1) according to the embodiment, it is possible to increase the precision of specifying the interference area.

[0067] Furthermore, the present disclosure is not limited to the embodiments described above and may be implemented in various other forms without departing from the essence of the disclosure. Accordingly, the above embodiments are merely examples in all respects and are not to be interpreted restrictively.

[0068] For example, in the above-described embodiments, a welding robot was used for explanation, but the invention is not limited thereto. For example, the present disclosure may be applied to industrial robots including handling robots that perform picking, etc. In this case, the welding program, welding target, welding line, and welding operation used in the above embodiments may be replaced with a work program, work target, work location, and work, respectively.

[0069] In addition, in the above-described embodiment, an image of the workspace including the manipulator (3) is captured using a single terminal device (1), but multiple terminal devices (1) may be used to capture images of the workspace including the manipulator (3) from different locations. By doing so, the presence or absence of interference can be determined for each captured location, and if interference is determined to exist, adjustments can be made to avoid the interference. Therefore, it is possible to increase the precision of avoiding interference. Explanation of the symbols

[0070] 1 : Terminal device 2 : Robot Control Unit 3 : Manipulator 11: Control unit 12 : Memory 13 : Communications Department 14 : Filming Department 15: Distance measuring unit 16 : Display section 21 : Control unit 22 : Memory 23 : Communications Department 100: Welding robot system 111 : Coordinate system setting section 112 : Coordinate assignment section 113 : Area specific part 114 : Point Cloud Generation Section for Evasion 115 : Program writing section M : Marker R: Robotics Area W : Work

Claims

Claim 1 A terminal device comprising: a coordinate system setting unit that sets a user coordinate system based on a marker included in an image including an industrial robot and a workspace of said industrial robot captured by a shooting unit; a coordinate assignment unit that assigns coordinates of said user coordinate system to point cloud data obtained by a distance measuring unit that measures the distance to an object included in said image; an area specification unit that specifies a robot area on said user coordinate system corresponding to said industrial robot based on shape size information of said industrial robot corresponding to a type of said industrial robot and attitude information of said industrial robot; and an avoidance point cloud generation unit that generates point cloud data for interference avoidance by removing point cloud data included in said robot area from the point cloud data obtained by the distance measuring unit. Claim 2 A terminal device according to claim 1, further comprising a program writing unit that writes a work program to enable the industrial robot to perform a work while avoiding interference with the interference avoidance point cloud data. Claim 3 A terminal device according to claim 1, wherein the area specifying part further utilizes shape and size information of the tool corresponding to the type of tool mountable to the industrial robot when specifying the robot area, and specifies the robot area on the user coordinate system corresponding to the industrial robot equipped with the tool. Claim 4 A terminal device that acquires type information of the industrial robot and attitude information of the industrial robot from a control device that controls the industrial robot, in accordance with claim 1. Claim 5 In claim 1, the terminal device wherein the posture information of the industrial robot is angle information of each axis of the industrial robot. Claim 6 A terminal device according to claim 1, wherein the robot area is an area larger than the size of the industrial robot determined based on shape size information of the industrial robot and posture information of the industrial robot.

Images