Terminal device
By setting the user coordinate system and using distance measurement technology through the terminal device, combined with the robot's shape and posture information, a welding program is generated, which solves the problem of low precision in the robot's interference area and achieves high-precision obstacle avoidance.
Patent Information
- Application Number
- CN202410434186.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2024-04-11
- Publication Date
- 2024-11-12
AI Technical Summary
Existing technologies are easily affected by factors such as changes in robot and workpiece color, stains, and lighting conditions when setting the robot interference area, leading to a decrease in the accuracy of interference area determination.
The user coordinate system is set through the terminal device, the robot area is determined based on the marks in the captured image, and the point group data is obtained using the distance measurement unit. Combined with the robot's shape and posture information, the point group data for interference avoidance is determined, and the welding program is generated to avoid interference.
This improves the accuracy of interference zone determination, ensuring that the robot can accurately avoid obstacles during operation.
Smart Images

Figure CN118927233A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a terminal device. Background Art
[0002] In the past, when a robot is used for work, in order to avoid interference between the robot and obstacles located around it, an interference area where obstacles exist is set around the robot, and control is performed so that the robot does not enter the interference area. As a method of setting an interference area around a robot, for example, the following patent document 1 discloses a method of setting an interference area by setting the robot and other objects to different colors. In this method, a 3D map is created based on an image of a 3D camera photographing the robot's surroundings, and the point column corresponding to the color of the robot is removed from each point column of the 3D map, and the interference area is set using the position data of the point column remaining in the 3D map.
[0003] Prior Art Literature
[0004] Patent Literature
[0005] Patent Document 1: JP Patent No. 5980873
[0006] However, the method of Patent Document 1 has the constraint that the robot and the workpiece must be different colors. Moreover, for example, it may become difficult to distinguish the color of the robot from other colors due to the adhesion of stains to the robot or the workpiece, changes in the color or brightness of the lighting, etc., and it may become impossible to correctly determine the interference area. Summary of the invention
[0007] Therefore, an object of the present disclosure is to provide a terminal device that can improve the accuracy of identifying an interference area.
[0008] A terminal device involved in one method of the present disclosure comprises: a coordinate system setting unit, which sets a user coordinate system based on a mark contained in an image captured by a capturing unit and including an industrial robot and a work space of the industrial robot; a coordinate assigning unit, which assigns coordinates of the user coordinate system to point group data obtained by a distance measuring unit that measures the distance to an object contained in the image; an area determination unit, which determines a robot area on the user coordinate system corresponding to the industrial robot based on shape and size information of the industrial robot corresponding to the type of the industrial robot and posture information of the industrial robot; and an avoidance point group creating unit, which creates interference avoidance point group data by removing the point group data contained in the robot area from the point group data obtained by the distance measuring unit.
[0009] According to this method, a user coordinate system can be set based on a mark contained in an image of an industrial robot and its work space, coordinates of the user coordinate system are assigned to point group data corresponding to an object contained in the image, a robot region on the user coordinate system corresponding to the industrial robot is determined, and point group data contained in the robot region is removed from point group data corresponding to the object contained in the image, thereby creating point group data for interference avoidance. Thus, a region formed based on point group data obtained by removing point group data corresponding to the industrial robot from point group data corresponding to the captured image can be determined as an interference region.
[0010] The above aspect may further include a program creation unit that creates an operation program so that the industrial robot performs the operation while avoiding interference with the interference avoiding point group data.
[0011] According to this aspect, the industrial robot can be operated while avoiding interference with objects other than the industrial robot.
[0012] In the above aspect, when determining the robot region, the region specifying unit may further use shape and size information of a tool corresponding to a type of a tool that can be equipped on the industrial robot to specify the robot region on the user coordinate system corresponding to the industrial robot equipped with the tool.
[0013] According to this aspect, since the area of the tool equipped on the industrial robot can be included in the robot area, the interference avoidance point group data can be created based on the robot area that is different for each equipped tool.
[0014] In the above aspect, the type of the industrial robot and the posture information of the industrial robot may be acquired from a control device that controls the industrial robot.
[0015] According to this aspect, since the robot area can be specified based on the information acquired from the control device that controls the posture of the industrial robot based on the type of the industrial robot, the accuracy of specifying the robot area can be improved.
[0016] In the above aspect, the posture information of the industrial robot may be angle information of each axis of the industrial robot.
[0017] According to this aspect, since the robot area can be determined based on the angle of each axis of the industrial robot, it is possible to determine the robot area that matches the angle of each axis that changes in accordance with the operation of the robot.
[0018] In the above aspect, the robot area may be an area larger than the size of the industrial robot determined based on the shape and size information of the industrial robot and the posture information of the industrial robot.
[0019] According to this aspect, since point group data included in a region larger than the size of the industrial robot can be removed, the accuracy of removing point group data corresponding to the industrial robot can be improved.
[0020] Effects of the Invention
[0021] According to the present disclosure, it is possible to provide a terminal device capable of improving the accuracy of determining an interference area. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is a diagram illustrating a system configuration of a welding robot system including a terminal device according to an embodiment.
[0023] Figure 2 is an example Figure 1 A diagram showing the physical structure of the terminal device and the robot control device.
[0024] Figure 3 is an example Figure 2 A diagram of the functional structure of the control unit of the terminal device shown.
[0025] Figure 4 is a schematic diagram for explaining the robot area.
[0026] Figure 5 This is a flowchart for explaining an example of the operation of the terminal device.
[0027] Explanation of symbols
[0028] 1…Terminal device, 2…Robot control device, 3…Manipulator, 11…Control unit, 12…Storage unit, 13…Communication unit, 14…Image capture unit, 15…Distance measurement unit, 16…Display unit, 21…Control unit, 22…Storage unit, 23…Communication unit, 100…Welding robot system, 111…Coordinate system setting unit, 112…Coordinate assignment unit, 113…Region determination unit, 114…Avoidance point group creation unit, 115…Program creation unit, M…Mark, R…Robot region, W…Workpiece DETAILED DESCRIPTION
[0029] The preferred embodiment of the present invention will be described with reference to the accompanying drawings. In addition, in each figure, the components with the same symbols have the same or similar structure. In addition, since the accompanying drawings are schematic, the size and ratio of each structural element are different from the actual one.
[0030] Figure 11 is a diagram illustrating a system configuration of a welding robot system including a terminal device according to an embodiment. The welding robot system 100 includes, for example, a terminal device 1, a robot control device 2, and a manipulator (industrial robot) 3. The terminal device 1 and the robot control device 2, and the robot control device 2 and the manipulator 3 are connected via a network, respectively. The terminal device 1 and the robot control device 2 are connected via wireless communication such as WiFi (Wireless Fidelity), and the robot control device 2 and the manipulator 3 are connected via a communication cable, for example. In addition, the network may be wired (including a communication cable) or wireless.
[0031] Furthermore, the welding robot system 100 may include a teaching pendant. The teaching pendant can be connected to the robot control device 2 and is an operating device used by an operator to teach the operation of the robot hand 3.
[0032] The robot 3 is a welding robot that performs arc welding on a workpiece W that is a welding target according to the operation conditions set in the robot control device 2. The robot 3 includes, for example, a multi-joint arm provided on a base member fixed to the floor of a factory, and a welding torch (tool) connected to the front end of the multi-joint arm. Here, the welding wire supplied to the welding torch is not included in the structure of the robot 3.
[0033] A marker M is mounted on the welding torch of the robot 3. The marker M may be an identifier that allows the terminal device 1 to recognize that the marker M exists in the workspace, and preferably an AR marker is used. By using the AR marker, when the AR marker existing in the workspace is recognized, it is possible to easily realize the overlap display of the user coordinate system with the AR marker as the origin and the actual image.
[0034] like Figure 2 As shown, the robot control device 2 is a control unit that controls the operation of the manipulator 3 , and includes, for example, a control unit 21 , a storage unit 22 , and a communication unit 23 .
[0035] The control unit 21 is a processor, and controls the robot 3 by executing a welding program or the like stored in the storage unit 22 .
[0036] The communication unit 23 is a communication interface, and controls communication with the terminal device 1 and the robot 3 connected via a network.
[0037] In addition, the robot control device 2 may further include a welding power supply unit. The welding power supply unit supplies welding current and welding voltage to the robot hand 3 according to predetermined welding conditions, for example, in order to generate an arc between the tip of the welding wire and the workpiece W. The welding power supply unit may be provided separately from the robot control device 2.
[0038] The terminal device 1 is, for example, a portable terminal with a digital camera. Portable terminals include portable terminals such as tablet terminals, smart phones, portable information terminals (PDAs), and notebook PCs (personal computers). The terminal device 1 includes, for example, a control unit 11, a storage unit 12, a communication unit 13, a camera unit 14, a distance measurement unit 15, and a display unit 16.
[0039] The control unit 11 is a processor, and controls each unit of the terminal device 1 by executing a program stored in the storage unit 12 .
[0040] The storage unit 12 is a computer-readable recording medium that stores programs for implementing various functions of the terminal device 1 and various data used in the programs. The various data include, for example, shape and size information of each type of the robot 3 described later and shape and size information of each type of tools equipped on the robot 3.
[0041] The communication unit 13 is a communication interface, and controls communication with the robot control device 2 connected via a network.
[0042] The imaging unit 14 is, for example, a 2D camera including a lens and an image sensor (imaging element), and converts light of a subject received by the lens into an electrical signal (digital image data).
[0043] The distance measuring unit 15 is, for example, a 3D camera equipped with a distance measuring sensor. The distance measuring sensor is a sensor that can measure the distance to an object. As the distance measuring sensor, for example, a LiDAR (Light Detection and Ranging) sensor, a millimeter wave sensor, an ultrasonic sensor, etc. can be used.
[0044] In addition, the distance measuring unit 15 may be only a 3D camera or a distance measuring sensor. In the case of only a 3D camera, it is preferred to estimate the three-dimensional coordinate data corresponding to the object based on a plurality of images obtained by photographing the object from a plurality of different positions. In this case, a three-dimensional measurement method based on a known stereo method can be used.
[0045] Here, by including a distance measuring sensor in the terminal device 1, the positional relationship between the image sensor and the distance measuring sensor can be fixed, and the timing of obtaining data by each sensor can be coordinated. Therefore, the processing efficiency of setting the specific position of the mark described later on the point group data can be improved. In addition, by including an image sensor and a distance measuring sensor in the terminal device 1, since the operator operating the terminal device 1 can freely move to any position where the robot 3 and the workpiece W can be photographed at the same time, the working efficiency can be improved.
[0046] Furthermore, a sensor having both the function of an image sensor for capturing an image and the function of a distance measuring sensor for measuring a distance may be included in the terminal device 1. In this way, since an image including the welding object and the distance to the welding object can be acquired from the same part at the same timing, the processing efficiency of setting a specific position of a mark described later on the point group data can be further improved.
[0047] The display unit 16 is, for example, a display having a touch panel, and displays the image of the subject captured by the image capturing unit 14 and receives inputs based on operation instructions from an operator. The display unit 16 may be provided separately from the terminal device 1 as a display device having a touch panel, for example.
[0048] Figure 3 This is a diagram illustrating a functional configuration of the control unit 11 of the terminal device 1 according to the present invention. The control unit 11 of the terminal device 1 includes, for example, a coordinate system setting unit 111, a coordinate providing unit 112, an area specifying unit 113, an avoidance point group creating unit 114, and a program creating unit 115 as a functional configuration.
[0049] The coordinate system setting unit 111 sets a three-dimensional user coordinate system based on the marker M included in the image captured by the imaging unit 14. Figure 1 As shown, the image captured by the imaging unit 14 includes, for example, the robot 3 and the workpiece W that exist in the space where the robot 3 performs work. A marker M is attached to the tip of the arm of the robot 3 .
[0050] Specifically, the coordinate system setting unit 111 sets a three-dimensional user coordinate system with a specific position of the marker M (eg, a corner of the marker, the center of the marker, etc.) as the origin.
[0051] In addition, the origin of the user coordinate system is not limited to the specific position of the mark M, and may be the origin of the robot coordinate system determined based on the specific position of the mark M. Specifically, since the specific position of the mark M can be determined on the robot coordinate system by fixing the position of the mark M relative to the manipulator 3, the origin of the robot coordinate system can be determined based on the specific position of the mark M. In this case, the user coordinate system becomes the robot coordinate system.
[0052] The coordinate assigning unit 112 assigns coordinates of the user coordinate system to point cloud data obtained by the distance measuring unit 15 that measures the distance to an object included in the image captured by the imaging unit 14 .
[0053] The specific description is as follows. The coordinate assigning unit 112 detects the specific position of the mark M based on the image captured by the capturing unit 14, and sets the detected specific position of the mark on the point group data obtained by the distance measuring unit 15. The coordinate assigning unit 112 assigns the coordinates of the user coordinate system with the specific position of the mark set on the point group data as the origin (reference) to the point group data. Thus, the point group data is drawn in the user coordinate system. Regarding the specific position of the mark set on the point group data, for example, the specific position of the mark on the point group data can be automatically identified by data analysis, or the operator can point out the specific position of the mark on the point group data to specify it.
[0054] The region identification unit 113 identifies the robot region on the user coordinate system corresponding to the manipulator 3 based on the shape and size information of the manipulator 3 corresponding to the type of the manipulator 3 and the posture information of the manipulator 3 .
[0055] The type of the robot 3 and the posture information of the robot 3 are preferably acquired from the robot control device 2 that controls the robot 3. The posture information of the robot 3 may be, for example, angle information of each axis of the robot 3.
[0056] When determining the robot region, the region identification unit 113 may further use the shape and size information of the tool corresponding to the type of the tool such as a welding torch equipped in the robot 3 to determine the robot region on the user coordinate system corresponding to the robot 3 when the tool is equipped.
[0057] The robot area is preferably set to an area larger than the size of the robot 3 when the tool is equipped, which is determined based on the shape and size information of the robot 3, the posture information of the robot 3, and the shape and size information of the tool. Figure 4 Let's specify the robot area.
[0058] exist Figure 4 2 shows an example of a workspace including a robot 3 equipped with a welding torch as a tool and a robot 3 having a fixture J for placing a workpiece. A robot region R is provided to surround the robot 3 equipped with the welding torch as a whole, and no robot region is provided in the workspace other than the fixture J. The robot region R is provided to be larger than the robot 3 equipped with the welding torch.
[0059] The range larger than the robot 3 equipped with a welding torch is, for example, a range that can absorb errors that may occur in the point group data obtained by the distance measuring unit 15, and is preferably set to have a size that is more than the size of the robot 3 when equipped with a welding torch. Thus, since the influence of errors that may occur in the point group data can be removed and the robot area R can be determined, the quality of the point group data for interference avoidance can be improved.
[0060] The avoidance point group creation unit 114 creates interference avoidance point group data by removing the point group data included in the robot region R from the point group data obtained by the distance measurement unit 15. Figure 4 In the above, the point group data included in the robot region R is removed from the point group data obtained by the distance measuring unit 15, and the remaining point group data corresponding to the fixture J is created as the interference avoidance point group data.
[0061] The program creation unit 115 creates a welding program so that the robot 3 performs a welding operation while avoiding interference with the interference avoiding point group data.
[0062] Here, interference in this embodiment means that the robot 3 contacts an object corresponding to the interference avoidance point group data during welding operation, thereby hindering the welding operation of the robot 3 (causing a failure in normal welding operation).
[0063] Specifically, the program creation unit 115 creates a welding program so that the robot 3 virtually placed in the user coordinate system avoids interference with the interference avoiding point group data drawn in the user coordinate system and moves along the welding line of the welding object.
[0064] More specifically, when it is determined that there is a portion in the motion trajectory of the robot 3 in the user coordinate system where the robot 3 interferes with the interference avoidance point group data, the program creation unit 115 changes a portion of the motion trajectory of the robot 3 in a direction that can avoid interference with the interference avoidance point group data. On the other hand, when it is determined that there is no portion in the motion trajectory that interferes with the interference avoidance point group data, the program creation unit 115 converts the motion trajectory in the user coordinate system into the motion trajectory in the robot coordinate system to create a welding program.
[0065] The position where the robot 3 is virtually placed can be arbitrarily designated by the operator, for example, by touching the image of the work space in accordance with guidance such as voice or text messages.
[0066] Furthermore, the virtually placed robot 3 may be, for example, a model to which a wire feeding device, various sensors, and the like are connected.
[0067] The program creation unit 115 sends the created welding program to the robot control device 2, and stores it in the storage unit 22 of the robot control device 2. Thus, when the robot 3 performs arc welding, the control unit 21 of the robot control device 2 reads the welding program and controls the robot 3 according to the welding steps specified in the welding program so that arc welding is performed while avoiding interference with the object.
[0068] refer to Figure 5 An example of the operation of the terminal device 1 will be described.
[0069] First, the coordinate system setting unit 111 of the terminal device 1 sets a three-dimensional user coordinate system based on the marker M included in the image captured by the imaging unit 14 (step S101 ).
[0070] Next, the coordinate assigning unit 112 of the terminal device 1 assigns coordinates of the user coordinate system to the point cloud data obtained by the distance measuring unit 15 (step S102 ).
[0071] Next, the area identification unit 113 of the terminal device 1 identifies the robot area on the user coordinate system corresponding to the manipulator 3 when the welding torch is equipped based on the shape and size information of the manipulator 3 , the posture information of the manipulator 3 , and the shape and size information of the tool (step S103 ).
[0072] Next, the avoidance point group creation unit 114 of the terminal device 1 removes the point group data included in the robot region specified in step S103 from the point group data obtained by the distance measurement unit 15 to create interference avoidance point group data (step S104).
[0073] Next, the program creation unit 115 of the terminal device 1 creates a welding program so that the robot 3 performs a welding operation while avoiding interference with the interference avoiding point group data created in step S104 (step S105 ). Then, this operation ends.
[0074] As described above, according to the terminal device 1 involved in the embodiment, the user coordinate system can be set based on the mark M included in the image obtained by photographing the robot 3 and its work space, the coordinates of the user coordinate system are assigned to the point group data corresponding to the object included in the image, the robot area R on the user coordinate system corresponding to the robot 3 is determined, and the point group data included in the robot area R is removed from the point group data corresponding to the object included in the image, thereby creating the point group data for interference avoidance. Thus, the area formed based on the point group data corresponding to the captured image by removing the point group data corresponding to the robot 3 can be determined as the interference area.
[0075] Therefore, according to the terminal device 1 involved in the embodiment, the accuracy of specifying the interference area can be improved.
[0076] In addition, the present disclosure is not limited to the above-mentioned embodiments, and can be implemented in various other forms within the scope not departing from the gist of the present disclosure. Therefore, the above-mentioned embodiments are merely illustrative in all points and are not to be interpreted as limiting.
[0077] For example, in the above-mentioned embodiments, the welding robot is used for description, but the present invention is not limited thereto. For example, the present disclosure can be applied to an industrial robot including an operating robot for picking up, etc. In this case, the welding program, welding object, welding line, and welding action used in the above-mentioned embodiments can be replaced with an operation program, an operation object, an operation part, and an operation, respectively.
[0078] In addition, in the above-mentioned embodiment, an image of the workspace including the robot 3 is captured by one terminal device 1, but a plurality of terminal devices 1 may be used to capture images of the workspace including the robot 3 from different positions. Thus, the presence or absence of interference can be determined at each captured position, and when it is determined that interference exists, adjustments are made to avoid interference. Thus, the accuracy of avoiding interference can be improved.
Claims
1. A terminal device comprising: a coordinate system setting unit that sets a user coordinate system based on a marker included in an image captured by the imaging unit and including the industrial robot and a work space of the industrial robot; a coordinate assigning unit that assigns coordinates of the user coordinate system to point group data obtained by a distance measuring unit that measures the distance to the object included in the image; an area determination unit that determines a robot area on the user coordinate system corresponding to the industrial robot based on shape and size information of the industrial robot corresponding to the type of the industrial robot and posture information of the industrial robot; and The avoidance point group creation unit creates interference avoidance point group data by removing point group data included in the robot region from the point group data obtained by the distance measurement unit.
2. The terminal device according to claim 1, wherein: The terminal device further includes a program creation unit that creates an operation program so that the industrial robot performs an operation while avoiding interference with the interference avoiding point group data.
3. The terminal device according to claim 1, wherein: When determining the robot region, the region identification unit further uses shape and size information of the tool corresponding to the type of the tool that can be equipped on the industrial robot to identify the robot region on the user coordinate system corresponding to the industrial robot equipped with the tool.
4. The terminal device according to claim 1, wherein: The terminal device obtains the type of the industrial robot and the posture information of the industrial robot from a control device that controls the industrial robot.
5. The terminal device according to claim 1, wherein: The posture information of the industrial robot is angle information of each axis of the industrial robot.
6. The terminal device according to claim 1, wherein: The robot area is an area larger than a size of the industrial robot determined based on shape and size information of the industrial robot and posture information of the industrial robot.
