Robot system

Abstract

This robotic system makes it easier to visualize how weld beads are formed. [Solution] The robot system 100 includes a display control unit 311 that displays an image on a display unit based on imaging information, which is captured by an imaging unit and includes at least the object to be welded by the welding robot, and a welding line detection unit 314 that detects the welding line of the object to be welded included in the imaging information. The display control unit 311 overlays a virtual bead image, which mimics the appearance of the welding bead formed by the welding work, onto the image based on the imaging information, along the detected welding line.

Description

Technical Field

[0001] The present invention relates to a robot system.

Background Art

[0002] Conventionally, in order to assist an operator in welding work, there is a technique of displaying a welding line, which is a welding portion of a workpiece, as a dotted line on an image of the workpiece to be welded.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, it is difficult to imagine how a welding bead is formed on a workpiece by simply displaying the welding line as a dotted line.

[0005] Therefore, an object of the present invention is to provide a robot system capable of easily imagining how a welding bead is formed.

Means for Solving the Problems

[0006] A robot system according to an aspect of the present invention includes a display control unit that causes a display unit to display an image based on imaging information imaged by an imaging unit and including at least a welding target on which a welding robot performs welding work, and a welding line detection unit that detects a welding line of the welding target included in the imaging information. The display control unit superimposes and displays a virtual bead image simulating the appearance of a welding bead formed by welding work along the detected welding line on the image based on the imaging information.

[0007] According to this embodiment, an image based on imaging information including the welding target captured by the imaging unit is displayed on the display unit, the welding line of the welding target is detected, and a virtual bead image that mimics the appearance of the welding bead formed by the welding operation is superimposed on the image including the welding target along the detected welding line.

[0008] In the above embodiment, when simulating welding work on a weld line, the display control unit may sequentially display virtual bead images along the detected weld line in accordance with the movement of the virtual welding torch displayed on the display unit.

[0009] According to this embodiment, a virtual bead image can be displayed in accordance with the movement of the virtual welding torch, allowing the operator to virtually confirm how the welding bead is formed during the welding process.

[0010] In the above embodiment, the display control unit may change the display of the virtual bead image depending on whether the detected weld line is hidden behind the object to be welded and not visible, or whether it is visible.

[0011] According to this embodiment, the worker can recognize the difference between the portion of the weld bead formed by the welding work that is hidden in the shadow of the object being welded and the portion that is visible.

[0012] In the above embodiment, the display control unit may display the portion of the detected weld line that is hidden behind the object being welded as a dotted line.

[0013] According to this embodiment, the worker can recognize that a weld bead is formed even in areas hidden and invisible behind the object being welded.

[0014] In the above embodiment, the display control unit may overlay the detected weld lines onto the image obtained from the imaging information, and when one of the displayed weld lines is selected, it may display a virtual bead image along the selected weld line.

[0015] According to this embodiment, the worker can check the appearance of the weld bead formed by the welding work for each selected weld line.

[0016] In the above embodiment, the display control unit may overlay the detected weld line onto the image obtained from the imaging information, further display a setting screen on the display unit for setting welding conditions when performing welding on the displayed weld line, and also overlay a virtual bead image of the weld bead formed based on the welding conditions set on the setting screen onto the image obtained from the imaging information.

[0017] According to this embodiment, the operator can set the welding conditions and simultaneously check the appearance of the weld bead, which changes according to those settings.

[0018] In the above embodiment, the display control unit may display the detected weld lines overlaid on the image obtained from the imaging information, and when one of the displayed weld lines is selected, it may further display an adjustment screen on the display unit for adjusting welding parameters that affect the welding conditions when welding the selected weld line.

[0019] According to this embodiment, the operator can adjust the welding parameters that affect the welding conditions when welding a weld line for each selected weld line.

[0020] In the above embodiment, the display control unit may change the shape of the virtual bead image in accordance with the values ​​of the welding parameters adjusted on the adjustment screen.

[0021] According to this embodiment, the operator can adjust the values ​​of the welding parameters and check the appearance of the weld bead, which changes as a result of those adjustments. [Effects of the Invention]

[0022] According to the present invention, it is possible to provide a robotic system that makes it easier to visualize how a weld bead is formed. [Brief explanation of the drawing]

[0023] [Figure 1] It is a diagram illustrating the configuration of the robot system according to the embodiment. [Figure 2] It is a diagram illustrating the functional configuration of the control unit of the terminal device. [Figure 3] It is an example of a screen displayed on the display unit. [Figure 4] It is an example of a screen displayed on the display unit. [Figure 5] It is an example of a screen displayed on the display unit. [Figure 6] It is an example of a screen displayed on the display unit. [Figure 7] It is a flowchart for explaining an example of the operation of the robot system according to the embodiment.

Mode for Carrying Out the Invention

[0024] Preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each figure, those denoted by the same reference numerals have the same or similar configurations. Also, since the drawings are schematic, the dimensions and ratios of each component are different from the actual ones.

[0025] FIG. 1 is a diagram illustrating the configuration of a robot system 100 according to an embodiment. The robot system 100 includes, for example, a robot control device 1, a manipulator 2, a terminal device 3, an imaging unit 4, and a display unit 5.

[0026] The robot control device 1 and the terminal device 3, and the terminal device 3 and the imaging unit 4 and the display unit 5 are connected via a network, respectively. The robot control device 1 and the manipulator 2 are connected by a motor and an encoder wire. The network may be by wireless communication such as WiFi (Wireless Fidelity), or may be by wired communication such as a communication cable. Note that the terminal device 3 may include the imaging unit 4 and the display unit 5.

[0027] Manipulator 2 is a welding robot that performs arc welding on a workpiece (base material) to be welded according to the working conditions set in the robot control device 1. Manipulator 2 has, for example, a multi-joint arm mounted on a base member fixed to the floor of a factory, and a welding torch connected to the tip of the multi-joint arm as one of the work tools.

[0028] Manipulator 2 is a collaborative robot (industrial robot) designed for joint work with humans, and can operate in response to external forces. Operators can directly apply external force to Manipulator 2 by hand, and move Manipulator 2 in response to that force to directly control it.

[0029] The robot control device 1 is a control unit that controls the operation of the manipulator 2. The robot control device 1 includes, for example, a control unit 11, a storage unit 12, and a communication unit 13.

[0030] The control unit 11 is a processor that controls the manipulator 2 by executing work programs and the like stored in the memory unit 12.

[0031] The memory unit 12 is a computer-readable recording medium that stores programs for realizing various functions in the robot control device 1, as well as various data used in those programs.

[0032] The communication unit 13 is a communication interface that controls communication with the manipulator 2, which is connected by motor and encoder wires, and with the terminal device 3, which is connected via a network.

[0033] The robot control device 1 may further include a welding power supply unit. The welding power supply unit supplies welding current, welding voltage, etc., to the welding torch connected to the manipulator 2, for example, in order to generate an arc between the tip of the welding wire and the workpiece, according to predetermined welding conditions. The welding power supply unit may be provided separately from the robot control device 1.

[0034] Terminal device 3 is a terminal that performs processing when an operator teaches the operation of manipulator 2, and is, for example, a tablet-type teach pendant. Some or all of the functions of terminal device 3, which will be described later, may be provided in the robot control device 1, or they may be provided in a server device on the cloud. Terminal device 3 includes, for example, a control unit 31, a storage unit 32, and a communication unit 33.

[0035] The control unit 31 is a processor that controls each part of the terminal device 3 by executing programs stored in the memory unit 32. The functions of the control unit 31 will be described later.

[0036] The memory unit 32 is a computer-readable recording medium that stores programs for realizing various functions in the terminal device 3, as well as various data used by those programs.

[0037] The communication unit 33 is a communication interface that controls communication with the robot control device 1, imaging unit 4, and display unit 5, which are connected via the network.

[0038] The imaging unit 4 is, for example, a 3D camera equipped with a distance measurement sensor, and also functions as a 2D camera. Therefore, the imaging unit 4 can provide 3D point cloud data and 2D image data as imaging information.

[0039] It is preferable to position the imaging unit 4 in a location that allows it to capture images of at least the work space including the workpiece. For example, the imaging unit 4 may be fixedly positioned around the manipulator 2 or in a robot cell, or it may be attached to the manipulator 2 near the component to which the welding torch is attached. Alternatively, the imaging unit 4 may be mounted on the terminal device 3.

[0040] A distance measuring sensor is a sensor capable of measuring the distance to an object. Examples of distance measuring sensors that can be used include LiDAR (Light Detection and Ranging) sensors, millimeter-wave sensors, and ultrasonic sensors.

[0041] The imaging unit 4 does not necessarily need to be equipped with a distance measurement sensor; the imaging unit 4 and the distance measurement sensor may be provided separately, or the distance measurement sensor may be omitted. If the distance measurement sensor is omitted, it is preferable to calculate the 3D coordinate data corresponding to the object based on multiple images taken of the object from multiple different positions using multiple cameras. In this case, a known 3D measurement method using stereo can be used.

[0042] The display unit 5 is, for example, a display device having a touch panel, which displays images (2D and 3D) of the subject captured by the imaging unit 4 and also accepts input such as operation instructions from the operator. The display unit 5 may be equipped on the terminal device 3 as, for example, a display having a touch panel.

[0043] Figure 2 illustrates the functional configuration included in the control unit 31 of the terminal device 3. The control unit 31 of the terminal device 3 includes, for example, a display control unit 311, an acquisition unit 312, a stringing unit 313, and a welding line detection unit 314.

[0044] The display control unit 311 acquires two-dimensional image data captured by the imaging unit 4 and displays the image (hereinafter also referred to as the "captured image") from the two-dimensional image data on the screen of the display unit 5. The two-dimensional image data may include the workpiece, the manipulator 2, or a virtual model of the manipulator 2.

[0045] When including the manipulator 2 in 2D image data, it is preferable to attach a marker to a specific location on the manipulator 2 (for example, near the member to which the welding torch is attached). In this case, the marker attached to the specific location on the manipulator 2 is managed so that it can be identified by the coordinates of the robot coordinate system. This makes it possible to identify the positions of the manipulator 2 and the workpiece displayed on the screen along with the marker as positions in the robot coordinate system, based on the position of the marker. The robot coordinate system can be any coordinate system that the manipulator 2, being a robot, can recognize. Furthermore, it is preferable to use an AR marker as the marker.

[0046] When including a virtual model of manipulator 2 in 2D image data, it is preferable to place a marker near the workpiece and display the virtual model of manipulator 2 based on that marker. In this case, the marker placed near the workpiece is managed so that it can be identified by the coordinates of the robot coordinate system. This makes it possible to identify the positions of the virtual model and workpiece displayed on the screen along with the marker as positions in the robot coordinate system based on the positions of the markers.

[0047] The acquisition unit 312 acquires 3D data corresponding to an object contained in the 2D image data. Preferably, the acquisition unit 312 acquires 3D data corresponding to at least the workpiece. The 3D data is point cloud data measured by the distance measurement unit, and each point cloud is represented by 3D coordinates. The 3D data may also be CAD data of the workpiece and workpiece stand.

[0048] The linking unit 313 links the 2D image data and the 3D point cloud data as data in the user coordinate system. As the user coordinate system, for example, a 3D coordinate system can be used with the origin set at a specific position of a marker included in the 2D image data. As described above, the marker's positional relationship with the manipulator 2 is fixed and managed so that it can be identified by the coordinates of the robot coordinate system. The specific position of the marker may be, for example, a corner or the center of the marker.

[0049] When linking 2D image data with 3D point cloud data, the 3D point cloud data may be displayed over the captured image shown on the screen, and the operator may be asked to specify the point in the point cloud data that corresponds to a specific position of the marker.

[0050] Furthermore, the origin of the user coordinate system is not limited to a specific position of the marker. For example, it may be a specific location on the workpiece whose positional relationship with the manipulator 2 is fixed and which is managed so as to be identifiable by the coordinates of the robot coordinate system, or a specific location of the imaging unit 4 (for example, the center of the lens).

[0051] The welding line detection unit 314 detects the welding line of a workpiece based on three-dimensional point cloud data represented by the user coordinate system. Specifically, the welding line detection unit 314 recognizes multiple planes corresponding to the workpiece based on the three-dimensional point cloud data, and detects the intersection of two planes included in those multiple planes as the welding line. If there are multiple combinations of two planes, a welding line is detected for each combination.

[0052] The welding line detection unit 314 may also detect a line as a welding line if it is traced by an operator on the captured image displayed on the display unit 5.

[0053] The display control unit 311 overlays the weld lines detected by the weld line detection unit 314 onto the captured image. This will be explained in detail with reference to Figure 3.

[0054] Figure 3 shows an example of the image display screen Sa displayed on the display unit 5. The image display screen Sa displays the captured image acquired from the imaging unit 4. The captured image includes a rectangular plate-shaped workpiece Wa placed on the work table B, a T-shaped workpiece Wb and marker M placed on top of workpiece Wa, and the welding torch T of the manipulator 2. The welding torch T may be a virtual model or an actual welding torch.

[0055] On the captured image, the four welding lines La to Ld detected by the welding line detection unit 314 are displayed as dotted lines. For the sake of clarity, each welding line La to Ld is displayed at a position away from the actual welding location. In addition, welding lines Lb and Lc are welding lines that are folded back and welded at the same location, so in reality both lines are at the same position, but for the sake of clarity, they are displayed separately.

[0056] The display control unit 311 overlays a virtual bead image, which simulates the appearance of a weld bead formed by welding, onto the captured image, along the weld line detected by the weld line detection unit 314. This will be explained in detail with reference to Figure 4.

[0057] Figure 4 illustrates a state in which virtual bead images Va to Vc, which mimic the appearance of a weld bead, are superimposed on the weld lines La to Ld shown in the image display screen Sa of Figure 3.

[0058] The display control unit 311 does not display a virtual weld bead image of the portion of the weld line detected by the weld line detection unit 314 that is hidden from view by the workpiece. This allows the operator to confirm the weld bead formed by the welding work with the same feeling as seeing it in person at the site.

[0059] Furthermore, the display control unit 311 may display the weld lines, for example with a dotted line, to indicate the presence of weld lines in the parts of the weld lines detected by the weld line detection unit 314 that are hidden from view by the workpiece. This makes it possible for the operator to recognize that weld beads are formed even in the parts hidden from view by the workpiece. A specific explanation will be given with reference to Figure 4.

[0060] In Figure 4, the virtual bead image Va of the weld bead does not show the portion of the weld line La that is hidden by the workpiece Wb. Instead, a dotted line is displayed in the portion hidden by the workpiece Wb to indicate the presence of the weld line La.

[0061] Furthermore, the display of the area hidden behind the workpiece Wb is not limited to a dotted line indicating the presence of the weld line La. For example, the virtual bead image Va could be displayed semi-transparently. In other words, it is sufficient to change the display of the virtual bead image Va between the area hidden behind the workpiece and the area that is visible, so that the operator can recognize the difference.

[0062] The timing for displaying the virtual bead images Va~Vc of the weld bead can be set as appropriate. For example, it may be at the timing when the detected weld line appears on the captured image, or it may be at the timing when the user presses the button to display the virtual bead image after the weld line has appeared on the captured image.

[0063] Furthermore, when an operator selects one of the weld lines displayed on the captured image, a virtual weld bead image corresponding to the selected weld line may be displayed. This allows the operator to check the appearance of the weld bead formed by the welding work for each selected weld line.

[0064] Furthermore, when simulating welding operations on a weld line using a virtual model of manipulator 2, the virtual bead images of the weld bead may be displayed sequentially in accordance with the movement of the virtual welding torch tip of the virtual model. This allows the operator to virtually observe how the weld bead is formed during the welding process.

[0065] The display control unit 311 may change the appearance of the weld bead represented by the virtual bead image at any time based on, for example, the settings in the welding condition setting screen Sb in Figure 5 or the settings adjusted in the welding parameter adjustment screen Sc in Figure 6. The welding condition setting screen Sb and the welding parameter adjustment screen Sc may be displayed overlaid at any location within the image display screen Sa, or they may be displayed as separate screens alongside the image display screen Sa. The welding condition setting screen Sb and the welding parameter adjustment screen Sc will be described below.

[0066] The welding condition setting screen Sb in Figure 5 is a screen for setting welding conditions. The welding condition setting screen Sb displays, as an example, a selection field Sba for selecting the welding mode, an input field Sbb for entering the bead width, a display field Sbc for displaying the welding current, a display field Sbd for displaying the welding voltage, and an input field Sbe for entering the welding speed. The welding mode is distinguished by the settings for each item, such as welding method, gas, wire material, wire diameter, and application.

[0067] Specifically, by selecting the welding mode in the selection field Sba, entering the bead width in the input field Sbb, entering the welding speed in the input field Sbe, and pressing the automatic calculation button Sbf, the welding current and welding voltage calculated based on the entered information will be displayed in the display fields Sbc and Sbd. In addition, the recommended display field Sbg will show the recommended values ​​for the torch angle and wire overhang length, along with a diagram illustrating the positional relationship between the workpiece and the welding torch from their respective sides.

[0068] The appearance of the weld bead displayed on the image display screen Sa in Figure 4 changes according to the welding conditions set on the welding condition setting screen Sb in Figure 5. This allows the operator to set the welding conditions and simultaneously check how the appearance of the weld bead changes according to those settings.

[0069] The welding parameter adjustment screen Sc in Figure 6 is a screen for adjusting the values ​​of welding parameters. The welding parameter adjustment screen Sc includes, for example, slider bars Sca for increasing or decreasing the values ​​of each welding parameter: leg length, penetration, weld bead, arc, and arc length. It is preferable to use parameters that affect the welding conditions when welding the weld line as the welding parameters to be adjusted.

[0070] For example, when an operator adjusts the leg length slider bar Sca to lengthen or shorten the leg length of the weld bead, the appearance of the weld bead displayed on the image display screen Sa changes accordingly. This allows the operator to adjust the welding parameter values ​​using the slider bar Sca and simultaneously check how the appearance of the weld bead changes as a result of those adjustments.

[0071] Furthermore, the welding condition setting screen Sb in Figure 5 and the welding parameter adjustment screen Sc in Figure 6 may be displayed with values ​​pre-entered for the selected welding line when one of the welding lines displayed on the image display screen Sa in Figure 4 is selected. This allows the operator to set or adjust the welding conditions and welding parameter values ​​for each selected welding line based on the currently set values.

[0072] Referring to Figure 7, an example of the operation of the robot system 100 according to this embodiment will be described.

[0073] First, the display control unit 311 of the terminal device 3 displays the captured image acquired from the imaging unit 4 on the screen of the display unit 5 (step S101).

[0074] Next, the display control unit 311 of the terminal device 3 displays the weld lines detected by the weld line detection unit 314 overlaid on the captured image (step S102).

[0075] Next, the display control unit 311 of the terminal device 3 overlays a virtual bead image of the weld bead formed based on the welding conditions set on the welding condition setting screen Sb onto the captured image (step S103).

[0076] As described above, according to the robot system 100 of the embodiment, an image including the workpiece captured by the imaging unit 4 can be displayed on the display unit 5, the welding line of the workpiece can be detected, and a virtual bead image that mimics the appearance of the welding bead formed by the welding operation can be superimposed on the image including the workpiece along the detected welding line.

[0077] Therefore, according to the robot system 100 of this embodiment, it becomes easier to visualize how the weld bead is formed.

[0078] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented in various other forms without departing from the spirit of the invention. For this reason, the above embodiments are merely illustrative in all respects and should not be interpreted restrictively. For example, the order of each processing step described above can be arbitrarily changed or executed in parallel, as long as there is no inconsistency in the processing content.

[0079] Furthermore, in the embodiment described above, the display control unit 311 acquires two-dimensional image data captured by the imaging unit 4 and displays it on the screen of the display unit 5. However, the display on the screen of the display unit 5 is not limited to two-dimensional image data captured by the imaging unit 4. For example, a three-dimensional computer graphics image may be created based on the two-dimensional image data, and this three-dimensional computer graphics image may be displayed on the screen of the display unit 5. When generating a three-dimensional computer graphics image from two-dimensional image data, for example, a generation AI may be used. In other words, the display on the screen of the display unit 5 may be an image based on imaging information including two-dimensional image data and three-dimensional data captured by the imaging unit 4.

[0080] Furthermore, when identifying a position on the screen as a position in the robot coordinate system, feature points with distinctive shapes or characteristics may be used instead of the markers used in the embodiments described above. In this case, for example, feature points on the screen may be identified by searching for and extracting portions of 3D point cloud data that match point cloud data representing the shape of the feature points.

[0081] Furthermore, in the embodiments described above, the origin of the user coordinate system is set to a specific position of the marker, a specific location on the workpiece, or a specific position on the imaging unit 4, but it is not limited to these. For example, a feature point with distinctive shape or other characteristics may be set as the origin of the user coordinate system. In other words, the origin of the user coordinate system can be set to the position of a feature point included in either the 2D image data or the 3D data, or to a specific position on the imaging unit 4. [Explanation of Symbols]

[0082] 1...Robot control device, 2...Manipulator, 3...Terminal device, 4...Imaging unit, 5...Display unit, 11...Control unit, 12...Storage unit, 13...Communication unit, 31...Control unit, 32...Storage unit, 33...Communication unit, 100...Robot system, 311...Display control unit, 312...Acquisition unit, 313...Linking unit, 314...Weld line detection unit, B...Work table, La~Ld...Weld line, M...Marker, Sa...Image display screen, Sb...Welding condition setting screen, Sc...Welding parameter adjustment screen, T...Welding torch, Va~Vc...Virtual bead image, Wa,Wb...Workpiece

Claims

1. A display control unit that displays an image on a display unit based on imaging information captured by an imaging unit, which includes at least the object to be welded by the welding robot, and A welding line detection unit for detecting the welding line of the target to be welded, included in the imaging information, Equipped with, The display control unit overlays a virtual bead image, which mimics the appearance of the weld bead formed by the welding operation along the detected weld line, onto the image obtained from the captured information. Robot system.

2. The display control unit, when simulating the welding operation on the weld line, sequentially displays the virtual bead image, which is to be displayed along the detected weld line, in accordance with the movement of the virtual welding torch displayed on the display unit. The robot system according to claim 1.

3. The display control unit changes the display of the virtual bead image based on the portion of the detected weld line that is hidden and invisible in the shadow of the object to be welded, and the portion that is visible. The robot system according to claim 1.

4. The display control unit displays the portion of the detected weld line that is hidden from view by the object being welded as a dotted line. The robot system according to claim 3.

5. The display control unit overlays the detected weld lines onto the image obtained from the imaging information, and when one of the displayed weld lines is selected, it displays the virtual bead image along the selected weld line. The robot system according to claim 1.

6. The display control unit displays the detected weld line superimposed on the image obtained from the imaging information, further displays a setting screen on the display unit for setting welding conditions when performing welding on the displayed weld line, and displays the virtual bead image of the weld bead formed based on the welding conditions set on the setting screen superimposed on the image obtained from the imaging information. The robot system according to claim 1.

7. The display control unit displays the detected weld lines overlaid on the image obtained from the imaging information, and when one of the displayed weld lines is selected, it further displays an adjustment screen on the display unit for adjusting welding parameters that affect the welding conditions when welding the selected weld line. The robot system according to claim 1.

8. The display control unit changes the shape of the virtual bead image in accordance with the values ​​of the welding parameters adjusted on the adjustment screen. The robot system according to claim 7.

Images