robot systems

Abstract

The robot can be easily operated exactly as the operator envisions. [Solution] The robot system comprises: a display control unit that displays image data on a screen, which is imaged by an imaging unit and includes at least the work object that the industrial robot will work on; an acquisition unit that acquires at least 3D data corresponding to the work object; a linking unit that links the image data and the 3D data as data in the user coordinate system; a designation acceptance unit that accepts the designation of the target position to which the industrial robot's work tool unit will move on the screen; a coordinate transformation unit that converts the target position in the user coordinate system to the target position in the robot coordinate system with a specific point on the industrial robot as the origin; a trajectory calculation unit that calculates the trajectory from the current position of the work tool unit to the target position; and an operation control unit that moves the work tool unit to the target position according to the calculated trajectory.

Description

Technical Field

[0001] The present invention relates to a robot system.

Background Art

[0002] Patent Document 1 below discloses a system for teaching the operation of a robot in a real environment by simulating a robot model in a virtual space. In this system, the operation of the robot is taught using teaching data created by executing offline teaching using the robot model in the virtual space. Offline teaching is a process of displaying a robot model and a target model indicating the shape of an object on a virtual space and setting teaching points according to user input. The teaching data is data that describes a plurality of teaching points in chronological order.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, the system of Patent Document 1 operates the robot in the real environment so as to approach the position and orientation of the robot model included in the simulation image when the robot model is simulated. Further, the robot model and the target model used in the simulation are created from CAD data. However, since the size and arrangement of the robot and the work object in the real environment may not be the same as those in the CAD data, positional deviation and the like are likely to occur, and it is difficult to operate the robot in the real environment as the operator imagines (operate the robot in a form that does not conflict with the real environment and reach the target position). Also, since it is difficult for the operator to grasp the direction of the robot coordinate system from the operator's viewpoint, it is difficult to make the direction in which the operator operates the robot on the system coincide with the direction in which the robot operates in the real environment.

[0005] Therefore, the present invention aims to provide a robot system that allows the robot to be easily operated according to the operator's vision. [Means for solving the problem]

[0006] A robot system according to one aspect of the present invention includes: a display control unit that displays image data on a screen, which is imaged by an imaging unit and includes at least a work object to be worked on by the industrial robot; an acquisition unit that acquires at least three-dimensional data corresponding to the work object; a linking unit that links the image data and the three-dimensional data as data in a user coordinate system with the origin being the position of a feature point included in either the image data or the three-dimensional data, or the imaging position; a designation reception unit that accepts the designation of a target position to which the industrial robot's work tool unit will move on a screen; a coordinate transformation unit that converts the target position in the user coordinate system to a target position in a robot coordinate system with the origin being a specific point on the industrial robot; a trajectory calculation unit that calculates the trajectory from the current position of the work tool unit to the target position; and an operation control unit that moves the work tool unit to the target position according to the calculated trajectory.

[0007] According to this embodiment, image data including the object to be worked on by the industrial robot is displayed on the screen, the image data and the 3D data corresponding to the object to be worked on are linked as data in the user coordinate system, the target position to which the work tool unit will move is specified on the screen, the target position in the user coordinate system is converted to the target position in the robot coordinate system, the trajectory from the current position of the work tool unit to the target position is calculated, and the work tool unit can be moved to the target position according to the calculated trajectory.

[0008] This allows the user to specify a target location on the screen, thereby moving the work tool to that specified location.

[0009] In the above embodiment, the 3D data may be 3D point cloud data corresponding to an object included in the image data measured by the distance measuring unit.

[0010] According to this embodiment, it becomes possible to represent objects within the measurement range of the distance measuring unit as three-dimensional point cloud data.

[0011] In the above embodiment, the designated reception unit does not need to accept the designation of a target location at locations where point cloud data does not exist.

[0012] According to this embodiment, it becomes possible to limit the destination of the work tool to the location where an object such as the work object exists.

[0013] In the above embodiment, the user coordinate system is a three-dimensional coordinate system with the origin at a specific position of a marker included in the image data, and the positional relationship of the marker with respect to the industrial robot may be specified.

[0014] According to this embodiment, even when the imaging unit moves, it is possible to identify the target position as coordinates in a user coordinate system based on a marker whose positional relationship with the industrial robot is fixed.

[0015] In the above embodiment, when the operation control unit is moving the work tool unit to the target position, and the designation reception unit specifies a target position different from the target position, the trajectory calculation unit may calculate the trajectory from the current position of the work tool unit to the other target position as the modified trajectory, and the operation control unit may move the work tool unit to the other target position according to the calculated modified trajectory.

[0016] According to this embodiment, it becomes possible to flexibly change the trajectory to the target position by specifying other target positions.

[0017] In the above embodiment, the designation reception unit may further accept the designation of a relative position for adjusting the target position, and the trajectory calculation unit may calculate the trajectory from the current position of the work tool unit to the target position after adjustment by the relative position.

[0018] According to this aspect, it becomes possible to easily perform fine adjustment of the target position by specifying the relative position.

Advantages of the Invention

[0019] According to the present invention, it is possible to provide a robot system that can easily operate a robot as the operator imagines.

Brief Description of the Drawings

[0020] [Figure 1] It is a diagram illustrating the configuration of a robot system according to an embodiment. [Figure 2] It is a diagram illustrating a functional configuration of the control unit of the arithmetic unit shown in FIG. 1. [Figure 3] It is an example of a screen displayed on the display unit. [Figure 4] It is an example of a screen displayed on the display unit. [Figure 5] It is a flowchart for explaining an example of the operation of the robot system.

Modes for Carrying Out the Invention

[0021] Preferred embodiments of the present invention will be described with reference to the accompanying drawings. In each figure, those denoted by the same reference numerals have the same or similar configurations. Also, since the drawings are schematic, the dimensions and ratios of each component are different from the actual ones.

[0022] FIG. 1 is a diagram illustrating the configuration of a robot system 100 according to an embodiment. The robot system 100 includes, for example, a robot control device 1, a manipulator (industrial robot) 2, an arithmetic unit 3, an imaging unit 4, and a display unit 5. Note that the arithmetic unit 3 may be included in the robot control device 1.

[0023] The robot control device 1 and the computing device 3, the robot control device 1 and the manipulator 2, and the computing device 3, the imaging unit 4, and the display unit 5 are all connected via a network. The network may be wireless communication such as WiFi (Wireless Fidelity) or wired communication such as a communication cable.

[0024] Manipulator 2 is a welding robot that performs arc welding on a workpiece according to the working conditions set in the robot control device 1. Manipulator 2 has, for example, a multi-joint arm mounted on a base member fixed to the floor of a factory, and a welding torch (work tool part) connected to the tip of the multi-joint arm as one of the work tools.

[0025] Manipulator 2 is a collaborative robot designed for working alongside humans and can operate in response to external forces. Workers can directly apply external forces to Manipulator 2 with their hands, moving it in response to those forces and directly controlling it.

[0026] The robot control device 1 is a control unit that controls the operation of the manipulator 2. The robot control device 1 includes, for example, a control unit 11, a storage unit 12, and a communication unit 13.

[0027] The control unit 11 is a processor that controls the manipulator 2 by executing work programs and the like stored in the memory unit 12.

[0028] The memory unit 12 is a computer-readable recording medium that stores programs for realizing various functions of the robot control device 1, as well as various data used in those programs.

[0029] The communication unit 13 is a communication interface that controls communication with the manipulator 2 and the computing unit 3, which are connected via the network.

[0030] The robot control device 1 may further include a welding power supply unit. The welding power supply unit supplies welding current, welding voltage, etc., to the manipulator 2 according to predetermined welding conditions in order to generate an arc between the tip of the welding wire and the workpiece. The welding power supply unit may be provided separately from the robot control device 1.

[0031] The computing device 3 is a device that performs processing when an operator teaches the operation of the manipulator 2, and is, for example, a tablet-type teach pendant. Some or all of the functions of the computing device 3, which will be described later, may be provided in the robot control device 1, or they may be provided in a server device on the cloud. The computing device 3 includes, for example, a control unit 31, a storage unit 32, and a communication unit 33.

[0032] The control unit 31 is a processor that controls each part of the arithmetic unit 3 by executing programs stored in the memory unit 32. The functions of the control unit 31 will be described later.

[0033] The memory unit 32 is a computer-readable recording medium that stores programs for realizing various functions of the arithmetic unit 3, as well as various data used by those programs.

[0034] The communication unit 33 is a communication interface and controls communication with the robot control device 1, which is connected via a network.

[0035] The imaging unit 4 is, for example, a 3D camera equipped with a distance measuring sensor, and also functions as a 2D camera. It is preferable to position the imaging unit 4 in a location that can capture at least the workspace including the workpiece. For example, the imaging unit 4 may be attached to the manipulator 2 near the member to which the welding torch is attached, or it may be fixedly positioned around the manipulator 2 or in the robot cell. Alternatively, the imaging unit 4 may be provided in the computing unit 3.

[0036] A distance measuring sensor is a sensor capable of measuring the distance to an object. Examples of distance measuring sensors that can be used include LiDAR (Light Detection and Ranging) sensors, millimeter-wave sensors, and ultrasonic sensors.

[0037] The imaging unit 4 does not necessarily need to be equipped with a distance measurement sensor; the imaging unit 4 and the distance measurement sensor may be provided separately, or the distance measurement sensor may be omitted. If the distance measurement sensor is omitted, it is preferable to calculate the 3D coordinate data corresponding to the object based on multiple images of the object taken from multiple different positions. In this case, a known 3D measurement method using stereo can be used.

[0038] The display unit 5 is, for example, a display device having a touch panel, which displays images (2D and 3D) of the subject captured by the imaging unit 4 and accepts input such as operation instructions from the operator. The display unit 5 may be provided in the arithmetic unit 3 as, for example, a display having a touch panel.

[0039] Figure 2 illustrates the functional configuration of the control unit 31 of the arithmetic unit 3. The control unit 31 of the arithmetic unit 3 has, for example, a display control unit 311, an acquisition unit 312, a linking unit 313, a designation reception unit 314, a coordinate transformation unit 315, a trajectory calculation unit 316, and an operation control unit 317.

[0040] The display control unit 311 acquires two-dimensional image data captured by the imaging unit 4 and displays it on the screen of the display unit 5. Preferably, the two-dimensional image data includes the workpiece and the manipulator 2.

[0041] Here, a marker may be attached to a specific location on the manipulator 2 (for example, near the component to which the welding torch is attached). In this case, it is preferable to manage the attached marker so that it can be identified in the robot coordinate system. This makes it possible to identify the positions of the manipulator 2 and the workpiece displayed on the screen along with the marker as the robot coordinate system position based on the marker's position. The robot coordinate system can be any coordinate system that the manipulator 2, being a robot, can recognize. Furthermore, it is preferable to use an AR marker as the marker.

[0042] The acquisition unit 312 acquires at least three-dimensional data corresponding to the workpiece. The three-dimensional data is point cloud data measured by the distance measurement unit, and is three-dimensional coordinate data corresponding to an object included in the two-dimensional image data. The three-dimensional data may also be CAD data of the workpiece and workpiece stand.

[0043] The linking unit 313 links the 2D image data and the 3D data as data in the user coordinate system. As the user coordinate system, for example, a 3D coordinate system can be used with the origin set at a specific position of a marker included in the 2D image data. It is preferable that the marker is managed so that its positional relationship with the manipulator 2 is fixed and it can be identified as a coordinate in the robot coordinate system. The specific position of the marker may be, for example, a corner or the center of the marker.

[0044] When linking 2D image data with 3D data, the 3D point cloud data may be displayed over the 2D image data shown on the screen, and the operator may be asked to specify the point in the point cloud data that corresponds to a specific position of the marker.

[0045] Furthermore, the origin of the user coordinate system is not limited to a specific position of the marker. For example, it may be a specific location on the workpiece whose positional relationship with the manipulator 2 is fixed and managed so that it can be identified as a coordinate in the robot coordinate system, or a specific location of the imaging unit 4 (for example, the imaging position such as the center of the lens).

[0046] The designation reception unit 314 accepts the designation of the target position to which the tip of the welding torch will move on the screen. This will be explained in detail with reference to Figure 3.

[0047] Figure 3 shows an example of screen 5A displayed on the display unit 5. Screen 5A displays two-dimensional image data captured by the imaging unit 4. This screen 5A displays workpieces W placed on multiple worktables B arranged in a row, the welding torch T of the manipulator 2, and a marker M placed on the worktable B in front of the workpiece W. The welding torch T may be a virtual model or an actual welding torch. When the operator touches and specifies the target position P to which the tip of the welding torch T will move on screen 5A, the specification reception unit 314 accepts the specification of the target position P.

[0048] The designated reception unit 314 may refuse to accept the designation of a target position P if point cloud data does not exist or if the position is outside the reach of the welding torch T. In such cases, it may output an error message or an error sound indicating that the position cannot be specified.

[0049] Furthermore, the designation reception unit 314 may also accept the designation of a relative position for adjusting the target position P. Specifically, it may accept input designations of relative positions such as ○ mm in the X direction, ○ mm in the Y direction, and ○ mm in the Z direction, and adjust the target position P.

[0050] The coordinate transformation unit 315 transforms the coordinates of the target position P in the user coordinate system to the coordinates of the target position P in the robot coordinate system (coordinate transformation). The target position P displayed on the screen is part of the 2D image data drawn in the user coordinate system, and the marker M included in the 2D image data is managed so that it can be identified as coordinates in the robot coordinate system. Therefore, the target position P included in the 2D image data can also be identified as coordinates in the robot coordinate system.

[0051] The trajectory calculation unit 316 calculates the trajectory from the current position of the tip of the welding torch T to the target position P. When calculating the trajectory, it is preferable to calculate a trajectory that allows the welding torch T to avoid collision with obstacles, based on two-dimensional image data and three-dimensional data. If it is not possible to calculate a trajectory that can reach the target position P, an error message or error sound indicating that the trajectory cannot be calculated may be output.

[0052] The motion control unit 317 moves the tip of the welding torch T to the target position P according to the calculated trajectory. The motion control unit 317 terminates the movement of the welding torch T when the current position of the tip of the welding torch T coincides with the target position P, or when the difference between the current position of the tip of the welding torch T and the target position P falls below a predetermined threshold. This will be explained in detail with reference to Figure 4.

[0053] Figure 4 illustrates the state after the tip of the welding torch T has moved to the target position P after the target position P has been specified on screen 5A in Figure 3.

[0054] The trigger for manipulator 2 to begin moving to target position P may be, for example, when the operator presses the position selection start button and then specifies the target position P on screen 5A, or when the operator presses the move start button while specifying the target position P.

[0055] The trajectory calculation unit 316 calculates the trajectory from the current position where the tip of the welding torch T is located at that time to the other target position, if the designation reception unit 314 specifies a target position other than the target position P while the operation control unit 317 is moving the tip of the welding torch T to the target position, as the modified trajectory.

[0056] The motion control unit 317 moves the tip of the welding torch T to the other target position according to the calculated modified trajectory.

[0057] Referring to Figure 5, an example of the operation of the robot system 100 will be described.

[0058] First, the designation reception unit 314 receives the designation of the target position P to which the tip of the welding torch T will move (step S101).

[0059] Next, the designated reception unit 314 determines whether point cloud data exists at the designated target position P (step S102). If this determination is NO (step S102; NO), an error notification is output (step S109) and this operation is terminated.

[0060] If the determination in step S102 above determines that point cloud data exists at the specified target position P (step S102; YES), the coordinate transformation unit 315 transforms the coordinates of the target position P in the user coordinate system to the coordinates of the target position P in the robot coordinate system (step S103).

[0061] Next, the trajectory calculation unit 316 calculates the trajectory from the current position of the tip of the welding torch T to the target position P (step S104).

[0062] Next, the trajectory calculation unit 316 determines whether it has been able to calculate a trajectory that can reach the specified target position P (step S105). If this determination is NO (step S105; NO), it outputs an error notification (step S109) and terminates this operation.

[0063] If the determination in step S105 above determines that a trajectory capable of reaching the specified target position P has been calculated (step S105; YES), the operation control unit 317 moves the tip of the welding torch T according to the trajectory calculated in step S104 (step S106).

[0064] Next, the designation reception unit 314 determines whether a target position other than target position P has been designated (step S107). If this determination is YES (step S107; YES), the process proceeds to step S102 described above.

[0065] If the determination in step S107 above determines that no other target position has been specified (step S107; NO), the operation control unit 317 determines whether the tip of the welding torch T has reached the target position P based on the current position of the tip of the welding torch T and the target position P (step S108). If this determination is NO (step S108; NO), the process proceeds to step S106 described above.

[0066] If the determination in step S108 above indicates that the tip of the welding torch T has reached the target position P (step S108; YES), this operation is terminated.

[0067] As described above, according to the robot system 100 of the embodiment, two-dimensional image data including the welding torch T (virtual model or actual welding torch) and workpiece W of the manipulator 2 is displayed on the screen, the image data and three-dimensional data corresponding to the workpiece W are linked as data in the user coordinate system, the user specifies the target position P to which the tip of the welding torch T will move on the screen, the target position P in the user coordinate system is converted to the target position P in the robot coordinate system, the trajectory from the current position of the tip of the welding torch T to the target position P is calculated, and the tip of the welding torch T can be moved to the target position P according to the calculated trajectory.

[0068] This allows the operator to specify a target position P on the screen, thereby moving the tip of the welding torch T to the specified target position P. In other words, it becomes possible to move the tip of the welding torch T to reach the target position P in a way that does not deviate from the actual environment.

[0069] Therefore, according to the robot system 100 of this embodiment, it is possible to easily operate the manipulator 2 exactly as the operator envisions.

[0070] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and can be implemented in various other forms without departing from the spirit of the invention. For this reason, the above embodiments are merely illustrative in all respects and should not be interpreted restrictively. For example, the order of each processing step described above can be arbitrarily changed or executed in parallel, as long as there is no inconsistency in the processing content.

[0071] Furthermore, in the embodiment described above, the display control unit 311 acquires two-dimensional image data captured by the imaging unit 4 and displays it on the screen of the display unit 5. However, the display on the screen of the display unit 5 is not limited to two-dimensional image data captured by the imaging unit 4. For example, a three-dimensional computer graphics image may be created based on the two-dimensional image data, and this three-dimensional computer graphics image may be displayed on the screen of the display unit 5. When generating a three-dimensional computer graphics image from two-dimensional image data, for example, a generation AI may be used. Also, the display on the screen of the display unit 5 may be image data from imaging information that includes three-dimensional data captured by the imaging unit 4.

[0072] Furthermore, in the embodiment described above, the display control unit 311 displays the manipulator 2 captured by the imaging unit 4 on the screen of the display unit 5. However, instead of the manipulator 2, a virtual model of the manipulator 2 may be displayed on the screen of the display unit 5. In this case, it is preferable to place a marker near the workpiece and display the virtual model of the manipulator 2 based on that marker. The positions of the virtual model and the workpiece displayed on the screen along with the marker in the robot coordinate system can be determined based on the marker as follows.

[0073] (1) The workpiece is photographed, including the markers nearby, and the 2D image data and 3D point cloud data are linked in the same manner as in the embodiment described above. (2) The operator inputs the position and orientation in the robot coordinate system of the tip of the welding torch, which has been moved to a specific position on the marker (e.g., the corner or center of the marker). This makes it possible to identify the position of the photographed marker in terms of the coordinates of the robot coordinate system. (3) The position in the robot coordinate system of the point cloud data linked to the image data in (1) above is calculated, and the position in the robot coordinate system of the photographed workpiece is also calculated. This makes it possible to identify the position on the screen as the position in the robot coordinate system, and thus it becomes possible to display a virtual model of the manipulator 2 on the screen.

[0074] Since the length of the links of manipulator 2 and the length of the welding torch are managed as robot information, the tip of the welding torch can be calculated using this robot information.

[0075] Furthermore, when identifying a position on the screen as a position in the robot coordinate system, feature points with distinctive shapes or characteristics may be used instead of the markers used in the embodiments and modifications described above. In that case, for example, the feature points displayed on the screen may be identified by searching for and extracting a portion of the 3D point cloud data that matches the point cloud data representing the shape of the feature points.

[0076] Furthermore, in the embodiments described above, the origin of the user coordinate system is set to a specific position of the marker, a specific location on the workpiece, or a specific position (imaging position) of the imaging unit 4, but it is not limited to these. For example, a feature point with distinctive shape or other characteristics may be set as the origin of the user coordinate system. In other words, the origin of the user coordinate system can be set to the position of a feature point included in either the 2D image data or the 3D data, or to the imaging position.

[0077] Furthermore, although the above-described embodiment used a welding robot manipulator 2, the present invention is not limited to this. The present invention can be applied to industrial robots, including handling robots that perform picking and other operations, and cutting and polishing robots that perform cutting and polishing operations. [Explanation of symbols]

[0078] 1...Robot control device, 2...Manipulator, 3...Calculation unit, 4...Imaging unit, 5...Display unit, 11...Control unit, 12...Storage unit, 13...Communication unit, 31...Control unit, 32...Storage unit, 33...Communication unit, 100...Robot system, 311...Display control unit, 312...Acquisition unit, 313...Linking unit, 314...Designation reception unit, 315...Coordinate transformation unit, 316...Trajectory calculation unit, 317...Motion control unit, B...Workpiece table, M...Marker, P...Target position, T...Welding torch, W...Workpiece

Claims

1. A display control unit that displays image data on a screen, which is imaged by an imaging unit and includes at least the work object that the industrial robot is working on, An acquisition unit that acquires at least three-dimensional data corresponding to the aforementioned work target, A linking unit that links the image data and the three-dimensional data as data in a user coordinate system whose origin is the position of a feature point included in either the image data or the three-dimensional data, or the imaging position. A designation reception unit that accepts the designation of the target position to which the work tool of the industrial robot will move, on a screen, A coordinate transformation unit that transforms the target position in the user coordinate system into the target position in a robot coordinate system with a specific point on the industrial robot as the origin, A trajectory calculation unit that calculates the trajectory from the current position of the work tool unit to the target position, An motion control unit that moves the work tool unit to the target position according to the calculated trajectory, A robotic system equipped with the following features.

2. The aforementioned three-dimensional data is three-dimensional point cloud data corresponding to an object included in the image data measured by the distance measurement unit. The robot system according to claim 1.

3. The designated reception unit will not accept the designation of the target position at locations where point cloud data does not exist. The robot system according to claim 2.

4. The user coordinate system is a three-dimensional coordinate system whose origin is a specific position of a marker included in the image data. The positional relationship of the marker with respect to the industrial robot is specified. The robot system according to claim 1.

5. When the operation control unit is moving the work tool unit to the target position, and the designation reception unit specifies a target position different from the target position, the trajectory calculation unit calculates the trajectory from the current position of the work tool unit to the other target position as the modified trajectory. The motion control unit moves the work tool unit to the other target position according to the calculated modified trajectory. The robot system according to claim 1.

6. The aforementioned designation receiving unit further receives the designation of a relative position for adjusting the target position, The trajectory calculation unit calculates the trajectory from the current position of the work tool unit to the target position after adjustment based on the relative position. The robot system according to claim 1.

Images