ROBOT SYSTEM AND METHOD FOR CONTROLLING ROBOT SYSTEM

Abstract

A robot system is provided that enables a user to accurately recognize an imaging range of a workpiece imaged by an imaging unit. [Solution] This appearance inspection system 100 includes a robot arm 12, an imaging unit 20 which includes an irradiation unit 21 which irradiates irradiation light onto a workpiece 200 and images the workpiece 200 irradiated with the irradiation light, a display unit 45 and a display unit 53 on which the workpiece 200 is displayed, and a processing unit 41 and a processing unit 51 which move the imaging unit 20 or the workpiece 200 using the robot arm 12 to acquire coordinates of the irradiation light in an image captured by the imaging unit 20 which is moved relative to the workpiece 200, and which performs processing to superimpose a virtual image A of the irradiation light generated by computer graphics onto the image of the workpiece 200 based on the acquired coordinates of the irradiation light and a control point TCP of the robot arm 12, and display the superimposed image on the display unit 45 and the display unit 53.

Description

[Technical field]

[0001] The present disclosure relates to a robot system and a control method for a robot system. [Background technology]

[0002] 2. Description of the Related Art A teaching device is known in the art (see, for example, Patent Document 1).

[0003] The above-mentioned Patent Document 1 discloses an offline teaching device for a vision sensor. This offline teaching device includes a storage device that stores the shapes and dimensions of a workpiece, a robot that performs work on the workpiece, and a vision sensor that measures the workpiece, and a display that displays images of the workpiece, the robot, and the vision sensor based on the data stored in the storage device. This offline teaching device arranges a modeled workpiece, robot, and vision sensor on the display, and teaches and adjusts the vision sensor offline. Specifically, this offline teaching device arranges the vision sensor on the display, captures an image of a measurement portion of the workpiece with the arranged vision sensor, and generates an image of the captured measurement portion of the vision sensor. This allows a user to recognize the imaging range of the workpiece imaged by the vision sensor by visually checking the image of the measurement portion imaged by the vision sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4266946 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the offline teaching device described in the above Patent Document 1, a vision sensor image of the measurement part of the workpiece is generated based on the modeled workpiece, robot, and vision sensor. Therefore, the modeled workpiece, robot, and vision sensor may contain shape errors and position errors compared to the real thing. Therefore, in the offline teaching device described in the above Patent Document 1, the user cannot accurately recognize which part of the workpiece can be imaged by the vision sensor. Therefore, it is desired that the user can accurately recognize the imageable range of the workpiece imaged by the imaging unit.

[0006] This disclosure has been made to solve the problems described above, and one objective of this disclosure is to provide a robot system and a control method for a robot system that enable a user to accurately recognize the imaging range of a workpiece imaged by an imaging unit. [Means for solving the problem]

[0007] A robot system according to a first aspect of the present disclosure includes a robot arm, an imaging unit including an irradiation unit that irradiates a workpiece with irradiation light and that images the workpiece irradiated with the irradiation light, a display unit on which the workpiece is displayed, and a display unit that displays the image of the workpiece by the robot arm. Is it okay? a processing unit that performs processing for moving the workpiece, acquiring coordinates of the irradiated light in an image captured by the imaging unit that is moved relative to the workpiece, and displaying a virtual image of the irradiated light generated by computer graphics on the image of the workpiece based on the acquired coordinates of the irradiated light and a control point of the robot arm on the display unit. The processing unit, during teaching of the operation of the robot arm, moves the imaging unit or the workpiece by the robot arm, sequentially acquires coordinates of the irradiated light in the image captured by the imaging unit which is moved relative to the workpiece, and performs processing to superimpose a virtual image of a line-shaped irradiated light, the position of which changes from moment to moment, on the image of the workpiece based on the sequentially acquired coordinates of the irradiated light, on the image of the workpiece, and to display the virtual image on the display unit. Note that the concept of "superimposing and displaying a virtual image on an image of a workpiece" includes the meaning of superimposing and displaying a virtual image on an image of a real workpiece, and the meaning of superimposing and displaying a virtual image on an image of a modeled workpiece.

[0008] In the robot system according to the first aspect of the disclosure, as described above, the robot arm moves the imaging unit or the workpiece, acquires the coordinates of the irradiated light in the image captured by the imaging unit that is moved relative to the workpiece, and includes a processing unit that performs processing to superimpose a virtual image of the irradiated light generated by computer graphics on the image of the workpiece based on the acquired coordinates of the irradiated light and the control point of the robot arm and display the superimposed image on the display unit. As a result, the virtual image of the irradiated light is displayed on the image of the workpiece based on the coordinates of the actual irradiated light and the control point of the actual robot arm. That is, since the shape and position are based on actual information, the error of the shape and position is small. Therefore, the user can visually recognize the virtual image of the irradiated light on the image of the workpiece to confirm how the irradiated light hits the workpiece, and therefore the imageable range of the workpiece captured by the imaging unit can be accurately recognized.

[0009] A control method for a robot system according to a second aspect of the present disclosure includes: moving, by a robot arm, an imaging unit including an irradiation unit that irradiates a workpiece with irradiation light and that images the workpiece irradiated with the irradiation light, or the workpiece; acquiring coordinates of the irradiation light in an image captured by the imaging unit that is moved relatively to the workpiece; and displaying, on a display unit, a virtual image of the irradiation light generated by computer graphics based on the acquired coordinates of the irradiation light and a control point of the robot arm, superimposing the virtual image on an image of the workpiece. Displaying the virtual image of the irradiation light on the display unit by superimposing it on the image of the work includes, during teaching of the operation of the robot arm, moving the imaging unit or the work by the robot arm, sequentially acquiring coordinates of the irradiation light in the image captured by the imaging unit which is moved relatively to the work, and sequentially superimposing a virtual image of a line-shaped irradiation light, the position of which changes from moment to moment, on the image of the work and displaying it on the display unit based on the sequentially acquired coordinates of the irradiation light. Note that the concept of "superimposing and displaying a virtual image on an image of a workpiece" includes the meaning of "superimposing and displaying a virtual image on an image of a real workpiece" and the meaning of "superimposing and displaying a virtual image on an image of a modeled workpiece."

[0010] In the control method of the robot system according to the second aspect of the disclosure, as described above, the coordinates of the irradiation light in the image captured by the imaging unit that is moved relative to the workpiece are acquired, and a virtual image of the irradiation light generated by computer graphics based on the acquired coordinates of the irradiation light and the control point of the robot arm is superimposed on the image of the workpiece and displayed on the display unit. As a result, the virtual image of the irradiation light is displayed on the image of the workpiece based on the coordinates of the actual irradiation light and the control point of the actual robot arm. That is, since the shape and position are based on actual information, the error of the shape and position is small. Therefore, by visually checking the virtual image of the irradiation light on the image of the workpiece, the user can confirm how the irradiation light hits the workpiece, and therefore a control method of the robot system that can accurately recognize the imageable range of the workpiece captured by the imaging unit can be provided. Effect of the Invention

[0011] According to the present disclosure, as described above, the user can accurately recognize the imageable range of the workpiece imaged by the imaging unit. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a visual inspection system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a visual inspection system according to a first embodiment. [Diagram 3] FIG. 1 is a diagram showing a wearable display device according to a first embodiment. [Figure 4] 3A and 3B are diagrams showing a virtual image displayed on a display unit of the wearable display device according to the first embodiment. [Diagram 5] 3A and 3B are diagrams showing a virtual image displayed on a display unit of the fixed display device according to the first embodiment. [Figure 6] FIG. 2 is a flow chart for explaining a control method of the visual inspection system according to the first embodiment. [Figure 7] 13A and 13B are diagrams showing a virtual image displayed on a display unit of the wearable display device according to the second embodiment. [Figure 8] FIG. 11 is a diagram showing a virtual image displayed on a display unit of a fixed display device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, specific embodiments of the present disclosure will be described with reference to the drawings.

[0014] [First embodiment] The overall configuration of the visual inspection system 100 according to the first embodiment will be described.

[0015] As shown in Fig. 1, the visual inspection system 100 is an appearance inspection system that inspects the appearance of a workpiece 200. The workpiece 200 is, for example, a product or part related to an automobile, an agricultural machine, ceramics, or a household electrical appliance. The workpiece 200 is not particularly limited. The visual inspection system 100 is also an example of a robot system.

[0016] The appearance inspection system 100 includes a robot 10, an imaging unit 20, a robot controller 30, a wearable display device 40, and a fixed display device 50.

[0017] The robot 10 is, for example, a vertical articulated robot. The robot 10 includes a base unit 11 and a robot arm 12 connected to the base unit 11. The base unit 11 is fixed to an installation surface such as a floor, a wall, or a ceiling. The base unit 11 may be attached to a movable cart. The robot arm 12 has a plurality of joints. Each of the plurality of joints has a servo motor as a driving source. In addition, the tip of the robot arm 12 holds an imaging unit 20. The robot 10 drives the plurality of joints of the robot arm 12 to move the imaging unit 20 held at the tip of the robot arm 12 relative to a fixed workpiece 200.

[0018] The imaging unit 20 is attached to the tip of the robot arm 12. As shown in FIG. 2, the imaging unit 20 includes an irradiation unit 21 that irradiates the workpiece 200 with irradiation light, and images the workpiece 200 irradiated with the irradiation light. The imaging unit 20 also includes an imaging element unit 22 that images the irradiation light irradiated by the irradiation unit 21 and the workpiece 200. For example, the imaging unit 20 is a line-type camera that is moved along the surface of the workpiece 200 by the robot 10 to scan and image the surface of the workpiece 200. The imaging unit 20 also includes a processing unit 23. The processing unit 23 performs image processing on the image captured by the imaging element unit 22. The processing unit 23 also controls the timing of imaging by the imaging unit 20.

[0019] As shown in FIG. 2, the robot controller 30 controls the operation of the robot 10. The robot controller 30 includes a processing unit 31, a storage unit 32, and a signal output unit 33. The processing unit 31 includes a processor and performs various processes related to the operation of the robot 10. The storage unit 32 includes a non-volatile memory and stores various information such as a program for operating the robot 10. The signal output unit 33 includes, for example, an FPGA (Field Programmable Gate Array) and is capable of quickly outputting the coordinates of a TCP (Tool Center Point), which is a control point of the robot 10. The signal output unit 33 is also capable of outputting a plurality of coordinates of the robot 10. For example, the signal output unit 33 is capable of outputting the coordinates of a point near the control point TCP in addition to the coordinates of the control point TCP.

[0020] As shown in FIG. 3, the wearable display device 40 displays a virtual image A generated by computer graphics by superimposing it on a real image visually recognized by the user U. That is, the wearable display device 40 is a display unit that displays mixed reality. The wearable display device 40 is worn by the user U. Specifically, the wearable display device 40 includes a goggle-type display unit that is worn on the head of the user U. As shown in FIG. 2, the wearable display device 40 includes a processing unit 41, a storage unit 42, an imaging unit 43, a sensor 44, and a display unit 45. The processing unit 41 includes a processor and performs various processes such as generating the virtual image A. The storage unit 42 includes a non-volatile memory and stores various information such as a program for generating the virtual image A. The imaging unit 43 captures a real image. The sensor 44 includes an acceleration sensor, a gyroscope, and the like. The sensor 44 detects the movement of the user U, and the like. 4, the display unit 45 is a see-through type display unit, and the user U can view real images of the robot 10, the workpiece 200, and the like through the display unit 45. In addition, the display unit 45 displays a virtual image A generated by computer graphics superimposed on the real image. The virtual image A is, for example, a hologram. The virtual image A will be described in detail later.

[0021] In the first embodiment, as shown in FIG. 5, the fixed display device 50 is fixedly disposed without being worn by the user U. The fixed display device 50 displays an image of a three-dimensional model M of the workpiece 200. The image of the three-dimensional model M of the workpiece 200 is, for example, an image of the workpiece 200 created using CAD (Computer Aided Design). As shown in FIG. 2, the fixed display device 50 includes a processing unit 51, a storage unit 52, and a display unit 53. The processing unit 51 includes a processor and performs various processes such as generating a virtual image A. The storage unit 52 includes a non-volatile memory and stores various information such as a program for generating the virtual image A. The display unit 53 is, for example, a liquid crystal display or an organic EL display.

[0022] (Function to display image capture range) The visual inspection system 100 has a function of displaying an imageable range of the workpiece 200 to be inspected. Specifically, a virtual image A of the irradiated light irradiated from the irradiation unit 21 to the workpiece 200 is generated by computer graphics, and the virtual image A of the irradiated light is displayed on the display unit 45 of the wearable display device 40 so as to be superimposed on the actual image visually recognized by the user U. Also, the virtual image A of the irradiated light is displayed on the display unit 53 of the fixed display device 50 so as to be superimposed on the image of the three-dimensional model M of the workpiece 200. Also, the function of displaying the imageable range is used, for example, when the user U teaches the robot 10 to operate in order to inspect the workpiece 200. A control method of the visual inspection system 100 for displaying the imageable range will be described in detail below.

[0023] 6, in step S1, the processing unit 31 of the robot controller 30 accepts an operation of the teaching pendant by a user U. Then, based on the accepted operation, the processing unit 31 moves the robot arm 12 to move the imaging unit 20 including the irradiation unit 21 relative to the workpiece 200.

[0024] In step S2, in the first embodiment, the processing unit 23 of the imaging unit 20 acquires coordinates of the irradiated light in an image captured by the imaging unit 20 that is moved relatively to the workpiece 200. Specifically, while the imaging unit 20 is in an on state, the processing unit 23 acquires an image captured by the imaging element unit 22.

[0025] In step S3, the processing unit 23 acquires the coordinates of the irradiated light in the image captured by the imaging unit 20 which is moved relatively to the workpiece 200. First, the processing unit 23 performs pre-processing of the acquired image.

[0026] In step S4, in the first embodiment, the processing unit 41 of the wearable display device 40 and the processing unit 51 of the fixed display device 50 each perform processing to superimpose a virtual image A of the irradiated light generated by computer graphics onto the image of the workpiece 200 based on the acquired coordinates of the irradiated light and the control point TCP of the robot arm 12, and display it on the display unit 45 of the wearable display device 40 and the display unit 53 of the fixed display device 50.

[0027] In addition, in the first embodiment, the processing unit 41 of the wearable display device 40 and the processing unit 51 of the fixed display device 50 each perform processing to superimpose a virtual image A of the irradiation light of a portion corresponding to a portion where the pixel value of the image of the irradiation light captured by the imaging unit 20 is equal to or greater than a threshold value onto the image of the workpiece 200, and display it on the wearable display device 40 and the fixed display device 50.

[0028] The processing of the processing unit 41 of the wearable display device 40 will be described in detail. As shown in FIG. 1, a marker 210 for aligning the real coordinates with the coordinates of the wearable display device 40 is arranged near the workpiece 200. The marker 210 is, for example, a QR code (registered trademark). When the user U wears the wearable display device 40 on his / her head and visually recognizes the marker 210, the imaging unit 43 of the wearable display device 40 images the marker 210. Then, the processing unit 41 of the wearable display device 40 aligns the real coordinates with the coordinates of the wearable display device 40 based on information such as the QR code (registered trademark) obtained from the marker 210 and the robot coordinate system of the robot 10. The coordinates of the real workpiece 200 are preset based on the robot coordinate system. The coordinates of the real workpiece 200 are input from the robot controller 30 to the wearable display device 40. Then, the processing unit 41 executes a process of displaying the virtual image A of the irradiated light on the wearable display device 40 so as to be superimposed on the image of the actual workpiece 200. Note that, as the position of the head of the user U moves, the position of the virtual image A of the irradiated light displayed on the display unit 45 of the wearable display device 40 also changes.

[0029] The processing of the processing unit 51 of the fixed display device 50 will be described in detail. As shown in FIG. 5, in the first embodiment, the processing unit 51 of the fixed display device 50 performs processing to display the virtual image A of the irradiated light on the display unit 53 of the fixed display device 50 so as to overlap the image of the three-dimensional model M of the workpiece 200. As described above, the processing unit 51 of the fixed display device 50 displays the virtual image A of the irradiated light so as to overlap the image of the three-dimensional model M of the workpiece 200. In addition, the image of the three-dimensional model M of the workpiece 200 displayed on the display unit 53 of the fixed display device 50 can be rotated or moved by an operation by the user U. The virtual image A of the irradiated light moves in accordance with the rotation or movement of the image of the three-dimensional model M.

[0030] Next, in step S5, the processing unit 31 of the robot controller 30 judges whether the operation of moving the robot arm 12 by the user U has ended. If the answer is no in step S6, the operations from step S1 to step S4 are repeated. That is, in the first embodiment, the processing unit 31 of the robot controller 30 moves the imaging unit 20 by the robot arm 12, and the processing unit 23 of the imaging unit 20 sequentially acquires the coordinates of the irradiated light in the image captured by the imaging unit 20 that is moved relatively to the workpiece 200. Then, the processing unit 41 of the wearable display device 40 and the processing unit 51 of the fixed display device 50 perform a process of superimposing a virtual image A of the linear irradiated light on the image of the workpiece 200 sequentially based on the coordinates of the irradiated light sequentially acquired by the imaging unit 20, and displaying the virtual image A on the display unit 45 of the wearable display device 40 and the display unit 53 of the fixed display device 50. That is, the display unit 45 of the wearable display device 40 and the display unit 53 of the fixed display device 50 display a virtual image A of the linear irradiated light whose position changes from moment to moment. If the answer is yes in step S6, the control process of the visual inspection system 100 for displaying the imageable range is completed.

[0031] (Effects of the first embodiment) The visual inspection system 100 includes a processing unit 41 and a processing unit 51 that perform processing to acquire the coordinates of the irradiated light in the image captured by the imaging unit 20 that is moved relative to the workpiece 200, and superimpose a virtual image A of the irradiated light generated by computer graphics on the image of the workpiece 200 based on the acquired coordinates of the irradiated light and the control point TCP of the robot arm 12, and display the virtual image A of the irradiated light on the display unit 45 and the display unit 53. As a result, the virtual image A of the irradiated light is displayed on the image of the workpiece 200 based on the coordinates of the actual irradiated light and the control point TCP of the actual robot arm 12. That is, since the shape and position are based on actual information, the error of the shape and position is small. Therefore, by visually checking the virtual image A of the irradiated light on the image of the workpiece 200, the user U can confirm how the irradiated light hits the workpiece 200, and therefore the imageable range of the workpiece 200 imaged by the imaging unit 20 can be accurately recognized.

[0032] The processing unit 31 moves the imaging unit 20 by the robot arm 12. The processing units 41 and 51 each sequentially acquire coordinates of the irradiated light in an image captured by the imaging unit 20 that is moved relatively to the workpiece 200, and perform processing to sequentially superimpose a virtual image A of the line-shaped irradiated light on the image of the workpiece 200 and display it on the display units 45 and 53 based on the sequentially acquired coordinates of the irradiated light. This allows the user U to recognize the inspection possible range based on the current posture of the robot arm 12 by visually recognizing the virtual image A of the line-shaped irradiated light.

[0033] The processing unit 41 of the wearable display device 40 performs processing to display the virtual image A of the irradiated light on the display unit 45 so as to be superimposed on the image of the actual workpiece 200. As a result, the virtual image A of the irradiated light is displayed so as to be superimposed on the image of the actual workpiece 200, so that the user U can accurately recognize the inspectable range of the actual workpiece 200.

[0034] The display unit 45 of the wearable display device 40 displays a virtual image A generated by computer graphics superimposed on the real image visually recognized by the user U. The processing unit 41 performs processing to display the virtual image A of the irradiated light on the display unit 45 of the wearable display device 40 so that it is superimposed on the image of the real workpiece 200 visually recognized by the user U. As a result, since the wearable display device 40 is worn by the user U, the virtual image A of the irradiated light also moves in response to the movement of the user U. As a result, the user U can check the real workpiece 200 and the inspectable range from various angles.

[0035] The processing unit 51 of the fixed display device 50 performs processing to display the virtual image A of the irradiated light on the display unit 53 so as to be superimposed on the image of the three-dimensional model M of the workpiece 200. As a result, the virtual image A of the irradiated light also moves in response to the rotation and movement of the three-dimensional model M of the workpiece 200 on the display unit 53, so that the workpiece 200 and the inspectable range can be confirmed from various angles.

[0036] [Second embodiment] A description will now be given of an appearance inspection system 100 according to the second embodiment. In the appearance inspection system 100 according to the second embodiment, a virtual image A1 of band-shaped irradiation light is displayed.

[0037] Specifically, as in the first embodiment, the processing unit 31 of the robot controller 30 moves the imaging unit 20 by the robot 10. Also, the processing unit 23 of the imaging unit 20 sequentially acquires the coordinates of the irradiated light in the image captured by the imaging unit 20 that is moved relative to the workpiece 200. Then, in the second embodiment, as shown in FIG. 7, the processing unit 41 of the wearable display device 40 performs a process of superimposing a virtual image A1 of the belt-shaped irradiated light on the image of the real workpiece 200 based on the sequentially acquired coordinates of the multiple irradiated lights, and displaying it on the display unit 45. Also, as shown in FIG. 8, the processing unit 51 of the fixed display device 50 performs a process of superimposing a virtual image A1 of the belt-shaped irradiated light on the image of the three-dimensional model M of the workpiece 200 based on the sequentially acquired coordinates of the multiple irradiated lights, and displaying it on the display unit 53. Specifically, the processing unit 41 of the wearable display device 40 and the processing unit 51 of the fixed display device 50 store the sequentially acquired coordinates of the multiple irradiated lights in the storage unit 42 of the wearable display device 40 and the storage unit 52 of the fixed display device 50, respectively. The coordinates of the multiple irradiated lights are stored from the start to the end of the movement of the imaging unit 20 by the robot arm 12. After the movement of the imaging unit 20 by the robot arm 12 is completed, the processing unit 41 of the wearable display device 40 reads out the coordinates of the multiple irradiated lights from the storage unit 42, and displays a virtual image A1 of the belt-shaped irradiated light on the image of the actual workpiece 200 based on the read coordinates of the multiple irradiated lights on the display unit 45. The processing unit 51 of the fixed display device 50 performs the same process, and performs a process of displaying the virtual image A1 of the belt-shaped irradiated light on the image of the three-dimensional model M of the workpiece 200 on the display unit 53. When the movement of the imaging unit 20 by the robot arm 12 is performed along multiple paths, multiple virtual images A1 of the belt-shaped irradiated light are displayed.

[0038] (Effects of the second embodiment) The processing unit 31 moves the imaging unit 20 or the workpiece 200 by the robot. The processing units 41 and 51 each sequentially acquire coordinates of the irradiated light in an image captured by the imaging unit 20 that is moved relative to the workpiece 200, and perform processing to superimpose a virtual image A1 of the belt-shaped irradiated light on the image of the workpiece 200 based on the sequentially acquired coordinates of the irradiated light and display the image on the display units 45 and 53. This allows the user U to visually recognize the virtual image A1 of the belt-shaped irradiated light and recognize the inspectable range within the range in which the robot arm 12 is moved.

[0039] (Modification) It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present disclosure is indicated by the claims, not by the description of the embodiments above, and further includes all modifications (variations) within the meaning and scope equivalent to the claims.

[0040] For example, in the above first and second embodiments, an example has been described in which the robot is a vertical articulated robot, but the present disclosure is not limited to this. In the present disclosure, the robot may be a robot other than a vertical articulated robot.

[0041] In the above first and second embodiments, an example has been described in which the robot moves the imaging unit, but the present disclosure is not limited to this. In the present disclosure, the robot may move the workpiece.

[0042] In the above first and second embodiments, the virtual image of the irradiation light of a portion corresponding to a portion where the pixel value of the image of the irradiation light captured by the imaging unit is equal to or greater than a threshold value is displayed on the display unit, but the present disclosure is not limited to this. In the present disclosure, the entire image of the irradiation light captured by the imaging unit may be displayed on the display unit.

[0043] In addition, in the above first and second embodiments, an example in which the virtual image of the irradiated light is displayed on both the wearable display device and the fixed display device has been described, but the present disclosure is not limited to this. In the present disclosure, the virtual image of the irradiated light may be displayed on only one of the wearable display device and the fixed display device.

[0044] In addition, the first embodiment in which a line-shaped virtual image of the irradiated light is displayed on the display unit and the second embodiment in which a band-shaped virtual image of the irradiated light is displayed on the display unit have been shown as separate embodiments, but the present disclosure is not limited to this. In the present disclosure, a user may switch between a mode in which a line-shaped virtual image of the irradiated light is displayed on the display unit and a mode in which a band-shaped virtual image of the irradiated light is displayed on the display unit.

[0045] In the above first and second embodiments, an example has been described in which the wearable display device is a display unit that displays mixed reality, but the present disclosure is not limited to this. In the present disclosure, the wearable display device may be a display unit that displays virtual reality. In this case, the wearable display device displays a virtual image of the irradiated light so as to be superimposed on an image of a three-dimensional model of the workpiece.

[0046] In the above first and second embodiments, an example has been shown in which an image of a 3D model of a workpiece is displayed on a fixed display device, but the present disclosure is not limited to this. In the present disclosure, an image of a real workpiece visually recognized by a user may be displayed on a fixed display device, and a virtual image of irradiated light may be displayed so as to be superimposed on the image of the real workpiece.

[0047] In the above first and second embodiments, the present disclosure is applied to a workpiece having a curved surface, but the present disclosure is not limited to this. The present disclosure may be applied to a flat workpiece that does not have a curved surface.

[0048] In the above first and second embodiments, the imaging unit is a line sensor, but the present disclosure is not limited to this. In the present disclosure, the imaging unit may be an area camera that captures a two-dimensional image.

[0049] In addition, in the above first and second embodiments, an example in which the robot system of the present disclosure is applied to a visual inspection system has been described, but the present disclosure is not limited to this. The robot system of the present disclosure may be applied to a system other than a visual inspection system.

[0050] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general purpose processors, special purpose processors, integrated circuits, Application Specific Integrated Circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. Processors are considered processing circuits or circuits because they include transistors and other circuits. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

[0051] [Aspects] It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0052] (Aspect 1) A robot arm; An imaging unit that includes an irradiation unit that irradiates a workpiece with irradiation light and images the workpiece irradiated with the irradiation light; A display unit on which the work is displayed; The robot arm moves the imaging unit or the workpiece to obtain coordinates of the irradiation light in the image captured by the imaging unit that is moved relative to the workpiece; a processing unit that performs processing to superimpose a virtual image of the irradiation light generated by computer graphics on an image of a workpiece based on the acquired coordinates of the irradiation light and a control point of the robot arm, and display the superimposed image on the display unit.

[0053] (Aspect 2) The processing unit includes: moving the imaging unit or the workpiece by the robot arm; Sequentially acquiring coordinates of the irradiation light in the images captured by the imaging unit which is moved relative to the workpiece; A robot system according to aspect 1, which performs a process of sequentially superimposing a virtual image of the line-shaped irradiation light on an image of the workpiece and displaying it on the display unit based on the coordinates of the irradiation light sequentially acquired.

[0054] (Aspect 3) The processing unit includes: moving the imaging unit or the workpiece by the robot arm; Sequentially acquiring coordinates of the irradiation light in the images captured by the imaging unit which is moved relative to the workpiece; A robot system according to aspect 1, which performs a process of superimposing a virtual image of the band-shaped irradiation light on an image of the workpiece and displaying it on the display unit based on the coordinates of multiple irradiation lights acquired sequentially.

[0055] (Aspect 4) The robot system according to any one of aspects 1 to 3, wherein the processing unit performs processing to display a virtual image of the irradiated light on the display unit so as to be superimposed on an image of an actual workpiece.

[0056] (Aspect 5) The display unit displays a virtual image generated by computer graphics on a real image visually recognized by a user, and includes a display unit of a wearable display device worn by the user; The robot system of aspect 4, wherein the processing unit performs processing to display a virtual image of the irradiated light on the display unit of the wearable display device so that the virtual image is superimposed on the image of the actual workpiece visually recognized by the user.

[0057] (Aspect 6) The display unit includes a display unit of a fixed display device on which an image of a three-dimensional model of the workpiece is displayed, The robot system according to any one of aspects 1 to 5, wherein the processing unit performs processing to display a virtual image of the irradiated light on the display unit of the fixed display device so as to be superimposed on an image of a three-dimensional model of the workpiece.

[0058] (Aspect 7) Moving an imaging unit that includes an irradiation unit that irradiates a workpiece with irradiation light and images the workpiece irradiated with the irradiation light, or the workpiece, by a robot arm; Acquiring coordinates of irradiation light in an image captured by the imaging unit that is moved relative to the workpiece; and displaying on a display unit a virtual image of the irradiation light generated by computer graphics based on the acquired coordinates of the irradiation light and a control point of the robot arm, superimposing the virtual image of the irradiation light on an image of the work. [Explanation of symbols]

[0059] 12 Robot Arm 20 Imaging unit 21 Irradiation unit 31 Processing section 40 Wearable display device 41 Processing section 45 Display section 50 Fixed Display Device 51 Processing section 53 Display section 100 Visual inspection system (robot system) 200 Work A Virtual Image A1 Virtual Image M 3D model TCP control points

Claims

1. A robot arm; An imaging unit that includes an irradiation unit that irradiates a workpiece with irradiation light and images the workpiece irradiated with the irradiation light; A display unit on which the work is displayed; The robot arm moves the imaging unit or the workpiece to obtain coordinates of the irradiation light in the image captured by the imaging unit that is moved relative to the workpiece; A processing unit that performs processing for superimposing a virtual image of the irradiation light generated by computer graphics on an image of a workpiece based on the acquired coordinates of the irradiation light and a control point of the robot arm, and displaying the superimposed image on the display unit, The processing unit, during teaching of the operation of the robot arm, moving the imaging unit or the workpiece by the robot arm; Sequentially acquiring coordinates of the irradiation light in the images captured by the imaging unit which is moved relative to the workpiece; A robot system that performs processing to sequentially superimpose a virtual image of a line-shaped irradiation light, the position of which changes from moment to moment, on an image of the workpiece based on the sequentially acquired coordinates of the irradiation light and display it on the display unit.

2. The robot system according to claim 1 , wherein the processing unit performs processing for displaying a virtual image of the irradiated light on the display unit so as to be superimposed on an image of an actual workpiece.

3. The display unit displays a virtual image generated by computer graphics on a real image visually recognized by a user, and includes a display unit of a wearable display device worn by the user; The robot system according to claim 2 , wherein the processing unit performs processing for displaying a virtual image of the irradiated light on the display unit of the wearable display device so as to be superimposed on an image of an actual workpiece visually recognized by the user.

4. The display unit includes a display unit of a fixed display device on which an image of a three-dimensional model of the workpiece is displayed, The robot system according to claim 1 , wherein the processing unit performs processing for displaying a virtual image of the irradiated light on the display unit of the fixed display device so as to be superimposed on an image of the three-dimensional model of the workpiece.

5. Moving an imaging unit that includes an irradiation unit that irradiates a workpiece with irradiation light and images the workpiece irradiated with the irradiation light, or the workpiece, by a robot arm; Acquiring coordinates of irradiation light in an image captured by the imaging unit that is moved relative to the workpiece; and displaying a virtual image of the irradiation light generated by computer graphics on a display unit by superimposing the virtual image of the irradiation light on an image of a workpiece based on the acquired coordinates of the irradiation light and the control point of the robot arm. A method for controlling a robot system, in which the virtual image of the irradiated light is superimposed on an image of the workpiece and displayed on the display unit, includes: during teaching of the operation of the robot arm, moving the imaging unit or the workpiece by the robot arm, sequentially acquiring coordinates of the irradiated light in an image captured by the imaging unit which is moved relative to the workpiece, and sequentially superimposing a single line-shaped virtual image of the irradiated light, whose position changes from moment to moment, on the image of the workpiece and displaying it on the display unit based on the sequentially acquired coordinates of the irradiated light.

Images