Inspection system and inspection method

Abstract

An inspection system (100) is provided with a processing unit (61) that performs: a process for detecting a target in a workpiece (200) in an inspection image (21) acquired by inspecting the workpiece (200) by an inspection unit (20); and a process for causing a virtual mark (210) indicating a position of the detected target (201) to be displayed on a wearable display device (60) in accordance with the movement of a user (U) so that the virtual mark (210) overlaps a real-world image viewed by the user (U).

Description

Inspection system and inspection method

[0001] This disclosure relates to inspection systems and methods.

[0002] Conventionally, inspection systems have been known. For example, Japanese Patent Application Laid-Open No. 2008-046103 discloses a surface inspection device that inspects the coated surface of a workpiece. In this surface inspection device, a workpiece is photographed by a camera. Then, scratches on the workpiece are detected from the photographed image of the workpiece. The positions of the detected scratches on the workpiece are converted into positions on a graphic of the workpiece displayed on a display device, and the converted positions are displayed on the display device.

[0003] JP 2008-046103 A

[0004] In the surface inspection device described in JP 2008-046103 A, the position of a flaw on a workpiece is displayed on a workpiece graphic displayed on a display device. Therefore, since the workpiece in the real world and the workpiece graphic displayed on the display device are different entities, the user must first check the position of the flaw on the workpiece using the graphic displayed on the display device, and then check the actual flaw on the workpiece again. This poses the problem of time and effort required to check the position of an object such as a flaw on the workpiece.

[0005] This disclosure has been made to solve the above-mentioned problems, and one purpose of this disclosure is to provide an inspection system and an inspection method that can save the user the trouble of checking the position of the target on the workpiece.

[0006] An inspection system according to a first aspect of this disclosure comprises a robot, an inspection unit that inspects a workpiece, a wearable display device that is worn by the user and that displays a virtual image generated by computer graphics superimposed on an image of the real world viewed by the user, and a processing unit that performs the following processes: acquiring an inspection image of the workpiece by having the robot move the inspection unit relative to the workpiece and inspecting the workpiece with the inspection unit; detecting an object of the workpiece in the acquired inspection image; and displaying a virtual marker as a virtual image indicating the position of the detected object on the wearable display device in accordance with the user's movements so that it is superimposed on the image of the real world viewed by the user.

[0007] As described above, the inspection system according to the first aspect of this disclosure includes a processing unit that performs processing to display a virtual marker, which is a virtual image indicating the position of a detected object, on the wearable display device in accordance with the user's movements so that the virtual marker is superimposed on the real-world image viewed by the user. This allows the user to view both the actual workpiece in the real world and the virtual image indicating the position of the object through the wearable display device, allowing the user to directly confirm the position of the detected object on the actual workpiece in the real world based on the virtual marker. This eliminates the need for the user to confirm the position of the object on the workpiece. Furthermore, because the virtual marker is displayed in accordance with the movements of the user wearing the wearable display device, the positional relationship between the workpiece and the virtual marker does not shift even if the relative positions of the user wearing the wearable display device and the workpiece change. Therefore, the position of the object can be properly confirmed even if the user wearing the wearable display device moves.

[0008] An inspection method according to a second aspect of this disclosure includes: acquiring an inspection image of the work by using a robot to move an inspection unit relative to the work and inspecting the work with the inspection unit; detecting an object of the work in the acquired inspection image; and displaying a virtual marker, which is a virtual image generated by computer graphics indicating the position of the detected object, on a wearable display device worn by the user in accordance with the user's movements so that the virtual marker is superimposed on an image of the real world viewed by the user.

[0009] As described above, the inspection method according to a second aspect of this disclosure includes displaying a virtual marker, which is a virtual image generated by computer graphics and indicates the position of a detected object, on a wearable display device worn by the user in accordance with the user's movements so as to be superimposed on a real-world image viewed by the user. This allows the user to view both the actual workpiece in the real world and the virtual image indicating the position of the object through the wearable display device, allowing the user to directly confirm the position of the detected object on the actual workpiece in the real world based on the virtual marker. This provides an inspection method that eliminates the user's effort in confirming the position of the object on the workpiece. Furthermore, because the virtual marker is displayed in accordance with the movements of the user wearing the wearable display device, the positional relationship between the workpiece and the virtual marker does not shift even if the relative positions of the user wearing the wearable display device and the workpiece change. This provides an inspection method that allows the user wearing the wearable display device to properly confirm the position of the object even when the user moves.

[0010] As described above, the inspection system and inspection method of the present disclosure can save the user the trouble of checking the position of the target on the workpiece.

[0011] 1 is a diagram showing an inspection system according to a first embodiment. FIG. 2 is a block diagram showing an inspection system according to the first embodiment. FIG. 3 is a diagram showing a wearable display device according to the first embodiment. FIG. 4 is a flowchart for explaining control processing of the inspection system according to the first embodiment. FIG. 5 is a diagram for explaining inspection of a workpiece according to the first embodiment. FIG. 6 is a diagram for explaining an inspection image according to the first embodiment. FIG. 7 is a diagram showing an image of the real world seen by a user through the display unit of the wearable display device, and a virtual image. FIG. 8 is a diagram showing an image of a workpiece seen by a user through the display unit of the wearable display device, multiple objects, and multiple virtual markers. FIG. 9 is a diagram showing a state in which the virtual markers are enlarged. FIG. 10 is a diagram showing a state in which the type of object for which virtual markers are displayed has been selected. FIG. 11 is a diagram showing an image of a workpiece seen by a user through the display unit of the wearable display device, the objects, the virtual markers, and work content for the objects. FIG. 12 is a diagram for explaining a method of aligning real-world coordinates with coordinates of the wearable display device based on the user's pointing. FIG. 13 is a diagram for explaining a method of aligning real-world coordinates with coordinates of the wearable display device by the user pinching and moving an arrow. FIG. 14 is a diagram showing a state in which an inspection image is displayed on the display unit of the wearable display device. FIG. 15 is a block diagram showing an appearance inspection system according to a second embodiment. FIG. 1 is a diagram showing a wearable display device according to a second embodiment. FIG. 2 is a diagram showing a virtual image displayed on a display unit of a wearable display device according to a second embodiment. FIG. 3 is a diagram showing a virtual image displayed on a display unit of a fixed display device according to a second embodiment. FIG. 4 is a flow chart for explaining a control method of the appearance inspection system according to the second embodiment. FIG. 5 is a diagram showing a virtual image displayed on a display unit of a wearable display device according to a third embodiment. FIG. 6 is a diagram showing a virtual image displayed on a display unit of a fixed display device according to a third embodiment.

[0012] Hereinafter, embodiments embodying the present disclosure will be described with reference to the drawings.

[0013] First Embodiment (Configuration of Inspection System) The overall configuration of an inspection system 100 according to a first embodiment will be described with reference to FIGS. 1 and 2. FIG.

[0014] 1, the inspection system 100 is an appearance inspection system that inspects the appearance of a workpiece 200. The workpiece 200 is, for example, a product or part related to an automobile, agricultural machinery, ceramics, or household electrical appliances. The workpiece 200 is not particularly limited.

[0015] The inspection system 100 includes a robot 10, an inspection unit 20, a robot controller 40, an image processing device 50, a wearable display device 60, and a fixed display device 70.

[0016] The robot 10 moves the inspection unit 20 relative to the workpiece 200. The robot 10 is a vertical articulated robot. The robot 10 includes a base unit 11 and an arm unit 12 connected to the base unit 11. The base unit 11 is fixed to an installation surface such as a floor, wall, or ceiling. The base unit 11 may also be attached to a movable cart. The arm unit 12 has multiple joints. Each of the multiple joints has a servo motor as a drive source. The tip of the arm unit 12 holds the inspection unit 20. The robot 10 moves the inspection unit 20 held at the tip of the arm unit 12 relative to the fixed workpiece 200 by driving the multiple joints of the arm unit 12.

[0017] The inspection unit 20 is disposed on the robot 10 and inspects the workpiece 200. The inspection unit 20 is an imaging unit and captures an image of the workpiece 200. Specifically, the inspection unit 20 is a line-type camera that is moved along the surface of the workpiece 200 by the robot 10 and scans and captures an image of the surface of the workpiece 200.

[0018] 2, the robot controller 40 controls the operation of the robot 10. The robot controller 40 includes a processing unit 41 and a storage unit 42. The processing unit 41 includes a processor and performs various processes related to the operation of the robot 10. The storage unit 42 includes a non-volatile memory and stores various information such as programs for operating the robot 10.

[0019] The image processing device 50 performs image processing on the images captured by the inspection unit 20. The image processing device 50 also controls the timing of imaging by the inspection unit 20. The image processing device 50 includes a processing unit 51 and a storage unit 52. The processing unit 51 includes a processor, and performs various processes related to the images captured by the inspection unit 20 and the timing of imaging by the inspection unit 20. The storage unit 52 includes a non-volatile memory, and stores various information such as programs for performing image processing.

[0020] As shown in FIG. 3 , the wearable display device 60 displays a virtual image generated by computer graphics superimposed on an image of the real world viewed by the user U. That is, the wearable display device 60 is a display unit that displays mixed reality. The wearable display device 60 is worn by the user U. Specifically, in the first embodiment, the wearable display device 60 includes a goggle-type display unit that is worn on the head of the user U. As shown in FIG. 2 , the wearable display device 60 includes a processing unit 61, a storage unit 62, an imaging unit 63, a sensor 64, and a display unit 65. The processing unit 61 includes a processor and performs various processes such as generating a virtual image. The storage unit 62 includes a non-volatile memory and stores various information such as programs for generating the virtual image. The imaging unit 63 captures an image of the real world. The sensor 64 includes an acceleration sensor, a gyroscope, etc. The sensor 64 detects the movement of the user U, etc. The display unit 65 is a see-through display unit, and the user U can view real-world images of the robot 10, the workpiece 200, and the like through the display unit 65. The display unit 65 also displays a virtual image generated by computer graphics superimposed on the real-world image. The virtual image is, for example, a hologram. Details of the virtual image will be described later.

[0021] Furthermore, in the first embodiment, the fixed display device 70 is fixedly disposed without being worn by the user U. The fixed display device 70 displays the image visually recognized by the user U through the wearable display device 60 and the virtual marker 210. That is, the same image as the image visually recognized by the user U is displayed on the fixed display device 70. Specifically, an image of the real world captured by the imaging unit 63 of the wearable display device 60 and the virtual marker 210 are displayed on the fixed display device 70. The fixed display device 70 is, for example, a liquid crystal display or an organic EL display.

[0022] (Control Processing of Inspection System) The control processing of the inspection system 100 will be described.

[0023] In step S1 shown in Fig. 4 , as shown in Fig. 5 , the processing unit performs processing to acquire an inspection image 21 of the workpiece 200 by having the robot 10 move the inspection unit 20 relatively to the workpiece 200 and inspect the workpiece 200 with the inspection unit 20. Specifically, the processing unit 41 of the robot controller 40 performs processing to operate the robot 10 based on the movement path 13 and inspect the workpiece 200 with the inspection unit 20. The movement path 13 is a path for operating the arm unit 12 of the robot 10, and a plurality of movement paths 13 are generated in order to inspect the workpiece 200. Furthermore, the movement paths 13 are generated in advance before the workpiece 200 is inspected.

[0024] For example, the processing unit 41 receives instructions from the user regarding the operation of the arm unit 12, and performs processing to generate the movement path 13 of the robot 10 based on the received instructions. Furthermore, for example, the processing unit 41 performs processing to automatically generate the movement path 13 of the robot 10, regardless of instructions from the user regarding the operation of the arm unit 12. Furthermore, the processing unit 41 performs processing to generate the movement path 13 that follows the surface of the workpiece 200, such as a curved surface.

[0025] 6 , a processing unit 51 of the image processing device 50 performs processing to acquire an inspection image 21 based on the output result of the inspection unit 20. The inspection image 21 is an image of the surface of the workpiece 200 captured by the inspection unit 20.

[0026] As shown in FIG. 5 , the processing unit 51 operates the inspection unit 20 to inspect the workpiece 200 at regular intervals along the movement path 13, thereby acquiring the inspection image 21. Specifically, the processing unit 51 operates the inspection unit 20 to capture an image of the workpiece 200 at regular intervals, thereby scanning and capturing the image of the workpiece 200. More specifically, the processing unit 41 outputs a pulse signal to the processing unit 51 at regular intervals. Based on the pulse signal from the processing unit 41, the processing unit 51 outputs a trigger signal to the inspection unit 20 at regular intervals. Based on the trigger signal, the inspection unit 20 captures an image of the workpiece 200 at regular intervals. Note that the regular intervals are the intervals between the control points of the robot 10. If the inspection unit 20 is an imaging unit, the control point of the robot 10 is set to the focal position of the imaging of the inspection unit 20. Furthermore, the focal position of the imaging of the inspection unit 20 is set near the surface of the workpiece 200. The control point of the robot 10 is provided so that the inspection unit 20 can perform the process of capturing an image of the workpiece 200 .

[0027] The processing unit 41 performs a process of inspecting the workpiece 200 by the inspection unit 20 for all of the movement paths 13. The processing unit 51 also performs a process of acquiring the inspection image 21 for all of the movement paths 13.

[0028] In step S2 shown in FIG. 4 , as shown in FIG. 6 , the processing unit 51 of the image processing device 50 performs processing to detect the object 201 of the workpiece 200 in the inspection image 21. The processing unit 51 performs predetermined image processing on the inspection image 21 to detect the object 201 in the inspection image 21. In the first embodiment, the object 201 is, for example, a defect such as a scratch, a foreign object, or a dent. The processing unit 51 performs processing to detect the object 201 in the inspection image 21 for all inspection images 21. That is, in the first embodiment, when multiple objects 201 exist in the workpiece 200, the processing unit 51 of the image processing device 50 performs processing to detect multiple objects 201 of the workpiece 200 in the acquired inspection image 21. In addition, the processing unit 51 performs processing to output the inspection results of the workpiece 200 to the processing unit 41 of the robot controller 40 and the processing unit 61 of the wearable display device 60.

[0029] In step S3 shown in FIG. 4 , the user U wears the wearable display device 60 on his / her head. As shown in FIG. 7 , a marker 202 is placed near the workpiece 200 to align the coordinates of the wearable display device 60 with those of the real world. The marker 202 is, for example, a QR code (registered trademark). When the user U wears the wearable display device 60 on his / her head and visually recognizes the marker 202, the imaging unit 63 of the wearable display device 60 captures an image of the marker 202. The processing unit 61 of the wearable display device 60 then aligns the coordinates of the real world with those of the wearable display device 60 based on information such as the QR code (registered trademark) obtained from the marker 202 and the robot coordinate system of the robot 10. The coordinates of the workpiece 200 in the real world are preset based on the robot coordinate system. The coordinates of the workpiece 200 in the real world are input from the robot controller 40 to the wearable display device 60. Specifically, the robot controller 40 holds a two-dimensional to three-dimensional coordinate conversion table. The coordinate conversion table is a table that converts coordinate values ​​in an inspection coordinate system, described later, of an inspection image 21 acquired by the inspection unit 20 inspecting the workpiece 200, into coordinate values ​​in a robot coordinate system and a workpiece coordinate system, which are three-dimensional coordinate systems. The robot controller 40 also holds the three-dimensional coordinates of the workpiece 200 and outputs the three-dimensional coordinates of the workpiece 200 to the wearable display device 60. The two-dimensional coordinates of the object 201 are input to the robot controller 40 from the image processing device 50, and the robot controller 40 converts the two-dimensional coordinates of the object 201 into three-dimensional coordinates using the coordinate conversion table and outputs the three-dimensional coordinates to the wearable display device 60.

[0030] In step S4 shown in FIG. 4 , in the first embodiment, as shown in FIG. 7 , the processing unit 61 of the wearable display device 60 performs a process of displaying a virtual marker 210 as a virtual image indicating the position of the detected target 201 on the wearable display device 60 in accordance with the movement of the user U, so that the virtual marker 210 is superimposed on the real-world image visually perceived by the user U. FIG. 7 shows an image visually perceived by the user U through the display unit 65 of the wearable display device 60. For example, as shown in FIG. 7 , the user U visually perceives the workpiece 200 and the robot 10 in the real world through the display unit 65. The display unit 65 also displays an arrow A pointing to the position of the target 201 as the virtual marker 210 indicating the position of the detected target 201. As shown in FIG. 8 , if the user U's field of view changes due to movement of the user U, the position of the arrow A on the display unit 65 also changes in accordance with the movement of the user U. Note that in FIG. 8 , the image of the robot 10 is omitted, and only the workpiece 200 is depicted. Since the coordinates of the real world and the coordinates of the wearable display device 60 are aligned, it is possible to link the change in the position of the arrow A with the movement of the user U. Furthermore, if the user U moves and becomes aware of an object 201 that was hidden in a blind spot of the workpiece 200 and could not be seen, a new arrow A pointing to this object 201 is displayed.

[0031] Furthermore, in the first embodiment, when the processing unit 51 of the image processing device 50 detects a plurality of targets 201 on the workpiece 200, the processing unit 61 of the wearable display device 60 performs processing to change the display mode of the virtual marker 210 depending on the types of the plurality of targets 201. Specifically, in FIG. 8 , it is assumed that the types of targets 201a and 201b are different. For example, the processing unit 61 executes processing to display the arrow A pointing to target 201a in a different color from the arrow A pointing to target 201b in a different color. In FIG. 8 , the colors are distinguished by hatching and no hatching.

[0032] In the first embodiment, as shown in FIG. 9 , when the processing unit 61 of the wearable display device 60 detects a user U's action of enlarging or reducing the virtual marker 210, the processing unit 61 performs a process of enlarging or reducing the virtual marker 210 in accordance with the user U's action of enlarging or reducing the virtual marker 210. For example, when the user U pinches the end of the arrow A with his / her fingers and pulls the virtual marker 210 in the outward direction of the arrow A, the processing unit 61 recognizes the user U's action from an image captured by the imaging unit 63 of the wearable display device 60. Then, the processing unit 61 performs a process of enlarging the arrow A and displaying it on the wearable display device 60 in accordance with the user U's action of pulling the virtual marker 210 in the outward direction of the arrow A. Furthermore, when the user U pinches the end of the arrow A with his / her fingers and pushes the virtual marker 210 inward direction of the arrow A, the virtual marker 210 is reduced in size. Note that the above-described action of enlarging or reducing the virtual marker 210 by the user U is merely an example, and the present invention is not limited to the above-described action.

[0033] In the first embodiment, as shown in FIG. 7 , the processing unit 61 of the wearable display device 60 performs processing to display a first virtual switch 221 as a virtual image that switches between displaying and hiding the virtual marker 210 on the wearable display device 60. For example, a rectangular, semitransparent, plate-like member 220 is displayed on the display unit 65 of the wearable display device 60. The first virtual switch 221 is disposed on the member 220. For example, when the user U presses the first virtual switch 221 with his / her finger while the virtual marker 210 is displayed, the processing unit 61 recognizes the user U's action and hides the virtual marker 210. Furthermore, when the user U presses the first virtual switch 221 with his / her finger again, the virtual marker 210 is displayed. Note that the above-described switching method based on pressing the first virtual switch 221 is merely an example and is not limited to the above-described switching method. Note that the first virtual switch 221 is an example of a display changeover switch.

[0034] In the first embodiment, as shown in FIG. 7 , the processing unit 61 of the wearable display device 60 performs processing to display a second virtual switch 222 as a virtual image for selecting the type of target 201 for which the virtual marker 210 is to be displayed from among the multiple targets 201. For example, the second virtual switch 222 is disposed on a member 220 displayed on the display unit 65 of the wearable display device 60. As shown in FIG. 8 , all targets 201 and arrows A pointing to all targets 201 are displayed until the user U presses the second virtual switch 222 with his / her finger. Then, when the user U presses the second virtual switch 222 with his / her finger, only targets 201 a and the arrow A pointing to target 201 a are displayed, as shown in FIG. 10 . When the user U presses the second virtual switch 222 again with his / her finger, the arrows A pointing to targets 201 a and 201 a disappear, and only targets 201 b and the arrow A pointing to target 201 b are displayed. Furthermore, when the user U presses the second virtual switch 222 with his / her finger, all of the objects 201 and an arrow A pointing to all of the objects 201 are displayed. Note that the above-described method of switching the displayed objects 201 and the arrow A based on the pressing of the second virtual switch 222 is an example, and is not limited to the above-described switching method. Note that the second virtual switch 222 is an example of an object type selection switch.

[0035] Furthermore, in the first embodiment, the processing unit 61 of the wearable display device 60 executes processing to cause the wearable display device 60 to display an image and a virtual marker 210 that are visually recognized by the user U through the wearable display device 60. That is, the wearable display device 60 displays an image of the workpiece 200 and the robot 10 in the real world captured by the imaging unit 63 of the wearable display device 60, and an arrow A indicating the position of the target 201.

[0036] Effect of the First Embodiment The inspection system 100 includes a processing unit 61 that performs processing to display a virtual marker 210, which is a virtual image indicating the position of the detected target 201, on the wearable display device 60 in accordance with the movement of the user U so that the virtual marker 210 is superimposed on the real-world image visually recognized by the user U. As a result, the user U can visually recognize both the actual workpiece 200 in the real world and the virtual marker 210 indicating the position of the target 201 through the wearable display device 60. Therefore, the user U can directly confirm the position of the detected target 201 on the actual workpiece 200 in the real world based on the virtual marker 210. This eliminates the need for the user U to check the position of the target 201 on the workpiece 200. Furthermore, because the virtual marker 210 is displayed in accordance with the movement of the user U wearing the wearable display device 60, the positional relationship between the workpiece 200 and the virtual marker 210 does not shift even if the relative positions of the user U wearing the wearable display device 60 and the workpiece 200 change. Therefore, even if the user U wearing the wearable display device 60 moves, the position of the target 201 can be properly confirmed.

[0037] The target 201 includes a defect in the workpiece 200. This can save the user U the trouble of checking the position of the defect in the workpiece 200.

[0038] The virtual marker 210 includes an arrow A that indicates the position of the target 201. As a result, the position of the target 201 is indicated by the arrow A, and the user U can easily recognize the position of the target 201.

[0039] When the processing unit 61 detects an action of the user U to enlarge or reduce the virtual marker 210, the processing unit 61 performs a process of enlarging or reducing the virtual marker 210 in accordance with the enlarging or reducing action of the user U. As a result, if the virtual marker 210 is small and difficult for the user U to view, the virtual marker 210 can be enlarged to make it easier to view. Also, if the virtual marker 210 is large and difficult to view the workpiece 200, the virtual marker 210 can be reduced to make it easier to view the workpiece 200. Also, if the virtual markers 210 are large and multiple virtual markers 210 overlap each other, the overlapping of the virtual markers 210 can be eliminated by reducing the virtual markers 210.

[0040] The processing unit 61 performs processing to display a first virtual switch 221 as a virtual image that switches between displaying and hiding the virtual marker 210 on the wearable display device 60. This makes it possible to make the workpiece 200 more easily visible by hiding the virtual marker 210, for example, when the workpiece 200 is hidden behind the virtual marker 210 and is difficult to see. In addition, the user U can operate the first virtual switch 221 to display the virtual marker 210 that was once hidden again.

[0041] The processing unit 51 performs processing to detect a plurality of targets 201 of the workpiece 200 in the acquired inspection image 21. The processing unit 61 also performs processing to change the display mode of the virtual marker 210 depending on the type of the plurality of targets 201. As a result, the display mode of the virtual marker 210 differs depending on the type of the plurality of targets 201, so that the user U can easily distinguish the type of the target 201.

[0042] The processing unit 61 performs processing to display a second virtual switch 222 as a virtual image for selecting the type of target 201 for which the virtual sign 210 is to be displayed from among the plurality of targets 201. As a result, by the user U operating the second virtual switch 222, only the virtual sign 210 for the type of target 201 desired by the user U can be displayed.

[0043] The inspection system 100 includes a fixed display device 70 that is not worn by the user U but is fixedly disposed, and displays an image and a virtual marker 210 that are visually recognized by the user U through the wearable display device 60. This allows workers other than the user U to visually recognize the target 201 on the workpiece 200 and the virtual marker 210 that points to the target 201.

[0044] The wearable display device 60 includes a goggle-type display unit that is worn on the head of the user U. As a result, since the wearable display device 60 is worn on the head of the user U, even if the viewing angle changes due to the movement of the user U's head, the virtual marker 210 can be moved in accordance with the change in viewing angle.

[0045] Second Embodiment The overall configuration of a visual inspection system 400 according to a second embodiment will be described.

[0046] 15 , the visual inspection system 400 is an appearance inspection system that inspects the appearance of a workpiece 200. The workpiece 200 is, for example, a product or part related to an automobile, agricultural machinery, ceramics, or household electrical appliances. The workpiece 200 is not particularly limited. The visual inspection system 400 is also an example of a robot system.

[0047] The appearance inspection system 400 includes a robot 310 , an imaging unit 320 , a robot controller 330 , a wearable display device 340 , and a fixed display device 350 .

[0048] The robot 310 is, for example, a vertical articulated robot. The robot 310 includes a base 311 and a robot arm 312 connected to the base 311. The base 311 is fixed to an installation surface such as a floor, a wall, or a ceiling. The base 311 may also be attached to a movable carriage. The robot arm 312 has multiple joints. Each of the multiple joints has a servo motor as a drive source. The tip of the robot arm 312 holds an imaging unit 320. The robot 310 moves the imaging unit 320 held at the tip of the robot arm 312 relative to the fixed workpiece 200 by driving the multiple joints of the robot arm 312.

[0049] The imaging unit 320 is attached to the tip of the robot arm 312. As shown in FIG. 16 , the imaging unit 320 includes an irradiation unit 321 that irradiates the workpiece 200 with irradiation light and captures an image of the workpiece 200 irradiated with the irradiation light. The imaging unit 320 also includes an imaging element unit 322 that captures an image of the irradiation light irradiated by the irradiation unit 321 and the workpiece 200. For example, the imaging unit 320 is a line-type camera that is moved along the surface of the workpiece 200 by the robot 310 to scan and capture an image of the surface of the workpiece 200. The imaging unit 320 also includes a processing unit 323. The processing unit 323 performs image processing on the image captured by the imaging element unit 322. The processing unit 323 also controls the timing of imaging by the imaging unit 320.

[0050] As shown in FIG. 16 , the robot controller 330 controls the operation of the robot 310. The robot controller 330 includes a processing unit 331, a storage unit 332, and a signal output unit 333. The processing unit 331 includes a processor and performs various processes related to the operation of the robot 310. The storage unit 332 includes a non-volatile memory and stores various information such as programs for operating the robot 310. The signal output unit 333 includes, for example, a field programmable gate array (FPGA) and is capable of quickly outputting the coordinates of a tool center point (TCP), which is a control point of the robot 310. The signal output unit 333 is also capable of outputting multiple coordinates of the robot 310. For example, the signal output unit 333 is capable of outputting the coordinates of points near the control point TCP in addition to the coordinates of the control point TCP.

[0051] As shown in FIG. 17 , the wearable display device 340 displays a virtual image B generated by computer graphics superimposed on a real image viewed by the user U. That is, the wearable display device 340 is a display unit that displays mixed reality. The wearable display device 340 is worn by the user U. Specifically, the wearable display device 340 includes a goggle-type display unit worn on the head of the user U. As shown in FIG. 16 , the wearable display device 340 includes a processing unit 341, a memory unit 342, an imaging unit 343, a sensor 344, and a display unit 345. The processing unit 341 includes a processor and performs various processes, such as generating the virtual image B. The memory unit 342 includes a non-volatile memory and stores various information, such as a program for generating the virtual image B. The imaging unit 343 captures a real image. The sensor 344 includes an acceleration sensor, a gyroscope, and the like. The sensor 344 detects the movement of the user U, etc. 18 , the display unit 345 is a see-through display unit, and the user U can view real images of the robot 310, the workpiece 200, and the like through the display unit 345. The display unit 345 also displays a virtual image B generated by computer graphics and superimposed on the real image. The virtual image B is, for example, a hologram. The virtual image B will be described in detail later.

[0052] In the second embodiment, as shown in FIG. 19 , the fixed display device 350 is not worn by the user U but is fixedly disposed. The fixed display device 350 displays an image of a three-dimensional model M of the workpiece 200. The image of the three-dimensional model M of the workpiece 200 is, for example, an image of the workpiece 200 created using CAD (Computer Aided Design). As shown in FIG. 16 , the fixed display device 350 includes a processing unit 351, a storage unit 352, and a display unit 353. The processing unit 351 includes a processor and performs various processes, such as generating a virtual image B. The storage unit 352 includes a non-volatile memory and stores various information, such as a program for generating the virtual image B. The display unit 353 is, for example, a liquid crystal display or an organic EL display.

[0053] (Function of Displaying Image Capable Range) The visual inspection system 400 has a function of displaying the imageable range of the workpiece 200 to be inspected. Specifically, a virtual image B of the irradiated light irradiated from the irradiation unit 321 onto the workpiece 200 is generated by computer graphics, and the virtual image B of the irradiated light is displayed on the display unit 345 of the wearable display device 340 so as to be superimposed on the actual image visually recognized by the user U. Also, the virtual image B of the irradiated light is displayed on the display unit 353 of the fixed display device 350 so as to be superimposed on the image of the three-dimensional model M of the workpiece 200. Also, the function of displaying the imageable range is used, for example, when the user U teaches the robot 310 an operation for inspecting the workpiece 200. Below, a control method of the visual inspection system 400 for displaying the imageable range will be described in detail.

[0054] 20 , in step S11, the processing unit 331 of the robot controller 330 accepts an operation of the teaching pendant by the user U. Then, based on the accepted operation, the processing unit 331 moves the robot arm 312 to move the imaging unit 320 including the irradiation unit 321 relative to the workpiece 200.

[0055] In step S12, in the second embodiment, the processing unit 323 of the imaging unit 320 acquires the coordinates of the irradiated light in the image captured by the imaging unit 320, which is moved relative to the workpiece 200. Specifically, while the imaging unit 320 is in the on state, the processing unit 323 acquires the image captured by the image sensor unit 322.

[0056] In step S13, the processing unit 323 acquires the coordinates of the irradiated light in the image captured by the imaging unit 320, which is moved relatively to the workpiece 200. First, the processing unit 323 performs pre-processing of the acquired image.

[0057] In step S14, in the second embodiment, the processing unit 341 of the wearable display device 340 and the processing unit 351 of the fixed display device 350 each perform processing to superimpose a virtual image B of the irradiated light generated by computer graphics onto the image of the workpiece 200 based on the coordinates of the acquired irradiated light and the control point TCP of the robot arm 312, and display it on the display unit 345 of the wearable display device 340 and the display unit 353 of the fixed display device 350.

[0058] In addition, in the second embodiment, the processing unit 341 of the wearable display device 340 and the processing unit 351 of the fixed display device 350 each perform processing to superimpose a virtual image B of the irradiation light of a part corresponding to a part where the pixel value of the image of the irradiation light captured by the imaging unit 320 is equal to or greater than a threshold value onto the image of the workpiece 200, and display it on the wearable display device 340 and the fixed display device 350.

[0059] The processing of the processing unit 341 of the wearable display device 340 will be described in detail. As shown in FIG. 15 , a marker MK is placed near the workpiece 200 to align the real coordinates with the coordinates of the wearable display device 340. The marker MK is, for example, a QR code (registered trademark). When a user U wears the wearable display device 340 on his or her head and visually recognizes the marker MK, the imaging unit 343 of the wearable display device 340 captures an image of the marker MK. The processing unit 341 of the wearable display device 340 then aligns the real coordinates with the coordinates of the wearable display device 340 based on information such as the QR code (registered trademark) obtained from the marker MK and the robot coordinate system of the robot 310. The coordinates of the real workpiece 200 are preset based on the robot coordinate system. The coordinates of the real workpiece 200 are also input from the robot controller 330 to the wearable display device 340. Then, the processing unit 341 executes a process of displaying the virtual image B of the irradiated light on the wearable display device 340 so that the virtual image B of the irradiated light is superimposed on the image of the actual workpiece 200. Note that as the position of the head of the user U moves, the position of the virtual image B of the irradiated light displayed on the display unit 345 of the wearable display device 340 also changes.

[0060] The processing of the processing unit 351 of the fixed display device 350 will be described in detail. As shown in FIG. 19 , in the second embodiment, the processing unit 351 of the fixed display device 350 performs processing to display a virtual image B of the irradiated light on the display unit 353 of the fixed display device 350 so that the virtual image B of the irradiated light is superimposed on the image of the three-dimensional model M of the workpiece 200. As described above, the processing unit 351 of the fixed display device 350 displays the virtual image B of the irradiated light superimposed on the image of the three-dimensional model M of the workpiece 200. Furthermore, the image of the three-dimensional model M of the workpiece 200 displayed on the display unit 353 of the fixed display device 350 can be rotated or moved by an operation by the user U. The virtual image B of the irradiated light moves in accordance with the rotation or movement of the image of the three-dimensional model M.

[0061] Next, in step S15, the processing unit 331 of the robot controller 330 determines whether the user U has finished moving the robot arm 312. If the result of step S15 is no, the operations from step S11 to step S14 are repeated. That is, in the second embodiment, the processing unit 331 of the robot controller 330 moves the imaging unit 320 using the robot arm 312, and the processing unit 323 of the imaging unit 320 sequentially acquires coordinates of the irradiated light in images captured by the imaging unit 320, which is moved relative to the workpiece 200. Then, the processing unit 341 of the wearable display device 340 and the processing unit 351 of the fixed display device 350 sequentially superimpose a virtual image B of the linear irradiated light on the image of the workpiece 200 based on the coordinates of the irradiated light sequentially acquired by the imaging unit 320, and display the image B on the display unit 345 of the wearable display device 340 and the display unit 353 of the fixed display device 350. That is, the display unit 345 of the wearable display device 340 and the display unit 353 of the fixed display device 350 display a virtual image B of a line of irradiated light whose position changes from moment to moment. If the answer is yes in step S15, the control process of the appearance inspection system 400 for displaying the imageable range is terminated. The operations from step S11 to step S15 in the second embodiment are performed, for example, before teaching the operation of the robot 10 in the first embodiment.

[0062] (Effects of the Second Embodiment) The visual inspection system 400 includes a processing unit 341 and a processing unit 351 that acquire the coordinates of the irradiated light in an image captured by the imaging unit 320, which moves relative to the workpiece 200. Based on the acquired coordinates of the irradiated light and the control point TCP of the robot arm 312, the visual inspection system 400 superimposes a virtual image B of the irradiated light generated by computer graphics onto the image of the workpiece 200 and displays the superimposed image on the display unit 345 and the display unit 353. As a result, the virtual image B of the irradiated light is displayed on the image of the workpiece 200 based on the actual coordinates of the irradiated light and the actual control point TCP of the robot arm 312. In other words, because the shape and position are based on actual information, errors in the shape and position are small. Therefore, by visually checking the virtual image B of the irradiated light on the image of the workpiece 200, the user U can confirm how the irradiated light hits the workpiece 200, thereby accurately recognizing the imageable range of the workpiece 200 captured by the imaging unit 320.

[0063] The processing unit 331 moves the imaging unit 320 using the robot arm 312. The processing units 341 and 351 each sequentially acquire coordinates of the irradiated light in an image captured by the imaging unit 320, which is moved relative to the workpiece 200, and perform processing to sequentially superimpose a virtual image B of the linear irradiated light on the image of the workpiece 200 and display it on the display units 345 and 353 based on the sequentially acquired coordinates of the irradiated light. This allows the user U to visually recognize the virtual image B of the linear irradiated light and recognize the inspection range based on the current posture of the robot arm 312.

[0064] The processing unit 341 of the wearable display device 340 performs processing to display the virtual image B of the irradiated light on the display unit 345 so as to be superimposed on the image of the actual workpiece 200. As a result, the virtual image B of the irradiated light is displayed so as to be superimposed on the image of the actual workpiece 200, so that the user U can accurately recognize the inspectable range of the actual workpiece 200.

[0065] The display unit 345 of the wearable display device 340 displays a virtual image B generated by computer graphics superimposed on the actual image visually recognized by the user U. The processing unit 341 performs processing to display the virtual image B of the irradiated light on the display unit 345 of the wearable display device 340 so that it is superimposed on the image of the actual workpiece 200 visually recognized by the user U. As a result, since the wearable display device 340 is worn by the user U, the virtual image B of the irradiated light also moves in response to the movement of the user U. As a result, the user U can check the actual workpiece 200 and the inspectable range from various angles.

[0066] The processing unit 351 of the fixed display device 350 performs processing to display the virtual image B of the irradiated light on the display unit 353 so as to be superimposed on the image of the three-dimensional model M of the workpiece 200. As a result, the virtual image B of the irradiated light also moves in response to the rotation and movement of the three-dimensional model M of the workpiece 200 on the display unit 353, so that the workpiece 200 and the inspectable range can be confirmed from various angles.

[0067] Third Embodiment A visual inspection system 400 according to a third embodiment will be described. In the visual inspection system 400 according to the third embodiment, a virtual image B1 of a band-shaped irradiated light is displayed.

[0068] Specifically, as in the second embodiment, the processing unit 331 of the robot controller 330 moves the imaging unit 320 using the robot 310. The processing unit 323 of the imaging unit 320 sequentially acquires coordinates of the irradiated light in images captured by the imaging unit 320, which is moved relative to the workpiece 200. In the third embodiment, as shown in FIG. 21 , the processing unit 341 of the wearable display device 340 performs processing to superimpose a virtual image B1 of the band-shaped irradiated light on the image of the actual workpiece 200 and display it on the display unit 345, based on the sequentially acquired coordinates of the plurality of irradiated light. As shown in FIG. 22 , the processing unit 351 of the fixed display device 350 performs processing to superimpose a virtual image B1 of the band-shaped irradiated light on the image of the three-dimensional model M of the workpiece 200 and display it on the display unit 353, based on the sequentially acquired coordinates of the plurality of irradiated light. Specifically, the processing unit 341 of the wearable display device 340 and the processing unit 351 of the fixed display device 350 store the sequentially acquired coordinates of the plurality of irradiated lights in the storage unit 342 of the wearable display device 340 and the storage unit 352 of the fixed display device 350, respectively. The storage of the coordinates of the plurality of irradiated lights is performed from the start to the end of the movement of the imaging unit 320 by the robot arm 312. After the movement of the imaging unit 320 by the robot arm 312 is completed, the processing unit 341 of the wearable display device 340 reads the coordinates of the plurality of irradiated lights from the storage unit 342, and displays a virtual image B1 of the band-shaped irradiated lights on the display unit 345, superimposing the virtual image B1 of the band-shaped irradiated lights on the image of the actual workpiece 200 based on the read coordinates of the plurality of irradiated lights. The processing unit 351 of the fixed display device 350 performs a similar process, superimposing the virtual image B1 of the band-shaped irradiated lights on the image of the three-dimensional model M of the workpiece 200, and displays the virtual image B1 of the band-shaped irradiated lights on the display unit 353. When the robot arm 312 moves the imaging unit 320 along multiple paths, multiple virtual images B1 of the belt-shaped irradiated light are displayed. The above-described operation of the third embodiment is performed, for example, before teaching the operation of the robot 10 of the first embodiment.

[0069] Effect of Third Embodiment The processing unit 331 moves the imaging unit 320 or the workpiece 200 using a robot. The processing units 341 and 351 each sequentially acquire coordinates of the irradiated light in an image captured by the imaging unit 320, which is moved relative to the workpiece 200, and perform processing to superimpose a virtual image B1 of the band-shaped irradiated light on the image of the workpiece 200 based on the sequentially acquired coordinates of the irradiated light and display the superimposed image on the display units 345 and 353. This allows the user U to visually recognize the virtual image B1 of the band-shaped irradiated light and recognize the inspectable range within the range in which the robot arm 312 is moved.

[0070] (Modifications) The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the description of the above-mentioned embodiments, and further includes all modifications (modifications) within the meaning and scope equivalent to the claims.

[0071] For example, in the first embodiment described above, an example was shown in which the inspection system is an appearance inspection system that inspects the appearance of a workpiece, but the present disclosure is not limited to this. In the present disclosure, the inspection system may be an inspection system that inspects the interior of a workpiece. In this case, the inspection unit may be an ultrasonic flaw detector that inspects the workpiece by transmitting ultrasonic waves to the interior of the workpiece and receiving ultrasonic waves reflected from the interior of the workpiece. Using the ultrasonic flaw detector, it is possible to detect objects such as defects inside the workpiece.

[0072] In addition, in the first embodiment, an example was shown in which the robot was a vertical articulated robot, but the present disclosure is not limited to this. In the present disclosure, the robot may be an industrial robot other than a vertical articulated robot.

[0073] In the first embodiment, an example has been described in which the robot moves the imaging unit to move the inspection unit relative to the workpiece, but the present disclosure is not limited to this. In the present disclosure, the robot may move the workpiece to move the inspection unit relative to the workpiece.

[0074] Furthermore, in the first embodiment, an example was shown in which the processing units of the robot controller, the image processing device, and the wearable display device shared the responsibilities of performing various processes, but the present disclosure is not limited to this. In the present disclosure, the number and configuration of the processing units are not particularly limited. One processing unit may perform the various processes of the first embodiment, or multiple processing units may perform the various processes of the first embodiment. The configuration of the storage unit is also not limited. The configurations of the robot controller, the image processing device, and the wearable display device are also not limited. For example, the wearable display device may be separated into a personal computer in which the processing unit and the storage unit are arranged, and a main body in which the imaging unit and the display unit are arranged.

[0075] In the first embodiment, the target is a defect in a workpiece, but the present disclosure is not limited to this. In the present disclosure, the target may be a teaching point for the operation of a robot, an inspection path along which an inspection unit moves, an inspection area that can be inspected by the inspection unit, etc.

[0076] In addition, in the first embodiment, an example has been described in which the virtual marker is an arrow pointing to the position of the target, but the present disclosure is not limited to this. For example, the virtual marker may be a ring-shaped marker that surrounds the target.

[0077] In the first embodiment, the virtual marker expands or contracts when the user performs an action to expand or contract the virtual marker, but the present disclosure is not limited to this. For example, a virtual switch for expanding or contracting the virtual marker may be displayed on the display unit of the wearable display device.

[0078] In the first embodiment, the wearable display device displays a first virtual switch as a virtual image that switches between displaying and hiding a virtual marker. However, the present disclosure is not limited to this. In the present disclosure, the wearable display device does not necessarily need to display the first virtual switch. In this case, the virtual marker is not hidden.

[0079] In addition, in the first embodiment, an example is shown in which the display mode of the virtual marker differs depending on the type of the plurality of targets, but the present disclosure is not limited to this. For example, a virtual marker having the same display mode may be displayed regardless of the type of the plurality of targets.

[0080] In the first embodiment, the second virtual switch is displayed as a virtual image for selecting the type of object for which the virtual marker is displayed. However, the present disclosure is not limited to this. In the present disclosure, the second virtual switch does not need to be displayed on the wearable display device. In this case, all types of objects are maintained as being displayed on the wearable display device.

[0081] In the first embodiment, the processing unit of the wearable display device performs processing to display a virtual marker indicating the position of the detected object on the wearable display device. However, the present disclosure is not limited to this. For example, as shown in FIG. 11 , the processing unit 61 may perform processing to display a virtual marker indicating the position of the object and a task content 230 for the object 201 as a virtual image on the wearable display device 60. The task content 230 is, for example, content regarding what processing to perform on the defect. The task content 230 is expressed using text or the like. In this way, by displaying the task content 230 on the wearable display device 60, the user U can easily recognize the task content for the object 201 as well as the position of the object 201.

[0082] In addition, although the first embodiment has described an example in which the inspection system is provided with a fixed display device, the present disclosure is not limited to this. For example, the inspection system may be provided with only a wearable display device without a fixed display device.

[0083] In addition, in the first embodiment, an example has been shown in which the wearable display device is a goggle-type display unit worn on the user's head, but the present disclosure is not limited to this. For example, the wearable display device may be worn on a part of the user other than the head.

[0084] In the first embodiment, the imaging unit 63 of the wearable display device 60 captures the image of the marker 202, and the processing unit 61 of the wearable display device 60 aligns the coordinates of the real world with the coordinates of the wearable display device 60 based on information such as a QR code (registered trademark) obtained from the marker 202 and the robot coordinate system of the robot 10. However, the present disclosure is not limited to this. For example, if the marker 202 is not in the field of view of the imaging unit 63 of the wearable display device 60, an arrow A is displayed with the origin of the coordinates preset in the wearable display device 60 as the reference point. Furthermore, multiple markers 202 may be arranged. This prevents the marker 202 from disappearing from the field of view of the imaging unit 63 of the wearable display device 60. Furthermore, the wearable display device 60 may be equipped with a self-position estimation system such as SLAM (Simultaneous Localization and Mapping), and even if the marker 202 is not within the field of view of the imaging unit 63, the coordinates of the real world and the coordinates of the wearable display device 60 may be aligned based on the recognized marker 202. Alternatively, as shown in Fig. 12, the wearable display device 60 may display an arrow A based on a preset origin of coordinates, and then the user U may point with both hands at the workpiece 200 or the stand of the robot 10, and the wearable display device 60 may recognize the coordinates of the intersection of the pointed positions, and the coordinates of the real world and the coordinates of the wearable display device 60 may be aligned based on the recognized coordinates. 13 , the wearable display device 60 may recognize the action of the user U grabbing an arrow A displayed on the wearable display device 60 and moving it to a desired position, and may align the coordinates of the real world with the coordinates of the wearable display device 60 based on the position of the moved arrow A. Alternatively, an external device may acquire position information using GPS or the like and transmit the position information to the wearable display device 60, and the wearable display device 60 may align the coordinates of the real world with the coordinates of the wearable display device 60 based on the transmitted position information.

[0085] Furthermore, in the first embodiment described above, an example has been shown in which the three-dimensional coordinates of the workpiece 200 and the three-dimensional coordinates of the target 201 are output from the robot controller 40 to the wearable display device 60, but the present disclosure is not limited to this. For example, the wearable display device 60 may receive a two-dimensional to three-dimensional coordinate conversion table from the robot controller 40 in advance. Then, the two-dimensional coordinates of the target 201 may be input from the image processing device 50 to the wearable display device 60, and the wearable display device 60 may convert the two-dimensional coordinates of the target 201 into three-dimensional coordinates using the coordinate conversion table. Note that the three-dimensional coordinates of the workpiece 200 are input from the robot controller 40 to the wearable display device 60. Furthermore, an example of the exchange of the two-dimensional coordinates of the target 201 and the three-dimensional coordinates of the workpiece 200 is not limited to this route.

[0086] 14 , a third virtual switch 223 may be displayed on the display unit 65 of the wearable display device 60, and the user U may press the third virtual switch 223 to display the test image 21 on the display unit 65. The test image 21 is stored in a server or the image processing device 50 itself.

[0087] For example, in the second and third embodiments described above, an example was shown in which the robot was a vertical articulated robot, but the present disclosure is not limited to this. In the present disclosure, the robot may be a robot other than a vertical articulated robot.

[0088] In the second and third embodiments, an example is shown in which the robot moves the imaging unit, but the present disclosure is not limited to this. In the present disclosure, the robot may move the workpiece.

[0089] In the second and third embodiments, examples have been shown in which a virtual image of the irradiation light corresponding to a portion of the image of the irradiation light captured by the imaging unit where the pixel value is equal to or greater than a threshold is displayed on the display unit, but the present disclosure is not limited to this. In the present disclosure, the entire image of the irradiation light captured by the imaging unit may be displayed on the display unit.

[0090] In addition, although the second and third embodiments have described examples in which the virtual image of the irradiated light is displayed on both the wearable display device and the fixed display device, the present disclosure is not limited to this. In the present disclosure, the virtual image of the irradiated light may be displayed on only one of the wearable display device and the fixed display device.

[0091] Furthermore, although the second embodiment in which a line-shaped virtual image of irradiated light is displayed on the display unit and the third embodiment in which a band-shaped virtual image of irradiated light is displayed on the display unit have been described as separate embodiments, the present disclosure is not limited thereto. In the present disclosure, a user may switch between the mode in which a line-shaped virtual image of irradiated light is displayed on the display unit and the mode in which a band-shaped virtual image of irradiated light is displayed on the display unit.

[0092] In the second and third embodiments, the wearable display device is a display unit that displays mixed reality, but the present disclosure is not limited to this. In the present disclosure, the wearable display device may be a display unit that displays virtual reality. In this case, the wearable display device displays a virtual image of the irradiated light so as to be superimposed on an image of a three-dimensional model of the workpiece.

[0093] In the second and third embodiments, an image of a three-dimensional model of a workpiece is displayed on a fixed display device, but the present disclosure is not limited to this. In the present disclosure, an image of a real workpiece visually recognized by a user may be displayed on a fixed display device, and a virtual image of the irradiated light may be displayed so as to be superimposed on the real image of the workpiece.

[0094] In addition, although the second and third embodiments have been described above as examples in which the present disclosure is applied to a workpiece having a curved surface, the present disclosure is not limited thereto and may be applied to a flat workpiece that does not have a curved surface.

[0095] In addition, in the second and third embodiments, examples have been shown in which the imaging unit is a line sensor, but the present disclosure is not limited to this. In the present disclosure, the imaging unit may be an area camera that captures a two-dimensional image.

[0096] Furthermore, in the second and third embodiments described above, examples have been shown in which the robot system of the present disclosure is applied to a visual inspection system, but the present disclosure is not limited to this. The robot system of the present disclosure may also be applied to systems other than visual inspection systems.

[0097] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0098] [Aspects] The above-described embodiments are specific examples of the following aspects.

[0099] (Aspect 1) An inspection system comprising: a robot; an inspection unit that inspects a workpiece; a wearable display device that is worn by a user and that displays a virtual image generated by computer graphics superimposed on an image of the real world that is viewed by the user; and a processing unit that performs the following processes: acquiring an inspection image of the workpiece by having the robot move the inspection unit relative to the workpiece and inspecting the workpiece with the inspection unit; detecting an object of the workpiece in the acquired inspection image; and displaying a virtual marker as the virtual image indicating the position of the detected object on the wearable display device in accordance with the movement of the user.

[0100] (Aspect 2) In the inspection system described in Aspect 1, the processing unit performs a process of detecting multiple objects of the workpiece in the acquired inspection image, and a process of changing the display mode of the virtual marker depending on the types of the multiple objects.

[0101] (Aspect 3) In the inspection system according to aspect 2, the processing unit performs processing to display an object type selection switch as the virtual image for selecting the type of object for which the virtual sign is to be displayed from among the plurality of objects.

[0102] (Aspect 4) The inspection system according to any one of Aspects 1 to 3, wherein the target includes a defect in the workpiece.

[0103] (Aspect 5) The inspection system according to any one of Aspects 1 to 4, wherein the virtual marker includes an arrow indicating the position of the object.

[0104] (Aspect 6) The inspection system according to any one of Aspects 1 to 5, wherein when the processing unit detects an action of enlarging or reducing the virtual marker by the user, the processing unit performs a process of enlarging or reducing the virtual marker in accordance with the user's action of enlarging or reducing the virtual marker.

[0105] (Aspect 7) The inspection system according to any one of Aspects 1 to 6, wherein the processing unit performs processing to display a display switch as the virtual image on the wearable display device that switches between displaying and hiding the virtual marker.

[0106] (Aspect 8) The inspection system according to any one of Aspects 1 to 7, wherein the processing unit performs processing to display the work content for the target as the virtual image on the wearable display device.

[0107] (Aspect 9) The inspection system according to any one of Aspects 1 to 8, further comprising a fixed display device that is not worn by the user but is fixedly disposed, and that displays the image viewed by the user through the wearable display device and the virtual marker.

[0108] (Aspect 10) The inspection system according to any one of Aspects 1 to 9, wherein the wearable display device includes a goggle-type display unit worn on the head of the user.

[0109] (Aspect 11) An inspection system according to any one of Aspects 1 to 10, wherein the inspection unit includes an irradiation unit that irradiates the workpiece with irradiation light, and an imaging unit that images the workpiece irradiated with the irradiation light, and the processing unit moves the imaging unit or the workpiece using a robot arm of the robot, acquires coordinates of the irradiation light in an image captured by the imaging unit that is moved relative to the workpiece, and performs processing to superimpose a virtual image of the irradiation light generated by computer graphics on an image of the workpiece based on the acquired coordinates of the irradiation light and a control point of the robot arm, and display the superimposed image on a display unit that includes at least one of the wearable display device and a fixed display device.

[0110] (Aspect 12) The processing unit is an inspection system according to Aspect 11, and is configured to move the imaging unit or the workpiece using the robot arm, sequentially acquire coordinates of the irradiated light in the image captured by the imaging unit that is moved relative to the workpiece, and sequentially superimpose, on the image of the workpiece, a line-shaped virtual image of the irradiated light based on the sequentially acquired coordinates of the irradiated light, and display the image on the display unit.

[0111] (Aspect 13) An inspection system according to aspect 11 or aspect 12, wherein the processing unit moves the imaging unit or the workpiece using the robot arm, sequentially acquires coordinates of the irradiated light in the image captured by the imaging unit that is moved relative to the workpiece, and, based on the sequentially acquired coordinates of the irradiated light, performs processing to superimpose a band-shaped virtual image of the irradiated light on an image of the workpiece and display it on the display unit.

[0112] (Aspect 14) The robot system according to any one of Aspects 11 to 13, wherein the fixed display device displays an image of a three-dimensional model of the workpiece, and the processing unit performs processing to display a virtual image of the irradiated light on the display device of the fixed display device so as to be superimposed on the image of the three-dimensional model of the workpiece.

[0113] (Mode 15) An inspection method comprising: acquiring an inspection image of the work by using a robot to move an inspection unit relative to a work and inspecting the work with the inspection unit; detecting an object of the work in the acquired inspection image; and displaying a virtual marker, which is a virtual image generated by computer graphics indicating the position of the detected object, on a wearable display device worn by the user in accordance with the user's movements so that the virtual marker is superimposed on an image of the real world viewed by the user.

[0114] (Aspect 16) A robot system comprising: a robot arm; an imaging unit including an irradiation unit that irradiates a workpiece with irradiation light and that images the workpiece irradiated with the irradiation light; a display unit that displays the workpiece; and a processing unit that performs processing to move the imaging unit or the workpiece with the robot arm, acquire coordinates of the irradiation light in an image captured by the imaging unit that is moved relative to the workpiece, and superimpose a virtual image of the irradiation light generated by computer graphics on an image of the workpiece based on the acquired coordinates of the irradiation light and a control point of the robot arm, and display the virtual image on the display unit.

[0115] (Aspect 17) The robot system according to aspect 16, wherein the processing unit performs processing to display a virtual image of the irradiated light on the display unit so as to be superimposed on an image of an actual workpiece.

[0116] (Aspect 18) The robot system according to Aspect 17, wherein the display unit includes a display unit of a wearable display device worn by the user, which displays a virtual image generated by computer graphics superimposed on an actual image viewed by the user, and the processing unit performs processing to display the virtual image of the irradiated light on the display unit of the wearable display device so that the virtual image is superimposed on the actual image of the workpiece viewed by the user.

Claims

1. An inspection system comprising: a robot; an inspection unit that inspects a workpiece; a wearable display device worn by a user that displays a virtual image generated by computer graphics superimposed on an image of the real world viewed by the user; and a processing unit that performs the following processes: a process of acquiring an inspection image of the workpiece by having the robot move the inspection unit relative to the workpiece and inspecting the workpiece with the inspection unit; a process of detecting an object of the workpiece in the acquired inspection image; and a process of displaying a virtual marker as the virtual image indicating the position of the detected object on the wearable display device in accordance with the movement of the user so that it is superimposed on the image of the real world viewed by the user.

2. The inspection system described in claim 1, wherein the processing unit performs a process of detecting multiple objects of the workpiece in the acquired inspection image, and a process of varying the display mode of the virtual marker depending on the types of the multiple objects.

3. The inspection system according to claim 2, wherein the processing unit performs processing for displaying an object type selection switch as the virtual image for selecting the type of object on which the virtual sign is to be displayed among the plurality of objects.

4. The inspection system of claim 1, wherein the object includes a defect in the workpiece.

5. The inspection system of claim 1, wherein the virtual marker includes an arrow pointing to a location of the object.

6. The inspection system of claim 1, wherein when the processing unit detects an action of enlarging or reducing the virtual sign by the user, the processing unit performs processing to enlarge or reduce the virtual sign in accordance with the user's enlarging or reducing action.

7. The inspection system according to claim 1, wherein the processing unit performs processing for displaying a display changeover switch as the virtual image on the wearable display device for switching between displaying and not displaying the virtual marker.

8. The inspection system according to claim 1, wherein the processing unit performs processing to display the work content for the object as the virtual image on the wearable display device.

9. The inspection system of claim 1, further comprising a fixed display device that is not worn by the user and is fixedly positioned, and displays the image and the virtual marker that are viewed by the user through the wearable display device.

10. The inspection system according to claim 1, wherein the wearable display device includes a goggle-type display unit worn on the user's head.

11. The inspection system according to claim 1, wherein the inspection unit includes an irradiation unit which irradiates the workpiece with irradiation light, and an imaging unit which images the workpiece irradiated with the irradiation light, and the processing unit performs processing to move the imaging unit or the workpiece with a robot arm of the robot, acquire coordinates of the irradiation light in an image captured by the imaging unit which is moved relative to the workpiece, and superimpose a virtual image of the irradiation light generated by computer graphics on an image of the workpiece based on the acquired coordinates of the irradiation light and a control point of the robot arm, and display the superimposed image on a display unit including at least one of the wearable display device and a fixed display device.

12. The inspection system according to claim 11, wherein the processing unit moves the imaging unit or the workpiece using the robot arm, sequentially acquires coordinates of the illumination light in an image captured by the imaging unit which is moved relative to the workpiece, and sequentially superimposes a line-shaped virtual image of the illumination light on an image of the workpiece based on the sequentially acquired coordinates of the illumination light, and displays the line-shaped virtual image on the display unit.

13. The inspection system according to claim 11, wherein the processing unit moves the imaging unit or the workpiece using the robot arm, sequentially acquires coordinates of the illumination light in an image captured by the imaging unit which is moved relative to the workpiece, and performs processing to superimpose a band-shaped virtual image of the illumination light on an image of the workpiece and display it on the display unit based on the sequentially acquired coordinates of the illumination light.

14. The inspection system described in claim 11, wherein the fixed display device displays an image of a three-dimensional model of the workpiece, and the processing unit performs processing to display a virtual image of the irradiated light on the display unit of the fixed display device so that the virtual image is superimposed on the image of the three-dimensional model of the workpiece.

15. An inspection method comprising: acquiring an inspection image of the work by using a robot to move an inspection unit relative to a work and inspecting the work with the inspection unit; detecting an object of the work in the acquired inspection image; and displaying a virtual marker, which is a virtual image generated by computer graphics indicating the position of the detected object, on a wearable display device worn by the user in accordance with the movements of the user so that it is superimposed on an image of the real world viewed by the user.

Images