Inspection system and inspection method

Abstract

To provide an inspection system capable of saving the labor of a user for confirming the position of an object of a work.SOLUTION: An inspection system 100 includes: a processing unit 51 of an image processing device 50 that performs a process of detecting a target 201 of a work 200 in an inspection image 21 acquired by relatively moving an inspection unit 20 with respect to the work 200 by a robot 10 and inspecting the work 200 by the inspection unit 20; and a processing unit 61 of a wearable display device 60 that performs a process of displaying a virtual marker 210 as a virtual image indicating the position of the detected target 201 on the wearable display device 60 in accordance with the movement of a user U so as to overlap the image of the real world visually recognized by the user U.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] This disclosure relates to an inspection system and an inspection method.

Background Art

[0002] Conventionally, inspection systems are known. For example, Patent Document 1 discloses a surface inspection apparatus that inspects a painted surface of a workpiece. In this surface inspection apparatus, the workpiece is photographed by a camera. Then, scratches on the workpiece are detected from the photographed image of the workpiece. And the position of the detected scratch on the workpiece is converted into the position on the figure of the workpiece displayed on the display device and displayed on the display device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the surface inspection apparatus described in Patent Document 1 above, the position of the scratch on the workpiece is displayed on the figure of the workpiece displayed on the display device. For this reason, since the actual workpiece in the real world and the figure of the workpiece displayed on the display device are different things, the user has to confirm the position of the scratch on the actual workpiece again after confirming the position of the scratch on the workpiece from the figure displayed on the display device. Therefore, there is a problem that it takes time to confirm the position of an object such as a scratch on the workpiece.

[0005] This disclosure has been made to solve the above problems, and one object of this disclosure is to provide an inspection system and an inspection method capable of saving the user's labor for confirming the position of an object on a workpiece.

Means for Solving the Problems

[0006] The inspection system according to the first aspect of this disclosure includes a robot, an inspection unit for inspecting a workpiece, a wearable display device worn by a user that overlays and displays a virtual image generated by computer graphics on a real-world image viewed by the user, a process of moving the inspection unit relative to the workpiece by the robot and inspecting the workpiece by the inspection unit to obtain an inspection image of the workpiece, a process of detecting an object of the workpiece in the obtained inspection image, and a process of displaying, on the wearable display device in accordance with the movement of the user, a virtual marker as a virtual image indicating the position of the detected object so as to overlap the real-world image viewed by the user, and a processing unit for performing the processes.

[0007] In the inspection system according to the first aspect of this disclosure, as described above, there is provided a processing unit that performs a process of displaying, on the wearable display device in accordance with the movement of the user, a virtual marker as a virtual image indicating the position of the detected object so as to overlap the real-world image viewed by the user. As a result, the user can view both the actual workpiece in the real world and the virtual image indicating the position of the object through the wearable display device, so that the user can directly confirm at which position of the actual workpiece in the real world the detected object exists based on the virtual marker. Therefore, it is possible to save the user's labor for confirming the position of the object of the workpiece. Further, since the virtual marker is displayed in accordance with the movement of the user wearing the wearable display device, even when the relative position between the user wearing the wearable display device and the workpiece changes, the positional relationship between the workpiece and the virtual marker does not shift. Therefore, even when the user wearing the wearable display device moves, the position of the object can be appropriately confirmed.

[0008] The inspection method according to the second aspect of this disclosure includes obtaining an inspection image of a workpiece by relatively moving an inspection unit with respect to the workpiece by a robot and inspecting the workpiece with the inspection unit, detecting an object of the workpiece in the obtained inspection image, and generating a virtual label as a virtual image generated by computer graphics indicating the position of the detected object so as to overlap with the image of the real world that the user visually recognizes, and displaying the virtual label on a wearable display device worn by the user in accordance with the movement of the user.

[0009] In the inspection method according to the second aspect of this disclosure, as described above, it includes displaying, on a wearable display device worn by the user, a virtual label as a virtual image generated by computer graphics indicating the position of the detected object so as to overlap with the image of the real world that the user visually recognizes, in accordance with the movement of the user. Thereby, the user can visually recognize both the actual workpiece in the real world and the virtual image indicating the position of the object through the wearable display device, so that the user can directly confirm at which position of the actual workpiece in the real world the detected object exists based on the virtual label. Therefore, it is possible to provide an inspection method that can save the user's labor for confirming the position of the object of the workpiece. Further, since the virtual label is displayed in accordance with the movement of the user wearing the wearable display device, even when the relative position between the user wearing the wearable display device and the workpiece changes, the positional relationship between the workpiece and the virtual label does not shift. Therefore, even when the user wearing the wearable display device moves, it is possible to provide an inspection method that can appropriately confirm the position of the object.

Advantages of the Invention

[0010] As described above, the inspection system and inspection method of the present disclosure can save the user's labor for confirming the position of the object of the workpiece.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments embodying the present disclosure will be described with reference to the drawings.

[0013] (Configuration of Inspection System) Referring to FIGS. 1 and 2, the overall configuration of the inspection system 100 according to one embodiment will be described.

[0014] As shown in FIG. 1, the inspection system 100 is an appearance inspection system that inspects the appearance of the workpiece 200. The workpiece 200 is, for example, a product or part related to automobiles, agricultural machinery, pottery, or household electrical appliances. The workpiece 200 is not particularly limited.

[0015] The inspection system 100 includes a robot 10, an inspection unit 20, a robot controller 40, an image processing device 50, a wearable display device 60, and a fixed display device 70.

[0016] The robot 10 relatively moves the inspection unit 20 with respect to the workpiece 200. The robot 10 is a vertical articulated robot. The robot 10 includes a base portion 11 and an arm portion 12 connected to the base portion 11. The base portion 11 is fixed to an installation surface such as a floor, a wall, or a ceiling. Note that the base portion 11 may be attached to a movable cart. The arm portion 12 has a plurality of joints. Each of the plurality of joints has a servo motor as a drive source. Further, the tip of the arm portion 12 holds the inspection unit 20. The robot 10 moves the inspection unit 20 held at the tip of the arm portion 12 with respect to the fixed workpiece 200 by driving the plurality of joints of the arm portion 12.

[0017] The inspection unit 20 is disposed on the robot 10 and inspects the workpiece 200. The inspection unit 20 is an imaging unit and images the workpiece 200. Specifically, the inspection unit 20 is a line-type camera and is moved along the surface of the workpiece 200 by the robot 10 to perform a scan imaging of the surface of the workpiece 200.

[0018] As shown in FIG. 2, the robot controller 40 controls the operation of the robot 10. The robot controller 40 includes a processing unit 41 and a storage unit 42. The processing unit 41 includes a processor and performs various processes related to the operation of the robot 10. The storage unit 42 includes a non-volatile memory and stores various information such as programs for operating the robot 10.

[0019] The image processing device 50 performs image processing on the images captured by the inspection unit 20. In addition, the image processing device 50 controls the imaging timing by the inspection unit 20. The image processing device 50 includes a processing unit 51 and a storage unit 52. The processing unit 51 includes a processor and performs various processes related to the images captured by the inspection unit 20 and the imaging timing by the inspection unit 20. The storage unit 52 includes a non-volatile memory and stores various information such as programs for performing image processing.

[0020] As shown in FIG. 3, the wearable display device 60 overlays and displays a virtual image generated by computer graphics on an image of the real world that the user U visually recognizes. That is, the wearable display device 60 is a display unit that displays mixed reality. Further, the wearable display device 60 is worn by the user U. Specifically, in the present embodiment, the wearable display device 60 includes a goggle-type display unit that is worn on the head of the user U. As shown in FIG. 2, the wearable display device 60 includes a processing unit 61, a storage unit 62, an imaging unit 63, a sensor 64, and a display unit 65. The processing unit 61 includes a processor and performs various processes such as generating a virtual image. The storage unit 62 includes a non-volatile memory and stores various types of information such as programs for generating virtual images. The imaging unit 63 images an image of the real world. The sensor 64 includes an acceleration sensor, a gyroscope, and the like. The movement of the user U and the like are detected by the sensor 64. The display unit 65 is a see-through type display unit, and the user U can visually recognize an image of the real world such as the robot 10 and the workpiece 200 through the display unit 65. Further, a virtual image generated by computer graphics is overlaid and displayed on the image of the real world on the display unit 65. The virtual image is, for example, a hologram. The details of the virtual image will be described later.

[0021] Further, in the present embodiment, the fixed display device 70 is fixedly arranged without being worn by the user U. Then, the fixed display device 70 displays an image that the user U visually recognizes through the wearable display device 60 and the virtual marker 210. That is, the same image as the image that the user U is visually recognizing is displayed on the fixed display device 70. Specifically, the image of the real world imaged by the imaging unit 63 of the wearable display device 60 and the virtual marker 210 are displayed on the fixed display device 70. The fixed display device 70 is, for example, a liquid crystal display, an organic EL display, or the like.

[0022] (Control Processing of Inspection System) The control processing of the inspection system 100 will be described.

[0023] In step S1 shown in FIG. 4, as shown in FIG. 5, the processing unit moves the inspection unit 20 relative to the workpiece 200 by the robot 10 and inspects the workpiece 200 by the inspection unit 20, thereby performing a process of acquiring an inspection image 21 of the workpiece 200. Specifically, the processing unit 41 of the robot controller 40 operates the robot 10 based on the movement path 13 and performs a process of inspecting the workpiece 200 by the inspection unit 20. The movement path 13 is a path for operating the arm unit 12 of the robot 10 and is generated in plural for inspecting the workpiece 200. Further, the movement path 13 is generated in advance before inspecting the workpiece 200.

[0024] For example, the processing unit 41 receives an instruction of the operation of the arm unit 12 by the user and performs a process of generating the movement path 13 of the robot 10 based on the received instruction. Further, for example, the processing unit 41 performs a process of automatically generating the movement path 13 of the robot 10 regardless of the instruction of the operation of the arm unit 12 by the user. Further, the processing unit 41 performs a process of generating the movement path 13 along the surface of the workpiece 200 such as a curved surface.

[0025] As shown in FIG. 6, the processing unit 51 of the image processing apparatus 50 performs a process of acquiring the inspection image 21 based on the output result of the inspection unit 20. The inspection image 21 is an imaging image of the surface of the workpiece 200 imaged by the inspection unit 20.

[0026] As shown in FIG. 5, the processing unit 51 performs a process of acquiring inspection images 21 by operating the inspection unit 20 to inspect the workpiece 200 at regular distance intervals along the movement path 13. Specifically, the processing unit 51 performs a process of scanning and imaging the workpiece 200 by operating the inspection unit 20 to image the workpiece 200 at regular distance intervals. More specifically, the processing unit 41 performs a process of outputting a pulse signal to the processing unit 51 at regular distance intervals. The processing unit 51 performs a process of outputting a trigger signal to the inspection unit 20 at regular distance intervals based on the pulse signal from the processing unit 41. The inspection unit 20 images the workpiece 200 at regular distance intervals based on the trigger signal. The regular distance interval is the distance interval of the control points of the robot 10. When the inspection unit 20 is an imaging unit, the control point of the robot 10 is set at the imaging focal position of the inspection unit 20. Also, the imaging focal position of the inspection unit 20 is set near the surface of the workpiece 200. The control points of the robot 10 are provided for the inspection unit 20 to perform a process of imaging the workpiece 200.

[0027] The processing unit 41 performs a process of inspecting the workpiece 200 by the inspection unit 20 for all the movement paths 13. Also, the processing unit 51 performs a process of acquiring inspection images 21 for all the movement paths 13.

[0028] In step S2 shown in FIG. 4, as shown in FIG. 6, the processing unit 51 of the image processing apparatus 50 performs a process of detecting the target 201 of the workpiece 200 in the inspection image 21. The processing unit 51 performs a process of detecting the target 201 in the inspection image 21 by performing predetermined image processing on the inspection image 21. In the present embodiment, the target 201 is, for example, a defect such as a scratch, a foreign object, or a dent. The processing unit 51 performs a process of detecting the target 201 in the inspection image 21 for all the inspection images 21. That is, in the present embodiment, when there are a plurality of targets 201 on the workpiece 200, the processing unit 51 of the image processing apparatus 50 performs a process of detecting a plurality of targets 201 of the workpiece 200 in the acquired inspection image 21. Also, the processing unit 51 performs a process of outputting the inspection result of the workpiece 200 to the processing unit 41 of the robot controller 40 and the processing unit 61 of the wearable display device 60.

[0029] In step S3 shown in FIG. 4, the user U wears the wearable display device 60 on the head. Also, as shown in FIG. 7, a marker 202 for aligning the coordinates of the real world and the coordinates of the wearable display device 60 is arranged near the work 200. The marker 202 is, for example, a QR code (registered trademark). With the user U wearing the wearable display device 60 on the head, the imaging unit 63 of the wearable display device 60 images the marker 202 by visually recognizing the marker 202. Then, the processing unit 61 of the wearable display device 60 aligns the coordinates of the real world and the coordinates of the wearable display device 60 based on information such as the QR code (registered trademark) obtained from the marker 202 and the robot coordinate system of the robot 10. Note that the coordinates of the work 200 in the real world are preset based on the robot coordinate system. Also, the coordinates of the work 200 in the real world are input from the robot controller 40 to the wearable display device 60. Specifically, the robot controller 40 holds a coordinate conversion table between two-dimensional and three-dimensional coordinates. The coordinate conversion table is a table that converts the coordinate values in the inspection coordinate system of an inspection image 21, which will be described later, obtained by inspecting the work 200 by the inspection unit 20, into the coordinate values in the robot coordinate system, which is a three-dimensional coordinate system, and the coordinate values in the work coordinate system. Also, the robot controller 40 holds the three-dimensional coordinates of the work 200 and outputs the three-dimensional coordinates of the work 200 to the wearable display device 60. Further, the two-dimensional coordinates of the object 201 are input to the robot controller 40 from the image processing device 50, and the two-dimensional coordinates of the object 201 are converted into three-dimensional coordinates using the coordinate conversion table and output to the wearable display device 60.

[0030] In step S4 shown in FIG. 4, in this embodiment, as shown in FIG. 7, the processing unit 61 of the wearable display device 60 performs a process of displaying a virtual label 210 as a virtual image indicating the position of the detected target 201 so as to overlap the image of the real world that the user U views, in accordance with the movement of the user U on the wearable display device 60. FIG. 7 shows an image that the user U views through the display unit 65 of the wearable display device 60. For example, as shown in FIG. 7, the user U views the real-world workpiece 200 and the robot 10 through the display unit 65. Further, on the display unit 65, an arrow A indicating the position of the detected target 201 is displayed as a virtual label 210 indicating the position of the target 201. Also, as shown in FIG. 8, when the field of view of the user U changes due to the movement of the user U, the position of the arrow A on the display unit 65 also changes in accordance with the movement of the user U. Note that in FIG. 8, the image of the robot 10 is omitted and only the workpiece 200 is shown. Since the coordinates of the real world and the coordinates of the wearable display device 60 are aligned, it is possible to link the change in the position of the arrow A to the movement of the user U. Also, when a target 201 that was hidden in the blind spot of the workpiece 200 and could not be viewed becomes visible due to the movement of the user U, a new arrow A indicating this target 201 is displayed.

[0031] Also, in this embodiment, when a plurality of targets 201 of the workpiece 200 are detected by the processing unit 51 of the image processing device 50, the processing unit 61 of the wearable display device 60 performs a process of varying the display mode of the virtual label 210 according to the types of the plurality of targets 201. Specifically, in FIG. 8, it is assumed that the types of the target 201a and the target 201b are different. The processing unit 61 executes, for example, a process of displaying the color of the arrow A indicating the target 201a and the color of the arrow A indicating the target 201b differently. In FIG. 8, the color difference is distinguished by the presence or absence of hatching.

[0032] In addition, in this embodiment, as shown in FIG. 9, when the processing unit 61 of the wearable display device 60 detects an operation of the user U to enlarge or reduce the virtual label 210, the processing unit 61 performs a process of enlarging or reducing the virtual label 210 in accordance with the enlargement or reduction operation of the user U. For example, when the user U pinches the end of the arrow A with a finger and performs an operation of pulling the virtual label 210 in the outer direction of the arrow A, the processing unit 61 recognizes the operation of the user U from the image captured by the imaging unit 63 of the wearable display device 60. Then, the processing unit 61 performs a process of enlarging the arrow A and displaying it on the wearable display device 60 in accordance with the operation of the user U to pull the virtual label 210 in the outer direction of the arrow A. Further, when the user U pinches the end of the arrow A with a finger and performs an operation of pushing the virtual label 210 in the inner direction of the arrow A, the virtual label 210 is reduced. Note that the above-described enlargement or reduction operation of the user U is an example and is not limited to the above operation.

[0033] In addition, in this embodiment, as shown in FIG. 7, the processing unit 61 of the wearable display device 60 performs a process of displaying a first virtual switch 221 as a virtual image for switching the display and non-display of the virtual label 210 on the wearable display device 60. For example, a rectangular and translucent plate-like member 220 is displayed on the display unit 65 of the wearable display device 60. The first virtual switch 221 is disposed on the member 220. For example, when the user U presses the first virtual switch 221 with a finger in a state where the virtual label 210 is displayed, the processing unit 61 recognizes the operation of the user U, and the virtual label 210 is made non-displayed. Further, when the user U presses the first virtual switch 221 with a finger again, the virtual label 210 is displayed. Note that the switching method based on the pressing of the first virtual switch 221 described above is an example and is not limited to the above switching method. Note that the first virtual switch 221 is an example of a display switching switch.

[0034] Also, in this embodiment, as shown in FIG. 7, the processing unit 61 of the wearable display device 60 performs a process of displaying a second virtual switch 222 as a virtual image for selecting the type of the target 201 on which the virtual label 210 is displayed among a plurality of targets 201. For example, the second virtual switch 222 is arranged on the member 220 displayed on the display unit 65 of the wearable display device 60. Until the user U presses the second virtual switch 222 with a finger, as shown in FIG. 8, all the targets 201 and the arrows A indicating all the targets 201 are displayed. When the user U performs an operation of pressing the second virtual switch 222 with a finger, as shown in FIG. 10, only the target 201a and the arrow A indicating the target 201a are displayed. When the user U performs an operation of pressing the second virtual switch 222 with a finger again, the target 201a and the arrow A indicating the target 201a disappear, and only the target 201b and the arrow A indicating the target 201b are displayed. Further, when the user U performs an operation of pressing the second virtual switch 222 with a finger, all the targets 201 and the arrows A indicating all the targets 201 are displayed. Note that the method of switching the displayed targets 201 and the arrows A based on the pressing of the second virtual switch 222 described above is an example and is not limited to the above-described switching method. Note that the second virtual switch 222 is an example of a target type selection switch.

[0035] Also, in this embodiment, the processing unit 61 of the wearable display device 60 executes a process of causing the wearable display device 60 to display an image and a virtual label 210 that are visually recognized by the user U through the wearable display device 60. That is, on the wearable display device 60, an image of the real-world workpiece 200 and the robot 10 captured by the imaging unit 63 of the wearable display device 60 and an arrow A indicating the position of the target 201 are displayed.

[0036] (Effect of this embodiment) The inspection system 100 includes a processing unit 61 that performs a process of causing the wearable display device 60 to display a virtual label 210 as a virtual image indicating the position of the detected target 201 so as to overlap the image of the real world that the user U views, in accordance with the movement of the user U. As a result, the user U can view both the actual workpiece 200 in the real world and the virtual label 210 indicating the position of the target 201 through the wearable display device 60. Therefore, the user U can directly confirm at which position of the actual workpiece 200 in the real world the detected target 201 exists based on the virtual label 210. Thus, the labor of the user U for confirming the position of the target 201 on the workpiece 200 can be saved. Further, since the virtual label 210 is displayed in accordance with the movement of the user U wearing the wearable display device 60, even when the relative position between the user U wearing the wearable display device 60 and the workpiece 200 changes, the positional relationship between the workpiece 200 and the virtual label 210 does not shift. Therefore, even when the user U wearing the wearable display device 60 moves, the position of the target 201 can be appropriately confirmed.

[0037] The target 201 includes a defect of the workpiece 200. Thus, the labor of the user U for confirming the position of the defect of the workpiece 200 can be saved.

[0038] The virtual label 210 includes an arrow A indicating the position of the target 201. As a result, since the position of the target 201 is indicated by the arrow A, the user U can easily recognize the position of the target 201.

[0039] When the processing unit 61 detects an operation of enlarging or reducing the virtual label 210 by the user U, the processing unit 61 performs a process of enlarging or reducing the virtual label 210 in accordance with the enlarging or reducing operation of the user U. Thereby, when it is difficult for the user U to visually recognize the virtual label 210 because it is small, the virtual label 210 can be made easier to visually recognize by enlarging the virtual label 210. Further, when it is difficult for the user U to visually recognize the workpiece 200 because the virtual label 210 is large, the workpiece 200 can be made easier to visually recognize by reducing the virtual label 210. Further, when a plurality of virtual labels 210 overlap each other because the virtual label 210 is large, the overlap between the virtual labels 210 can be eliminated by reducing the virtual label 210.

[0040] The processing unit 61 performs a process of displaying a first virtual switch 221 as a virtual image for switching the display and non-display of the virtual label 210 on the wearable display device 60. Thereby, when the workpiece 200 is hidden behind the virtual label 210 and it is difficult to visually recognize the workpiece 200, the workpiece 200 can be made easier to visually recognize by making the virtual label 210 non-displayed. Further, by the user U operating the first virtual switch 221, the virtual label 210 that has been once made non-displayed can be displayed again.

[0041] The processing unit 51 performs a process of detecting a plurality of targets 201 of the workpiece 200 in the acquired inspection image 21. Further, the processing unit 61 performs a process of varying the display mode of the virtual label 210 according to the types of the plurality of targets 201. Thereby, since the display mode of the virtual label 210 varies according to the types of the plurality of targets 201, the user U can easily discriminate the types of the targets 201.

[0042] The processing unit 61 performs a process of displaying a second virtual switch 222 as a virtual image for selecting the type of the target 201 on which the virtual label 210 is to be displayed among the plurality of targets 201. Thereby, by the user U operating the second virtual switch 222, only the virtual label 210 for the target 201 of the type desired by the user U can be displayed.

[0043] The inspection system 100 includes a fixed display device 70 that is fixedly arranged without being worn by the user U and displays an image and a virtual marker 210 that the user U visually recognizes through the wearable display device 60. As a result, workers other than the user U can visually recognize the target 201 of the workpiece 200 and the virtual marker 210 indicating the target 201.

[0044] The wearable display device 60 includes a goggle-type display unit that is worn on the head of the user U. As a result, since the wearable display device 60 is worn on the head of the user U, even when the viewing angle changes due to the movement of the head of the user U, the virtual marker 210 can be moved in accordance with the change in the viewing angle.

[0045] (Modification example) It should be noted that the embodiments disclosed this time should be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the claims rather than the description of the above embodiments, and further includes all changes (modification examples) within the meaning and scope equivalent to the claims.

[0046] For example, in the above embodiment, an example where the inspection system is a visual inspection system for inspecting the appearance of a workpiece is shown, but the present disclosure is not limited to this. In the present disclosure, the inspection system may be an inspection system for inspecting the inside of a workpiece. In this case, the inspection unit may be an ultrasonic flaw detection unit that inspects the workpiece by transmitting ultrasonic waves into the workpiece and receiving the ultrasonic waves reflected inside the workpiece. By using the ultrasonic flaw detection unit, it is possible to detect targets such as defects inside the workpiece.

[0047] Also, in the above embodiment, an example where the robot is a vertically articulated robot is shown, but the present disclosure is not limited to this. In the present disclosure, the robot may be an industrial robot other than a vertically articulated robot.

[0048] In the above-described embodiment, an example was shown in which the robot moves the imaging unit to relatively move the inspection unit with respect to the workpiece. However, the present disclosure is not limited to this. In the present disclosure, the robot may relatively move the inspection unit with respect to the workpiece by moving the workpiece.

[0049] In the above-described embodiment, an example was shown in which the processing unit of the robot controller, the processing unit of the image processing apparatus, and the processing unit of the wearable display device share and perform various processes. However, the present disclosure is not limited to this. In the present disclosure, the number and configuration of the processing units are not particularly limited. One processing unit may perform various processes of the above-described embodiment, or a plurality of processing units may perform various processes of the above-described embodiment. Also, the configuration of the storage unit is not limited. Also, the configurations of the robot controller, the image processing apparatus, and the wearable display device are not limited. For example, the wearable display device may be separated into a personal computer in which the processing unit and the storage unit are arranged, and a main body unit in which the imaging unit and the display unit are arranged.

[0050] In the above-described embodiment, an example was shown in which the object is a defect of the workpiece. However, the present disclosure is not limited to this. In the present disclosure, the object may be a teaching point of the operation of the robot, an inspection path along which the inspection unit moves, an inspectable region of the inspection unit, or the like.

[0051] In the above-described embodiment, an example was shown in which the virtual marker is an arrow indicating the position of the object. However, the present disclosure is not limited to this. For example, the virtual marker may be an annular marker surrounding the object.

[0052] In the above-described embodiment, an example was shown in which the virtual marker expands or contracts when the user performs an operation of expanding or contracting the virtual marker. However, the present disclosure is not limited to this. For example, a virtual switch for expanding or contracting the virtual marker may be displayed on the display unit of the wearable display device.

[0053] In addition, in the above-described embodiment, an example in which a first virtual switch as a virtual image for switching the display and non-display of a virtual label is displayed on the wearable display device has been shown. However, the present disclosure is not limited to this. In the present disclosure, the first virtual switch may not be displayed on the wearable display device. In this case, the virtual label is not made non-displayed.

[0054] In addition, in the above-described embodiment, an example in which the display mode of the virtual label differs according to the types of a plurality of objects has been shown. However, the present disclosure is not limited to this. For example, virtual labels having the same display mode may be displayed regardless of the types of a plurality of objects.

[0055] In addition, in the above-described embodiment, an example in which a second virtual switch as a virtual image for selecting the type of object on which the virtual label is to be displayed is displayed has been shown. However, the present disclosure is not limited to this. In the present disclosure, the second virtual switch may not be displayed on the wearable display device. In this case, the state in which all types of objects are displayed on the wearable display device is maintained.

[0056] In addition, in the above-described embodiment, an example in which the processing unit of the wearable display device performs processing to cause the wearable display device to display a virtual label indicating the detected position of the object has been shown. However, the present disclosure is not limited to this. For example, as shown in FIG. 11, the processing unit 61 may perform processing to cause the wearable display device 60 to display the work content 230 for the object 201 as a virtual image together with the virtual label indicating the position of the object. The work content 230 is, for example, the content regarding what kind of processing is to be performed on the defect. Further, the work content 230 is represented by characters or the like. Thus, by displaying the work content 230 on the wearable display device 60, the user U can easily recognize the work content for the object 201 together with the position of the object 201.

[0057] In addition, in the above-described embodiment, an example in which the inspection system is provided with a fixed display device has been shown. However, the present disclosure is not limited to this. For example, the inspection system may be provided only with a wearable display device without being provided with a fixed display device.

[0058] In the above-described embodiment, an example where the wearable display device is a goggle-type display unit worn on the user's head is shown, but the present disclosure is not limited to this. For example, the wearable display device may be worn on a part other than the user's head.

[0059] In the above-described embodiment, an example is shown in which the imaging unit 63 of the wearable display device 60 images the marker 202, and the processing unit 61 of the wearable display device 60 aligns the coordinates of the real world and the coordinates of the wearable display device 60 based on information such as a QR code (registered trademark) obtained from the marker 202 and the robot coordinate system of the robot 10. However, the present disclosure is not limited to this. For example, when the marker 202 is not in the field of view of the imaging unit 63 of the wearable display device 60, the arrow A is displayed based on the origin of the coordinates preset in the wearable display device 60. Also, a plurality of markers 202 may be arranged. Thereby, it is suppressed that the marker 202 disappears from the field of view of the imaging unit 63 of the wearable display device 60. Further, a self-position estimation system such as SLAM (Simultaneous Localization and Mapping) may be mounted on the wearable display device 60, and even when the marker 202 is not in the field of view of the imaging unit 63, the coordinates of the real world and the coordinates of the wearable display device 60 may be aligned based on the marker 202 that has been recognized once. As shown in FIG. 12, the arrow A is displayed based on the origin of the coordinates preset in the wearable display device 60, and then the user U points at the workpiece 200 or the pedestal of the robot 10 with the fingers of both hands, and the wearable display device 60 recognizes the coordinates of the intersection of the pointed positions, and based on the recognized coordinates, the coordinates of the real world and the coordinates of the wearable display device 60 may be aligned. As shown in FIG. 13, the wearable display device 60 recognizes an operation in which the user U grasps the arrow A displayed on the wearable display device 60 and moves it to a desired position, and based on the position of the moved arrow A, the coordinates of the real world and the coordinates of the wearable display device 60 may be aligned. Further, an external device may acquire position information by GPS or the like, transmit the position information to the wearable display device 60, and align the coordinates of the real world and the coordinates of the wearable display device 60 based on the transmitted position information.

[0060] Further, in the above embodiment, an example in which the three-dimensional coordinates of the workpiece 200 and the three-dimensional coordinates of the object 201 are output from the robot controller 40 to the wearable display device 60 has been shown, but the present disclosure is not limited thereto. For example, the wearable display device 60 may have previously received a two-dimensional and three-dimensional coordinate conversion table from the robot controller 40. Then, the two-dimensional coordinates of the object 201 may be input from the image processing device 50 to the wearable display device 60, and the wearable display device 60 may convert the two-dimensional coordinates of the object 201 into three-dimensional coordinates using the coordinate conversion table. The three-dimensional coordinates of the workpiece 200 are input from the robot controller 40 to the wearable display device 60. Further, the example of the transfer of the two-dimensional coordinates of the object 201 and the three-dimensional coordinates of the workpiece 200 is not limited to such a path.

[0061] Also, as shown in FIG. 14, a third virtual switch 223 may be displayed on the display unit 65 of the wearable display device 60, and the inspection image 21 may be displayed on the display unit 65 when the user U presses the third virtual switch 223. The inspection image 21 is stored in the server or in the image processing device 50 itself.

[0062] The functions of the elements disclosed in this specification can be executed using a circuit or a processing circuit including a general-purpose processor, a dedicated processor, an integrated circuit, an ASIC (Application Specific Integrated Circuits), a conventional circuit, and / or a combination thereof, which are configured or programmed to execute the disclosed functions. Since a processor includes transistors and other circuits, it is regarded as a processing circuit or a circuit. In the present disclosure, a circuit, a unit, or a means is hardware that executes the listed functions or hardware programmed to execute the listed functions. The hardware may be the hardware disclosed in this specification or other known hardware that is programmed or configured to execute the listed functions. When the hardware is a processor considered to be a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used for the configuration of the hardware and / or the processor.

[0063] [Aspect] The above-described embodiments are specific examples of the following aspects.

[0064] (Aspect 1) A robot, An inspection unit for inspecting a workpiece, A wearable display device that overlays and displays a virtual image generated by computer graphics on a real-world image visible to a user and is worn by the user, A process of obtaining an inspection image of the workpiece by relatively moving the inspection unit with respect to the workpiece by the robot and inspecting the workpiece by the inspection unit, A process of detecting a target of the workpiece in the obtained inspection image, A process of displaying a virtual label as the virtual image indicating the position of the detected target on the wearable display device in accordance with the movement of the user, and a processing unit. An inspection system comprising:

[0065] (Aspect 2) The processing unit performs processing to detect a plurality of the targets of the workpiece in the acquired inspection image, and processing to vary the display mode of the virtual label according to the types of the plurality of targets, and is the inspection system according to Aspect 1.

[0066] (Aspect 3) The processing unit performs processing to display a target type selection switch as the virtual image that selects the type of the target on which the virtual label is to be displayed among the plurality of targets, and is the inspection system according to Aspect 2.

[0067] (Aspect 4) The target includes a defect of the workpiece, and is the inspection system according to any one of Aspects 1 to 3.

[0068] (Aspect 5) The virtual label includes an arrow indicating the position of the target, and is the inspection system according to any one of Aspects 1 to 4.

[0069] (Aspect 6) When the processing unit detects an operation of enlarging or reducing the virtual label by the user, the processing unit performs processing to enlarge or reduce the virtual label in accordance with the operation of enlarging or reducing by the user, and is the inspection system according to any one of Aspects 1 to 5.

[0070] (Aspect 7) The processing unit performs processing to display a display switching switch as the virtual image that switches the display and non-display of the virtual label on the wearable display device, and is the inspection system according to any one of Aspects 1 to 6.

[0071] (Aspect 8) The processing unit performs processing to display the work content for the target as the virtual image on the wearable display device, and is the inspection system according to any one of Aspects 1 to 7.

[0072] (Aspect 9) The inspection system according to any one of Aspects 1 to 8, comprising a fixed display device that is fixedly arranged without being worn by the user and displays an image that the user visually recognizes through the wearable display device and the virtual marker.

[0073] (Aspect 10) The inspection system according to any one of Aspects 1 to 9, wherein the wearable display device includes a goggle-type display unit worn on the user's head.

[0074] (Aspect 11) Moving the inspection unit relative to the workpiece by a robot, inspecting the workpiece by the inspection unit, thereby acquiring an inspection image of the workpiece; Detecting an object of the workpiece in the acquired inspection image; Performing a process of displaying, on a wearable display device worn by the user, a virtual marker as a virtual image generated by computer graphics indicating the position of the detected object so as to overlap an image of the real world that the user visually recognizes, in accordance with the movement of the user. The inspection method includes these steps.

Description of Signs

[0075] 10 Robot 20 Inspection unit 21 Inspection image 51 Processing unit 60 Wearable display device 61 Processing unit 70 Fixed display device 100 Inspection system 200 Workpiece 201 Object 210 Virtual marker 221 First virtual switch (display switching switch) 222 Second virtual switch (object type selection switch) 230 Work content A Arrow U User

Claims

1. A robot, An inspection unit for inspecting a workpiece, A wearable display device that overlays and displays a virtual image generated by computer graphics on an image of the real world that the user visually recognizes, and is worn by the user, A process of obtaining an inspection image of the workpiece by relatively moving the inspection unit with respect to the workpiece by the robot and inspecting the workpiece by the inspection unit, A process of detecting an object of the workpiece in the acquired inspection image, A processing unit that performs a process of displaying a virtual marker as the virtual image indicating the position of the detected object so as to overlap the image of the real world that the user visually recognizes on the wearable display device according to the movement of the user. An inspection system comprising:

2. The processing unit, A process of detecting a plurality of objects of the workpiece in the acquired inspection image, The inspection system according to claim 1, wherein the processing unit performs a process of varying a display mode of the virtual marker according to types of the plurality of objects.

3. The inspection system according to claim 2, wherein the processing unit performs a process of displaying a target type selection switch as the virtual image for selecting a type of the object on which the virtual marker is to be displayed among the plurality of objects.

4. The inspection system according to claim 1, wherein the object includes a defect of the workpiece.

5. The inspection system according to claim 1, wherein the virtual marker includes an arrow indicating the position of the object.

6. The inspection system according to claim 1, wherein when the processing unit detects an operation of enlarging or reducing the virtual marker by the user, the processing unit performs a process of enlarging or reducing the virtual marker according to the enlarging or reducing operation of the user.

7. The inspection system according to claim 1, wherein the processing unit performs a process of displaying a display switching switch as the virtual image for switching display and non-display of the virtual marker on the wearable display device.

8. The inspection system according to claim 1, wherein the processing unit performs a process of displaying work content for the object as the virtual image on the wearable display device.

9. The inspection system according to claim 1, further comprising a fixed display device that is fixedly arranged without being worn by the user and displays an image and the virtual marker that the user visually recognizes through the wearable display device.

10. The inspection system according to claim 1, wherein the wearable display device includes a goggle-type display unit worn on the head of the user.

11. Moving a inspection unit relative to a workpiece by a robot, and acquiring an inspection image of the workpiece by inspecting the workpiece with the inspection unit; Detecting an object of the workpiece in the acquired inspection image; A virtual label as a virtual image generated by computer graphics indicating the position of the detected object is displayed on a wearable display device worn by the user in accordance with the movement of the user so as to overlap an image of the real world viewed by the user. An inspection method comprising:

Images