Mask image generating method, inspection method and inspection device

Abstract

To generate a mask image generation method, an inspection method, and an inspection apparatus that can generate a mask image with high accuracy even when imaging a three-dimensional curved surface is inclined.SOLUTION: A plurality of virtual images are acquired when postures of three-dimensional data is changed to a plurality of specific posture respectively and an object to be imaged is imaged by a virtual imaging unit.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a mask image generating method, an inspection method, and an inspection apparatus. [Background technology]

[0002] In the inspection method described in Patent Document 1, a virtual image of what would happen if the workpiece were photographed is obtained based on the position and direction of the arm of an articulated robot, three-dimensional data representing the shape of the workpiece, and the position and direction of the workpiece, and a mask image that masks part of the actual image is generated based on this virtual image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2012-22600 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the conventional inspection method described above, when imaging a three-dimensional curved surface at an angle, depending on the posture, if the inspection area is shaded by unevenness on the surface of the object being imaged, there is a risk that a mask image cannot be generated accurately. An object of the present invention is to provide a mask image generating method, an inspection method, and an inspection apparatus that can generate a mask image with high accuracy even when imaging a three-dimensional curved surface at an angle. [Means for solving the problem]

[0005] A mask image generating method according to one embodiment of the present invention changes the orientation of three-dimensional data to a plurality of specific orientations, and obtains a plurality of virtual images when an object is imaged by a virtual imaging unit. [Effects of the Invention]

[0006] Therefore, in the present invention, even when an image of a three-dimensional curved surface is captured at an angle, a mask image can be generated with high accuracy. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram of a visual inspection device 1 according to a first embodiment. [Figure 2] 1 is a flowchart showing the flow of a result candidate detection process in the visual inspection method of the first embodiment. [Figure 3] FIG. 1 shows each captured image (views 1 to 25). [Figure 4] The mask images of each view are obtained when the processed surfaces 11 and 12 are the inspection areas, and the casting surface 13 and background 14 are the non-inspection areas. [Figure 5] The mask images are of each view when the casting surface 13 is the inspection area and the processed surfaces 11 and 12 and the background 14 are the non-inspection areas. [Figure 6] FIG. 10 is a diagram showing the detection of defect candidates for each view by the first AI. [Figure 7] FIG. 10 is a diagram showing grouping of identical defect candidates in each view. [Figure 8] 1 is a flowchart showing the flow of a mask image generation process according to the first embodiment. [Figure 9] 1A and 1B are diagrams illustrating problems with a conventional mask image generation method. [Figure 10] 10A and 10B are diagrams illustrating the removal of an unrecognizable inspection area in the mask image generating method of the first embodiment. [Figure 11] 10A and 10B are diagrams illustrating a mismatch between a non-mask area of ​​a mask image and an inspection area of ​​a captured image. [Figure 12] 10A and 10B are diagrams illustrating a reduction correction effect of a non-inspection region in the mask image generating method of the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] [Embodiment 1] FIG. 1 is a schematic diagram of a visual inspection device 1 according to the first embodiment. The visual inspection device 1 (hereinafter referred to as the inspection device 1) includes a robot 2, a camera (imaging unit) 3, a lighting device 4, an image processing device 5, and a control device 6. The robot 2 is an articulated robot and has a hand 8 that holds an object to be imaged, either a raw engine piston or a finished engine piston (hereinafter simply referred to as a piston) 7. The camera 3 is supported with its lens facing vertically downward. The lighting device 4 irradiates light onto the top surface 7a of the piston 7 and includes a dome 9 and a ring light 10. The dome 9 houses the piston 7 and the ring light 10. The dome 9 reflects and diffuses the light emitted from the ring light 10. The dome 9 is located vertically below the camera 3 and is fixed integrally with the camera 3. The camera 3 images the top surface 7a of the piston 7 through an opening at the top of the dome 9. The ring light 10 is an annular LED light arranged to surround the piston 7. The ring light 10 is supported by a support device (not shown) so as to be rotatable around the central axis of the camera 3 relative to the camera 3 and the dome 9 .

[0009] The image processing device 5 extracts defect candidate portions from the surface image (hereinafter also referred to as the captured image) of the crown surface 7a captured by the camera 3, and performs image processing to generate, for example, an 8-bit (256 gradations) grayscale image from the defect candidate portions. The control device 6 outputs a command to the robot 2 to change the relative viewing angle (attitude angle) of the crown surface 7a of the piston 7 with respect to the camera 3. Here, when a first axis L1 is a line that passes through the center of the piston 7 and extends vertically, and a second axis L2 is a line that is inclined relative to the first axis L2, each viewing angle is obtained by rotating the piston 7 so that the second axis L2 rotates about the first axis L1. The control device 6 also outputs a command to the camera 3 to capture an image of the top surface 7a. The control device 6 captures images of the top surface 7a from 25 angles while changing the viewing angle, acquiring 25 grayscale images (views 1 to 25). The control device 6 then determines the presence or absence of defects (blowholes, scratches, dents, etc.) based on pre-stored learning results, from changes in the brightness distribution of the same defect candidate in each view. The control device 6 stores the learning results obtained by machine learning using multiple defective sample images. The machine learning is learning using a neural network, and in the first embodiment, learning by deep learning is adopted. The multiple defective sample images are generated by converting a previously formed three-dimensional (i.e., sterically formed) defect model into a point cloud, and combining this with a surface image of the piston 7.

[0010] The defect is modeled as a geometric envelope, and a two-dimensional image of the defect shape is generated by adding brightness to the defect model. The brightness of the defect model is a brightness distribution within a predetermined range that includes the defect and a predetermined area surrounding the defect. The defects specifically include voids, scratches, dents, and bulges. Each defect includes multiple defects of different sizes. The brightness distribution is calculated based on a predetermined illumination direction angle for the coordinates of a point cloud on the surface within a predetermined range converted from the defect model, a predetermined imaging direction angle for the point cloud coordinates, and an angle of the normal direction to the defect on a plane including the illumination direction and imaging direction. The control device 6 generates a two-dimensional image of the defect shape based on the defect model, and combines the two-dimensional image with an image of the crown surface 7a of the piston 7 to generate a defective sample image.

[0011] The crown surface 7a of the piston 7 includes an as-cast cast surface and a machined surface. The two surfaces have different shapes and sizes of defects due to differences in characteristics such as surface roughness, and therefore must be inspected under different conditions. Therefore, the image processing device 5 generates a mask image for masking a non-inspection region (second region) for each view of the crown surface 7a captured by the camera 3, and then inspects the inspection region (first region) by masking the surface image with the mask image. When the inspection region is the machined surface, the image processing device 5 generates a mask image for masking the cast surface and background (portions other than the crown surface 7a). On the other hand, when the inspection region is the cast surface, the image processing device 5 generates a mask image for masking the machined surface and background.

[0012] FIG. 2 is a flowchart showing the flow of the result candidate detection process in the visual inspection method of the first embodiment. In step S1, a checkerboard-like calibration pattern is used to perform camera calibration, and the intrinsic parameters K and extrinsic parameters (rotational parameters R) of camera 3 are calculated. N , translational element T N , N=1 to 25) and distortion coefficients are estimated. The internal parameter K is a transformation parameter from the camera coordinate system to the image coordinate system, and is dependent on the camera and lens. The rotation element R N and the translation element T N is a transformation parameter from the world coordinate system to the camera coordinate system, and depends on the viewing angle. By estimating each parameter, the perspective projection transformation formula P N ≡K(R N |T N ) is obtained. This step is performed only the first time, and is not performed after the perspective projection transformation formula is obtained.

[0013] In step S2, the crown surface 7a is imaged at 25 angles while changing the viewing angle, and 25 captured images (views 1 to 25) are obtained, which are actual images, as shown in FIG. 3. The shape of the crown surface 7a will be described from the captured images in FIG. 3. The crown surface 7a has a first machined surface 11 and a pair of second machined surfaces 12, 12. The first machined surface 11 is a machined circular recess located in the center of the crown surface 7a. The pair of second machined surfaces 12, 12 are formed by chamfering and are located symmetrically across the first machined surface 11. The approximately annular region of the crown surface 7a, excluding the machined surfaces 11, 12, is a cast surface 13. In FIG. 3, the area other than the crown surface 7a is designated as a background 14. In step S3, a mask image generation process is performed to generate a mask image for each view. Fig. 4 shows a mask image for each view when the machining surfaces 11 and 12 are the inspection areas, and the casting surface 13 and background 14 are the non-inspection areas. On the other hand, Fig. 5 shows a mask image for each view when the casting surface 13 is the inspection area, and the machining surfaces 11 and 12 and background 14 are the non-inspection areas. Details of the mask image generation process will be described later. Below, a case where the machining surfaces 11 and 12 are the inspection areas, and the casting surface 13 and background 14 are the non-inspection areas will be described. In step S4, the captured image for each view is masked with the generated mask image, and defect candidates for each view (view 1 to view 25) are detected using the first AI, as shown in Fig. 6. Fig. 7 shows grouping of identical defect candidates for each view, and time-series data of feature amounts (contrast) ordered according to multiple different viewing angles is created from images around each identical defect candidate.

[0014] Next, the mask image generation process of the first embodiment will be described. FIG. 8 is a flowchart showing the flow of the mask image generation process according to the first embodiment. In step S21, the three-dimensional data (facet information of 3D CAD) used in the design of the piston 7 is used to create a shape model of the crown surface 7a on the image coordinate system. In step S22, the inspection area is designated on each of the processed surfaces 11 and 12 or the casting surface 13. In step S23, 0 is substituted into the second variable k. In step S24, it is determined whether the second variable k is smaller than the total number of angles, 25. If YES, the process proceeds to step S25, and if NO, the process ends. In step S25, the second variable k is incremented. When the second variable k is incremented, the angle at which the mask image is generated is switched. In step S26, the orientation of the shape model of the crown surface 7a on the image coordinate system is changed according to the current angle, thereby obtaining a virtual image of the crown surface 7a captured by a virtual camera at the current angle.

[0015] In step S27, an inspection area is extracted from the virtual image, and a mask image is generated in which the area other than the extracted inspection area is used as a mask area. In the mask image, the inspection area is made transparent, and the mask area uses a single color that can be distinguished from the crown surface 7a and background when superimposed on the actual captured image. In step S28, the boundary (outline) between the masked and non-masked regions in the generated mask image is corrected in a direction that reduces the non-masked region. When reducing the non-masked region, the reduction ratio is the same vertically and horizontally. The reduction ratio is set to an optimal value taking into account factors such as chucking error when holding the piston 7 with the hand 8 and dimensional tolerances. Furthermore, the non-masked region is reduced so that the envelope curve obtained when the inspection region of the virtual image is rotated by a predetermined angle (e.g., 2 to 3 degrees) around the center of the shape model of the crown surface 7a on the image coordinate system does not overlap with the corrected non-masked region. For example, in the case of Figure 5, of the two boundaries between the inspection region and the non-inspection region, the outer boundary of the inspection region is moved outward, and the inner boundary of the inspection region is moved inward. In step S29, the generated mask image is saved as a mask image of the corresponding view.

[0016] Next, the effects of the first embodiment will be described. In conventional mask image generation methods, a virtual image of the inspection area on an image coordinate system is acquired based on the three-dimensional data of the object to be imaged and the position and orientation of the object to be imaged, and a mask image is generated by treating the area other than the inspection area as a non-inspection area. Therefore, in conventional mask image generation methods, when a mask image (only the non-mask area is shown in FIG. 9 for ease of viewing) is superimposed on an actual captured image as shown in FIG. 9, the non-inspection area on the image, which is invisible due to the shadow caused by the unevenness of the surface of the object to be imaged (shown by hatching in FIG. 9), may be erroneously recognized as the inspection area in the composite image. Therefore, conventional mask image generation methods may not be able to generate a mask image with high accuracy, particularly when imaging a three-dimensional curved surface at an angle.

[0017] In contrast, in the mask image generation method of the first embodiment, as shown in FIG. 10, a three-dimensional model (crown surface shape model) of the crown surface 7a is prepared on an image coordinate system, and the posture of the three-dimensional model is changed according to each angle to obtain a virtual image of the crown surface 7a captured by a virtual camera. Next, an inspection area is extracted from the virtual image, and a mask image is generated by masking areas other than the inspection area. Here, invisible areas do not appear in the virtual image. Therefore, by generating a mask image from the virtual image, a mask image can be generated in which the shadowed parts of the inspection area, i.e., the invisible areas, have been removed (masked areas). As a result, a mask image can be generated with high accuracy, particularly when capturing an image of a three-dimensional curved surface at an angle.

[0018] Here, because the mask image is generated based on the design data of the crown surface 7a, the masked and non-masked regions always have the same shape and position at the same angle. On the other hand, captured images of the actual crown surface 7a vary in shape and position due to individual differences in the piston 7, chucking errors, and the like, even at the same angle. Therefore, when the mask image is superimposed on the captured image, as shown in FIG. 11 , a problem occurs in that the portion of the inspection region of the captured image that overlaps with the masked region of the mask image (the hatched portion in FIG. 11 ) is not inspected. Therefore, in the first embodiment, the non-inspection region in the virtual image is reduced in a direction away from the inspection region, and then the non-inspection region is masked (set as a mask region) from the virtual image to generate a mask image. More specifically, the portion of the non-inspection region outside the inspection region is reduced outward, and the portion inside the inspection region is reduced inward. Fig. 12 shows a reduced mask area in the mask image of Fig. 11. As shown in Fig. 12, even if there is variation in the shape and position of the crown surface 7a on the captured image, the overlap between the inspection area and the mask area is reduced. As a result, it is possible to prevent part of the inspection area from being left uninspected.

[0019] In this case, the reduction ratio of the non-inspection area is the same vertically and horizontally. If the vertical and horizontal reduction ratios are different, the inspection area and the mask area are more likely to overlap in the direction with the larger vertical or horizontal reduction ratio. Therefore, by reducing the non-inspection area by the same reduction ratio vertically and horizontally, overlap between the inspection area and the mask area can be effectively suppressed. The non-masked area is reduced so that the envelope of the inspection area in the virtual image rotated by a predetermined angle around the center of the shape model of the crown surface 7a does not overlap with the corrected non-masked area. Chucking errors when holding the piston 7 with the hand 8 cause variations in the angle of the crown surface 7a in the captured image. Therefore, by setting the mask area taking into account this variation in angle, it is possible to suppress overlap between the inspection area and the mask area.

[0020] In the first embodiment, virtual images are generated for all captured images with 25 viewing angles, and a mask image is generated for each virtual image. This makes it possible to generate mask images in which unrecognizable areas are removed (masked areas) for all captured images with different viewing angles. The 25 viewing angles are obtained by fixing the camera 3 and rotating the piston 7 so that the second axis L2, which is tilted relative to the predetermined first axis L1, rotates around the first axis L1. This allows the crown surface 7a to be imaged at 25 different viewing angles without moving the camera 3.

[0021] In the appearance inspection method of the first embodiment, the surface of the crown surface 7a is inspected by masking an image captured by the camera 3 of the crown surface 7a with a mask image generated using the mask image generation method of the first embodiment. This allows the surface of the crown surface 7a to be inspected by masking with a highly accurate mask image, thereby improving the accuracy of defect detection. In the visual inspection method of the first embodiment, a captured image masked by a mask image and a two-dimensional image of the defect shape created based on a previously created three-dimensional defect model are combined with a surface image of the crown surface 7a to generate multiple defective product sample images. Then, defects on the crown surface 7a are detected based on the learning results obtained by machine learning using the multiple defective product sample images. This allows for the generation of defective product sample images that seamlessly match actual defects, thereby improving the inspection accuracy of the crown surface 7a. Furthermore, since pseudo sample images are used, there is no need to collect and store defective product samples. The time required to generate defective product sample images is shorter than the time required to collect defective product samples, thereby reducing actual work time. Furthermore, there is no need to repeat the learning process when changing product models, replacing imaging devices, or upgrading equipment. This significantly improves inspection efficiency. This effect is particularly noticeable in production lines with low defect rates or that produce a wide variety of products, where collecting defective product samples takes time.

[0022] Other Embodiments The above describes an embodiment for carrying out the present invention, but the specific configuration of the present invention is not limited to the configuration of the embodiment, and design changes and the like that do not deviate from the gist of the invention are also included in the present invention. In the embodiment, an example of detecting defects on the piston crown surface has been shown as an inspection method, but the present invention is not limited to defects and can also be applied to reading two-dimensional codes, characters, markings, etc. on the surface of the object to be imaged, and the same effects as those of the embodiment can be obtained. [Explanation of symbols]

[0023] 1 Appearance inspection device (inspection device), 3 Camera (imaging unit), 7 Piston (imaged object)

Claims

1. 1. A mask image generating method for capturing surface images of an object from a plurality of angles by changing a relative posture between an object to be captured and an imaging unit that captures an image of the object, and generating a mask image for masking a partial area from the surface images from the plurality of angles, specifying a first area of ​​a surface and a plurality of specific orientations in three-dimensional data representing the shape of the object to be imaged; a step of acquiring a plurality of virtual images when the posture of the three-dimensional data is changed to each of the plurality of specific postures and the object to be imaged is imaged by a virtual imaging unit; generating the mask image from the virtual image and the first region; A mask image generation method comprising:

2. 2. The mask image generation method according to claim 1, the step of generating the mask image generates the mask image by masking a second region other than the first region from the plurality of virtual images; Mask image generation method.

3. 3. The mask image generating method according to claim 2, and generating the mask image by reducing the second region in the plurality of virtual images in a direction away from the adjacent first region, and then masking the second region from the plurality of virtual images. Mask image generation method.

4. 4. The mask image generating method according to claim 3, When the second region is reduced in the plurality of virtual images in a direction away from the adjacent first region, the reduction ratio of the second region is the same vertically and horizontally. Mask image generation method.

5. 4. The mask image generating method according to claim 3, a portion of the second region outside the first region contracts outward, and a portion of the second region inside the first region contracts inward; Mask image generation method.

6. 4. The mask image generating method according to claim 3, The second region is formed so as to encompass a state obtained by rotating the first region by a predetermined angle. Mask image generation method.

7. 7. A mask image generating method according to claim 6, comprising: the second region is formed so as to include an envelope obtained by rotating the three-dimensional data. Mask image generation method.

8. 3. The mask image generating method according to claim 2, the step of generating the mask image generates the mask image by masking a second region other than the first region and a portion that is shaded by the first region from the plurality of virtual images; Mask image generation method.

9. a step of capturing surface images of the object from a plurality of angles by changing a relative attitude between the object and an imaging unit that images the object, and generating a mask image for masking a partial area from the surface images from the plurality of angles by generating the mask image using the mask image generating method according to claim 1; a step of masking an actual image of the object captured by the imaging unit with the mask image and inspecting a surface of the object; An inspection method comprising:

10. The inspection method according to claim 9, Detecting surface defects from the masked actual image using machine learning. Testing method.

11. 2. The mask image generation method according to claim 1, generating the virtual images in all of the plurality of specific postures; Mask image generation method.

12. 2. The mask image generation method according to claim 1, the plurality of specific postures are acquired by fixing the imaging unit and rotating the object to be imaged such that a second axis inclined with respect to a predetermined first axis rotates around the first axis; Mask image generation method.

13. 1. An inspection device that inspects a surface of an object to be imaged by changing a relative posture between an object to be imaged and an imaging unit that images the object to be imaged and capturing surface images of the object from a plurality of angles, The inspection device includes: three-dimensional data representing the shape of the object to be imaged; In the three-dimensional data, a first region of the surface and a plurality of specific orientations are specified; changing the posture of the three-dimensional data to the plurality of specific postures, and acquiring a plurality of virtual images when the object to be imaged is imaged by a virtual imaging unit; generating a mask image by masking a second region other than the first region from the plurality of virtual images; an actual image of the object captured by the imaging unit is masked with the mask image to inspect the surface of the object; In this regard, the mask image is generated by shrinking the second region in the plurality of virtual images in a direction away from the adjacent first region, and then masking the second region from the plurality of virtual images. Inspection equipment.

14. 1. An inspection device that inspects a surface of an object to be imaged by changing a relative posture between an object to be imaged and an imaging unit that images the object to be imaged and capturing surface images of the object from a plurality of angles, The inspection device includes: three-dimensional data representing the shape of the object to be imaged; In the three-dimensional data, a first region of the surface and a plurality of specific orientations are specified; changing the posture of the three-dimensional data to the plurality of specific postures, and acquiring a plurality of virtual images when the object to be imaged is imaged by a virtual imaging unit; generating a mask image by masking a second region other than the first region from the plurality of virtual images; an actual image of the object captured by the imaging unit is masked with the mask image to inspect the surface of the object; In this regard, the mask image is generated by masking the second region other than the first region and a portion that is shaded by the first region from the plurality of virtual images. Inspection equipment.

Images