Inspection system and inspection method

Abstract

To provide a technology for enabling highly accurate inference in a short time, even when an object includes areas that cause specular reflection on its surface.SOLUTION: The inspection system according to one embodiment is provided with: a mask image generating unit for generating a mask image to mask a group of pixels corresponding to a mask region on an image of an object; a learning unit for learning an inference model configured to detect whether the object contained in an input image is good or defective, using a learning image generated using the mask image, as input; an inference image generation unit for generating an inference image containing the object, using the mask image; and an inference unit for determining whether the object is good or defective from the inference image, using the learned inference model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE The present invention relates to an inspection system and an inspection method. [Background technology]

[0002] In the visual inspection of objects (e.g., products), inference models based on machine learning such as deep learning are increasingly being used. As an example, inference models are used to detect whether an object has a defect from an image of the object captured by a camera. Machine learning requires a 2D (two-dimensional) image of the object. Cited Document 1 discloses a technology for creating a 2D image of an object by rendering a 3D (three-dimensional) model that represents the shape and texture of the object. When rendering, a viewpoint and lighting are set for the object in the 3D model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 075252 Summary of the Invention [Problem to be solved by the invention]

[0004] If the surface of an object contains a part that causes specular reflection, the contrast of that part of the object changes significantly in the 2D image when the relative angle between the object and the lighting changes. This means that the model obtained by machine learning using these 2D images has a problem of low recognition (inference) accuracy. In particular, small scratches on the surface of resin are often visible or invisible depending on the relative angle between the object and the lighting, making them difficult to detect.

[0005] In the cited reference 1, a method is devised to prevent the influence of lighting from appearing in the training 2D images by rendering the model. However, when an inspection is performed, some kind of lighting is always set, and the 2D image affected by that lighting is the target of the inspection. Therefore, there is a risk that the 2D image to be inspected will be significantly different from the training data, leading to an incorrect inspection.

[0006] Additionally, the inspection must be performed within a very limited time frame, and capturing and rendering multiple images takes time, making it impossible to perform the inspection in a short time.

[0007] The problem to be solved by the present invention is to provide a technique capable of performing highly accurate inference in a short time even when an object includes a portion on its surface that causes specular reflection. [Means for solving the problem]

[0008] An inspection system according to an embodiment includes a mask image generating unit, a learning unit, an inference image generating unit, and an inference unit. The mask image generating unit generates a plurality of mask images for masking pixel groups corresponding to a mask region set for an object on an image of the object, the mask images corresponding to a plurality of orientations of the object when photographed for inspection of the object. The learning unit uses a plurality of learning images generated using the plurality of mask images corresponding to the plurality of orientations to learn an inference model configured to input an image and detect whether an object included in the input image is a good product or a defective product, using the plurality of learning images generated using the plurality of mask images corresponding to the plurality of orientations. The inference image generating unit generates a plurality of inference images including the object, corresponding to the plurality of orientations, using the plurality of mask images corresponding to the plurality of orientations. The inference unit uses the learned inference model to determine whether the object is a good product or a defective product from the plurality of inference images. Effect of the Invention

[0009] According to the present invention, a technique is provided that enables highly accurate inference in a short time even when an object includes a portion on its surface that causes specular reflection. [Brief description of the drawings]

[0010] [Figure 1] 5A to 5C are views for explaining a visual inspection according to the embodiment. [Diagram 2] A diagram to explain how changing the relative angle between the product and the lighting causes a change in contrast in the image. [Diagram 3] FIG. 1 is a functional block diagram showing an inspection device according to an embodiment. [Figure 4] FIG. 4 is a functional block diagram showing the learning image generation unit shown in FIG. 3 . [Diagram 5] 6A to 6C are diagrams showing an example of a method for a user to specify a mask region according to the embodiment. [Figure 6] 11A and 11B are views showing another example of a method for a user to specify a mask region according to the embodiment. [Figure 7] 1A and 1B are diagrams showing learning images or inference images according to an embodiment; [Figure 8] 1A to 1C are diagrams for explaining an inspection method according to an embodiment. [Figure 9] 1A to 1C are diagrams for explaining an inspection method according to an embodiment. [Figure 10] FIG. 4 is a block diagram showing an example of a hardware configuration of the inspection apparatus shown in FIG. 3. [Figure 11] 4 is a flowchart showing a learning process according to the embodiment. [Figure 12] 1 is a flowchart showing an inference process according to the embodiment. [Figure 13] 1A to 1C are diagrams for explaining an inspection method according to an embodiment. [Figure 14] 1A and 1B are diagrams for explaining the reason for using a mask image according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment will be described with reference to the drawings. Similar parts are designated by similar reference numerals throughout the drawings, and duplicated descriptions will be omitted where appropriate.

[0012] The embodiment relates to a technology applicable to applications such as appearance inspection performed in a factory. Appearance inspection is performed, for example, for the purpose of ensuring the quality of a product, and inspects the appearance of the product to determine whether the product is good or defective. For example, a product is judged to be good if it does not have defects such as scratches or stains, or if it has defects but the defects are within an acceptable range, and is judged to be defective if it has defects outside the acceptable range. In appearance inspection, as shown in FIG. 1, a product 10 to be inspected is placed on a stand 11, lighting is applied to the product 10 using a lighting device 12 as a light source, an image of the product 10 is taken by a camera 13, and an inspection based on the image is performed by a computer 14. The image is taken by the camera 13 when the relative position of the product 10 to the lighting device 12 and the camera 13 is in a predetermined positional relationship.

[0013] A defect in the product may be found by changing the relative attitude (relative angle) of the product 10 with respect to the lighting device 12 and the camera 13. For example, changing the attitude of the product 10 changes the reflection of the light by the defect, and thereby the defect of the product 10 is revealed in the image obtained by the camera 13. Therefore, the camera 13 takes an image while changing the relative attitude of the product 10 with respect to the lighting device 12 and the camera 13. For example, the lighting device 12 and the camera 13 are fixed to the base 11, and a drive unit (not shown) is provided that tilts the base 11 so as to rotate the product 10 around an axis perpendicular to the optical axis of the camera 13. Note that the relative attitude of the product 10 with respect to the lighting device 12 and the camera 13 may be changed by moving the lighting device 12 and the camera 13 with respect to the product 10 placed on the base 11. The description of rotating the product includes rotating the product itself and moving the lighting device 12 and the camera 13. The product 10 is photographed by the camera 13 while changing the position of the product 10 relative to the lighting device 12 and the camera 13, and the presence or absence of defects in the product 10 is determined from the multiple images obtained thereby.

[0014] In this embodiment, the object to be inspected is assumed to be a product 10 including a portion that causes specular reflection on the surface. The portion that causes specular reflection changes contrast greatly on the image depending on the relative angle between the lighting device 12 and the product 10. In the example shown in FIG. 1, the product 10 is disk-shaped, made entirely of resin, and a central portion 101 is made of shiny metal. In this example, the central portion 101 of the product 10 is the area that causes specular reflection. When lighting is applied to the product 10 from directly above, the reflection of the lighting does not enter the camera 13, and as shown in the upper part of FIG. 2, the central portion 101 of the product 10 becomes slightly dark on the image. When the product 10 is rotated, the reflection of the lighting enters the camera 13 at a certain rotation angle, and as shown in the lower part of FIG. 2, the central portion 101 of the product 10 becomes white on the image. Such a large change in contrast before and after rotation reduces the inspection accuracy. In this embodiment, a technology is provided that enables inspection to be performed with high accuracy even when the product 10 includes a portion whose contrast changes depending on the relative angle between the lighting device 12 and the product 10.

[0015] FIG. 3 is a schematic diagram of an inspection system 30 according to an embodiment. The inspection system 30 shown in FIG. 3 can be used for the visual inspection described above with reference to FIG. 1. For example, a machine device (not shown) places the product 10 to be inspected on a platform 11, and the inspection system 30 judges whether the product 10 is good or defective by visual inspection. The machine device sorts the product 10 according to the judgment result. For example, the machine device places the product 10 on a first conveyor when the inspection system 30 judges the product 10 to be good, and places the product 10 on a second conveyor when the inspection system 30 judges the product 10 to be defective. By repeating this operation, a plurality of products 10 are visually inspected in sequence and sorted into good and defective products.

[0016] The inspection system 30 generates an inference model by machine learning, and performs inference (specifically, visual inspection) using the inference model. The inference model is configured to receive an image including the product 10 to be inspected (an image showing the product 10) as an input, and detect whether the product 10 is a good product or a defective product. The inference model outputs, for example, inspection information indicating the likelihood that the product 10 is a good product. For example, the inspection information is defined so that the value ranges from 0 to 1, and the closer the value is to 0, the more likely the product 10 is a good product. As the inference model, for example, a deep neural network (DNN) such as R-CNN (Region Convolutional Neural Network) or SSD (Single Shot MultiBox Detector) can be used. Here, a case where an inference model is generated by good product learning will be mainly described, but the inference model may be generated by supervised learning.

[0017] As shown in Fig. 3, the inspection system 30 includes a mask image generating unit 31, a mask image storage unit 32, a learning image generating unit 33, a learning image storage unit 34, a learning unit 35, a model storage unit 36, an imaging unit 37, an inference image generating unit 38, and an inference unit 39. In the example shown in Fig. 3, it is assumed that the mask image generating unit 31, the mask image storage unit 32, the learning image generating unit 33, the learning image storage unit 34, the learning unit 35, the model storage unit 36, the inference image generating unit 38, and the inference unit 39 are implemented in a single device (e.g., a computer). Note that they may be implemented in multiple devices. For example, the mask image generation unit 31, mask image memory unit 32, learning image generation unit 33, learning image memory unit 34, learning unit 35, and model memory unit 36, which are involved in the learning phase of learning the inference model, may be implemented in a learning device, and the mask image memory unit 32, model memory unit 36, inference image generation unit 38, and inference unit 39, which are involved in the inference phase of performing inference using the inference model, may be implemented in an inference device (inspection device).

[0018] The mask image generation unit 31 generates a plurality of mask images used to generate learning images and inference images, and stores the mask images in the mask image storage unit 32. As described above, the central portion 101 of the product 10 to be inspected causes specular reflection. The mask image is used to mask the central portion 101 of the product on the image. The learning image refers to an image used to train an inference model, and the inference image refers to an image to be input to the inference model.

[0019] As described above with reference to FIG. 1, during inspection, the lighting device 12 and the camera 13 are fixed relative to the product 10, and the product 10 is rotated at multiple rotation angles relative to the lighting device 12 and the camera 13 to capture the product 10 with the camera 13. Thus, the position and shape of the central portion (part causing specular reflection) 101 of the product 10 on the image obtained at each rotation angle are determined, and it is possible to prepare multiple mask images corresponding to each of the multiple rotation angles in advance. For example, when capturing images while changing the rotation angle in 1° increments in the range from -10° to 10°, 21 mask images corresponding to 21 rotation angles are prepared. In this embodiment, learning images are generated using an environment equivalent to the inspection environment. Thus, mask images can be shared in generating learning images and inference images.

[0020] The mask image generating unit 31 will be described with reference to Fig. 4. As shown in Fig. 4, the mask image generating unit 31 includes a 3D (three-dimensional) model storage unit 311, a mask region setting unit 312, a mask processing unit 313, and a mask image generating unit 314.

[0021] The 3D model storage unit 311 stores a 3D model of the product 10 to be inspected. The 3D model is created in advance. The 3D model includes 3D shape information indicating the 3D shape of the product 10. The 3D model may be created by any method. For example, the 3D model may be created from a large number of 2D (two-dimensional) images obtained by photographing the actual product 10 from multiple viewpoints, or may be created from design data of the product 10.

[0022] Based on an input from a user, the mask area setting unit 312 sets, as a mask area, an area indicating a portion of the surface of the product 10 whose contrast changes depending on the relative angle between the product 10 and the lighting device, specifically, an area indicating a portion of the surface of the product 10 that causes specular reflection. For example, the mask area setting unit 312 displays a 2D image or 3D model of the product 10 on a display device (not shown) and accepts designation of a mask area from the user via an input device (not shown) such as a mouse.

[0023] In one example, as shown in FIG. 5, the mask area setting unit 312 displays a 2D image 51 of the product 10 and provides a 2D image editing tool to the user. The user specifies the mask area by filling in the area to be masked in the 2D image 51 with a virtual pen 52 or the like. The user can specify as many mask areas as necessary by changing the angle on the tool. The 2D image 51 of the product 10 may be an image obtained by photographing the actual product 10. Alternatively, the 2D image 51 of the product 10 may be an image obtained by projecting a 3D model stored in the 3D model storage unit 311 onto a 2D plane. In this case, the mask area setting unit 312 may read out the 3D model from the 3D model storage unit 311 to generate the 2D image 51.

[0024] 6, the mask area setting unit 312 displays a 3D model 61 of the product 10 and provides a 3D data editing tool to the user. The user directly selects a point group to be specified as the mask area with the tool. The 2D image editing tool and the 3D data editing tool may be existing tools or may be newly created.

[0025] 4 again, the mask processing unit 313 receives mask region information indicating a mask region designated by a user from the mask region setting unit 312, and reads out a 3D model from the 3D model storage unit 311. The mask processing unit 313 identifies a point group on the 3D model that corresponds to the mask region indicated by the mask region information.

[0026] The mask image generating unit 314 receives information identifying the 3D model and the point cloud corresponding to the mask region from the mask processing unit 313. The mask image generating unit 314 generates a 2D image of the product corresponding to the 3D model by rendering the 3D model according to the rendering parameters. Rendering the 3D model refers to projecting the 3D model onto a 2D plane. The rendering parameters include the position and orientation of a light source relative to the 3D model, the intensity of the illumination from the light source, and the position and orientation of a viewpoint relative to the 3D model. The rendering parameters are set in advance to reproduce an actual inspection environment such as that shown in FIG. 1. For example, the position and orientation of the viewpoint relative to the 3D model are determined according to the position and orientation of the camera 13 relative to the product 10 shown in FIG. 1. The mask image generating unit 314 rotates the 3D model at multiple rotation angles and renders the 3D model at each of the multiple rotation angles to generate multiple images corresponding to each of the multiple rotation angles. The mask image generating unit 314 generates a mask image by identifying a pixel group corresponding to a mask area on the image based on information identifying a point group corresponding to the mask area received from the mask processing unit 313 and removing pixels other than the pixel group corresponding to the mask area. The mask image generating unit 314 processes each mask image so that each pixel corresponding to the mask area has a pixel value determined according to the color of the stand 11 on which the product is placed during inspection and the color and brightness of the lighting during inspection.

[0027] 3 again, the mask image storage unit 32 stores a plurality of mask images corresponding to a plurality of rotation angles generated by the mask image generation unit 31. The mask images stored in the mask image storage unit 32 are used by the learning image generation unit 33 and the inference image generation unit 38.

[0028] The learning image generating unit 33 generates learning images used to learn the inference model. The learning image generating unit 33 acquires a plurality of sample images corresponding to a plurality of rotation angles. The sample image is also called a subject image. The sample image is an image including a product (sample) equivalent to the product 10 to be inspected. For example, the sample image is generated by providing a camera and a lighting device in the same arrangement as the inspection environment and photographing an actual product with the camera. In good product learning, the sample image is acquired by photographing a good product with the camera. In an environment equivalent to the inspection environment, the product is rotated with respect to the camera and the lighting device at a plurality of rotation angles, and the product is photographed at each of the plurality of rotation angles, thereby generating a plurality of sample images corresponding to each of the plurality of rotation angles.

[0029] The learning image generating unit 33 may generate the sample image using CG (Computer Graphics). When generating the sample image using CG, the learning image generating unit 33 may use a 3D model of the product used by the mask image generating unit 31. In this case, the 3D model includes texture information indicating the texture of the product surface in addition to the 3D shape information. The texture includes, for example, color and texture. The texture may include reflectance. The 3D model may be created by any method. For example, the 3D model is created by generating a 3D model including 3D shape information from a large number of 2D images obtained by photographing an actual product from multiple viewpoints, and performing texture mapping on the generated 3D model using some of the 2D images. The learning image generating unit 33 adds a shadow of the product to a 2D image obtained by rendering the 3D model, and superimposes the 2D image with the shadow of the product added on the background image. The background image is an image of the background, and in this embodiment, the background is a stand 11 on which a product 10 is placed as shown in FIG. 1. An image in which each pixel has a pixel value determined according to the color of the actual table 11 and the color and brightness of the actual lighting is prepared as a background image.

[0030] The learning image generation unit 33 generates a plurality of learning images by applying a plurality of mask images corresponding to a plurality of rotation angles to a plurality of sample images corresponding to a plurality of rotation angles. For example, the learning image generation unit 33 applies a mask image corresponding to a rotation angle θ to a sample image corresponding to a rotation angle θ. By applying a mask image to an image as shown in Fig. 2, an image in which a central portion 101 of the product 10 is masked as shown in Fig. 7 is generated as a learning image.

[0031] The learning image storage unit 34 stores a plurality of learning images generated by the learning image generation unit 33 .

[0032] The learning unit 35 performs machine learning of the inference model using the learning images stored in the learning image storage unit 34 as learning data, and determines the values ​​of parameters that configure the inference model. The learning unit 35 stores the trained inference model including the parameter values ​​obtained by machine learning in the model storage unit 36.

[0033] The imaging unit 37 includes an imaging device such as a camera, and captures an image of a product placed on a table as a subject to generate a captured image including the product. The imaging unit 37 corresponds to the camera 13 shown in Fig. 1. The imaging unit 37 rotates the product relative to the imaging unit 37 and a light source (e.g., the lighting device 12 shown in Fig. 1) at a plurality of rotation angles, and captures an image of the product at each of the plurality of rotation angles to generate a captured image. The imaging unit 37 obtains a plurality of captured images corresponding to each of the plurality of rotation angles.

[0034] The image-for-inference generating unit 38 generates an image for inference to be used for inference by the inference unit 39. The image-for-inference generating unit 38 generates a plurality of images for inference by applying a plurality of mask images corresponding to a plurality of rotation angles stored in the mask image storage unit 32 to a plurality of captured images corresponding to a plurality of rotation angles obtained by the imaging unit 37. For example, the image-for-inference generating unit 38 applies a mask image corresponding to the rotation angle θ to a captured image corresponding to the rotation angle θ, and obtains the captured image to which the mask image has been applied as an image for inference. By applying the mask image to an image as shown in FIG. 2, an image in which the central portion 101 of the product 10 is masked as shown in FIG. 7 is generated as an image for inference. This process is performed for each of a plurality of rotation angles to obtain a plurality of images for inference corresponding to each of a plurality of rotation angles.

[0035] The inference unit 39 receives a plurality of inference-use images from the inference-use image generating unit 38, and inspects the product based on the plurality of inference-use images. Specifically, the inference unit 39 reads out an inference model from the model storage unit 36, inputs the inference-use images to the inference model, and obtains inspection information output from the inference model. The inference unit 39 obtains inspection information for each of the plurality of inference-use images. The inference unit 39 judges whether the product is a good product or not based on the inspection information output from the inference model. For example, the inference unit 39 judges the product to be a good product when all the inspection information is below a predetermined threshold, and judges the product to be a defective product when any of the inspection information is equal to or above a predetermined threshold. The inference unit 39 corresponds to the computer 14 shown in FIG. 1.

[0036] An inspection method according to an embodiment will be described with reference to Figures 8 and 9. For the sake of explanation, two products 10 (products 10-1 and 10-2) are shown to be inspected simultaneously in Figures 8 and 9, but the products 10 can be inspected individually. Here, it is assumed that the product 10-1 on the left side has no defects and the product 10-2 on the right side has a defect.

[0037] Fig. 8 shows a case where mask processing is performed during inspection. The left side of Fig. 8 shows a state where the product 10 is not rotated, and no defects are found in either of the two products 10. The right side of Fig. 8 shows a state where the table 11 is tilted and the product 10 is rotated, and a defect 80 is found in the product 10-2. The inference unit 39 judges the product 10-1 to be a non-defective product, and judges the product 10-2 to be a defective product.

[0038] Fig. 9 shows a case where no masking is performed during inspection. The left side of Fig. 9 shows a state where the product 10 is not rotated, and no defects are found in either of the two products 10. The right side of Fig. 9 shows a state where the table 11 is tilted and the product 10 is rotated, and in this case, the central portion 101 of the product 10 has a large change in contrast on the image, so the inference unit 39 determines that both of the products 10 are defective.

[0039] When acquiring an image of the product 10 to be actually inspected, if the rotation angle of the product 10 is intentionally controlled, it is possible to know how much the product 10 has rotated. By applying a mask image corresponding to the rotation angle of the product 10 to the image acquired by the camera 13, an image with reduced specular reflection can be generated. By performing inspection using the image to which the mask image has been applied, it is possible to determine whether there are defects in areas other than those designated as the mask area.

[0040] With reference to FIG. 14, it will be described how the learning time can be shortened by the mask process.

[0041] The diagram on the left side of Figure 14 shows a case where there are two locations of specular reflection with two levels. In this case, depending on the relative angle between the lighting and the product, the number of possible light and dark cases is 2 x 2 = 4. Similarly, in the center diagram of Figure 14, the number of cases is 27, and in the right diagram of Figure 14, the number is 3125. Depending on the location of specular reflection and the number of levels, the number of possible light and dark cases increases, and a large number of training images are required for learning.

[0042] The time required for training increases as the number of training images increases. For example, in the case of the central figure, if the training time for a product without specular reflection was 1 hour, the training time would be 27 hours, and training would not be completed in one day.

[0043] When mask processing is performed on the product shown in the center of Figure 14, the learning time will be one hour, just like for a product without specular reflection.

[0044] Fig. 10 shows an outline of an example of the hardware configuration of the inspection system 30. As shown in Fig. 10, the inspection system 30 includes an imaging device 1010 and a computer 1020 as hardware components.

[0045] The imaging device 1010 is, for example, a camera. The imaging device 1010 functions as the imaging unit 37 shown in FIG.

[0046] The computer 1020 includes a processor 1021, a random access memory (RAM) 1022, a program memory 1023, a storage device 1024, an input / output interface 1025, and a bus 1026. The processor 1021 exchanges signals with the RAM 1022, the program memory 1023, the storage device 1024, and the input / output interface 1025 via the bus 1026.

[0047] The processor 1021 typically includes a general-purpose circuit such as a central processing unit (CPU) and a graphics processing unit (GPU). The RAM 1022 is used by the processor 1021 as a working memory. The RAM 1022 includes a volatile memory such as a synchronous dynamic random access memory (SDRAM). The program memory 1023 stores programs executed by the processor 1021, such as a learning image generation program, a learning program, and an inspection program. For example, a read-only memory (ROM) is used as the program memory 1023. Alternatively, a partial area of ​​the storage device 1024 may be used as the program memory 1023. The processor 1021 expands the program stored in the program memory 1023 into the RAM 1022, and interprets and executes the program. When the learning image generation program is executed by the processor 1021, the learning image generation program causes the processor 1021 to execute the processes described with respect to the mask image generation unit 31 and the learning image generation unit 33. In other words, the processor 1021 functions as the mask image generating unit 31 and the learning image generating unit 33 in accordance with the learning image generating program. The learning program, when executed by the processor 1021, causes the processor 1021 to execute the processes described in relation to the learning unit 35. The inspection program, when executed by the processor 1021, causes the processor 1021 to execute the processes described in relation to the inference image generating unit 38 and the inference unit 39.

[0048] A program such as the learning image generation program may be provided to the computer 1020 in a state stored in a computer-readable storage medium. In this case, for example, the computer 1020 is provided with a drive that reads data from the storage medium, and acquires the program from the storage medium. Examples of the storage medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. The program may also be stored in a server on a network, and the computer 1020 may download the program from the server.

[0049] The storage device 1024 stores data. The storage device 1024 includes a non-volatile memory such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage device 1024 functions as the mask image storage unit 32, the learning image storage unit 34, and the model storage unit 36.

[0050] The input / output interface 1025 is an interface for communicating with an external device. The input / output interface 1025 is connected to the imaging device 1010 via a cable such as a LAN (Local Area Network) cable, and receives an image from the imaging device 1010. Furthermore, the input / output interface 1025 is connected to an output device such as an input device or a display device. Note that the input / output interface 1025 may include a wireless module, and the computer 1020 may be wirelessly connected to the imaging device 1010.

[0051] In addition, the processor 1021 may include a dedicated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) instead of or in addition to a general-purpose circuit, and at least a part of the processing described in relation to the mask image generation unit 31, the learning image generation unit 33, the learning unit 35, the inference image generation unit 38, and the inference unit 39 may be realized by the dedicated circuit.

[0052] Next, the operation of the inspection system 30 will be described.

[0053] 11 is a schematic diagram showing an example of a procedure of the learning process according to the embodiment. The learning process shown in FIG. 11 is executed by the mask image generating unit 31, the learning image generating unit 33, and the learning unit 35 of the inspection system 30.

[0054] 11, the mask image generating unit 31 sets an area on the surface of the product to be inspected, which is specified by the user, as a mask area. The user specifies a portion of the product surface that causes specular reflection as the mask area. For example, the mask image generating unit 31 displays a 2D image of the product on a display device, and the user specifies the mask area on the 2D image.

[0055] In step S1102, the mask image generating unit 31 generates a plurality of mask images corresponding to a plurality of rotation angles based on the mask region set in step S1101, and stores the mask images in the mask image storage unit 32. For example, the mask image generating unit 31 rotates a 3D model of a product at a plurality of rotation angles, and obtains a 2D image of the product corresponding to each of the plurality of rotation angles by rendering the 3D model at each rotation angle. The mask image generating unit 31 identifies a pixel group corresponding to the mask region set in step S1101 on each 2D image, sets the pixel group corresponding to the mask region to a predetermined pixel value, and removes pixels other than the pixel group corresponding to the mask region, thereby obtaining a mask image.

[0056] In step S1103, the learning image generating unit 33 sets a rotation angle. For example, the learning image generating unit 33 selects one of the multiple rotation angles at which a mask image is generated. In step S1104, the learning image generating unit 33 rotates a non-defective product according to the rotation angle set in step S1103, and captures an image of the product using the imaging unit 37 to obtain a sample image. For example, the learning image generating unit 33 controls a driving unit (not shown) that changes the inclination of the table 11 in order to rotate the product 10 at the set rotation angle. The learning image generating unit 33 may be configured to automatically correct positional deviations caused by the rotation of the product on the image.

[0057] In step S1105, the learning image generation unit 33 reads out the mask image corresponding to the rotation angle set in step S1103 from the mask image storage unit 32, and superimposes the read mask image on the sample image obtained in step S1104. The learning image generation unit 33 stores the sample image on which the mask image is superimposed in the learning image storage unit 34 as a learning image.

[0058] If generation of learning images has not been completed for all rotation angles (step S1106; No), the flow returns to step S1103, where another rotation angle is set and the subsequent processes are repeated.

[0059] When the generation of learning images has been completed for all rotation angles (step S1106; Yes), the flow proceeds to step S1107. In step S1107, the learning unit 35 uses the learning images stored in the learning image storage unit 34 as learning data to learn an inference model configured to input an image including a product to be inspected and determine whether the product is good or defective. The learning unit 35 stores the learned inference model in the model storage unit 36.

[0060] In the example shown in Fig. 11, all mask images are generated first, then a sample image is generated at each rotation angle, and the mask images are applied to the sample images. In another example, the mask images and sample images may be generated in parallel at each rotation angle, and the mask images may be applied to the sample images.

[0061] Fig. 12 is a schematic diagram of an inference method according to an embodiment. The process shown in Fig. 12 is executed by the inference image generating unit 38 and the inference unit 39 of the inspection system 30. The process shown in Fig. 12 uses the inference model generated in the process described with reference to Fig. 11.

[0062] In step S1201 of Fig. 12, the inference image generating unit 38 sets a rotation angle for rotating the product 10 to be inspected. For example, the inference image generating unit 38 selects one of the multiple rotation angles for which a mask image is generated. In step S1202, the inference image generating unit 38 rotates the product 10 according to the rotation angle set in step S1201, and captures an image of the product 10 using the imaging unit 37 to obtain a captured image. For example, the inference unit 39 controls a drive unit (not shown) that changes the inclination of the platform 11 in order to rotate the product 10 at the set rotation angle.

[0063] In step S1203, the image-for-inference generating unit 38 performs a mask process on the captured image obtained in step S1202. For example, the image-for-inference generating unit 38 selects a mask image corresponding to the rotation angle set in step S1201 from among the multiple mask images stored in the mask image storage unit 32, and obtains an image for inference by superimposing the selected mask image on the captured image obtained in step S1202. The image-for-inference generating unit 38 is configured to automatically correct positional deviations caused by rotation of the product 10 on the image.

[0064] In step S1204, the inference unit 39 uses the inference model stored in the model storage unit 36 ​​to determine whether or not a defect exists in the product 10 from the image for inference generated in step S1203. For example, the inference unit 39 inputs the image for inference to the inference model, obtains inspection information output from the inference model, and determines whether or not a defect exists in the product 10 based on the inspection information.

[0065] If a defect exists in the product 10 (step S1205; Yes), the inference unit 39 determines that the product 10 is defective, and ends the process for the product 10.

[0066] If there is no defect in the product 10 (step S1205; No), in step S1206, the inference unit 39 determines whether or not all rotation angles have been processed. If there are unprocessed rotation angles (step S1206; No), the flow returns to step S1201. In step S1201, the unprocessed rotation angle is set and the subsequent processes are repeated.

[0067] If the processing has been completed for all rotation angles (step S1206; Yes), the inference unit 39 determines that the product 10 is a non-defective product, and ends the processing for the product 10.

[0068] In the inference phase, the mask images used in the learning phase can be used, thereby reducing the inspection time for each product 10.

[0069] The setting of the rotation angle for rotating the product (sample) in the learning phase does not have to be the same as the setting of the rotation angle for rotating the product in the inference phase. For example, in the learning phase, the rotation angle is changed in 1° increments in the range from -10° to 10°, and in the inference phase, the rotation angle is changed in 2° increments in the range from -10° to 10°. In this case, 21 mask images corresponding to 21 rotation angles, namely -10°, -9°, -8°, . . . , 10°, are generated, and all 21 mask images are used in the learning phase, and 11 mask images corresponding to 11 rotation angles, namely -10°, -8°, -6°, . . . , 10°, are used in the inference phase.

[0070] A method of inspecting the central portion 101 that causes specular reflection in the product 10 will be described with reference to Fig. 13. Although Fig. 13 shows two products 10 (products 10-1 and 10-2) being inspected simultaneously for the sake of explanation, the products 10 can be inspected individually. Here, it is assumed that the product 10-1 on the left side has no defects and the product 10-2 on the right side has a defect.

[0071] The mask image generating unit 31 generates a mask image for masking an area other than the mask area designated by the user, that is, a mask image for masking an area other than the central part 101 of the product 10 on the image. The learning image generating unit 33 obtains a learning image by applying the mask image to a photographed image obtained in the same manner as described above. The learning image obtained in this manner has an area other than the central part 101 of the product 10 masked. The learning unit 35 uses the learning image to learn an inference model. At this time, an image in which no specular reflection occurs is selected for learning. In this manner, an inference model for inspecting the central part 101 of the product 10 is generated. During inspection, a photographed image is obtained under conditions (angles) in which the central part 101 of the product 10 does not cause specular reflection, and inspection is performed using the inference model to determine whether or not there is a defect. By performing the inspection in this manner, a defect 80 is found in the central part 101 of the product 10-2. When inspecting the central part 101 of the product 10, it is not necessary to mask the area other than the central part 101 on the photographed image. By carrying out the inspection of the central portion 101 of the product 10 and the inspection of the portions other than the central portion 101 in parallel, an increase in processing time can be prevented.

[0072] As described above, the inspection system 30 sets a portion of the surface of the product to be inspected that causes specular reflection as a mask region, and generates a plurality of mask images for masking a pixel group corresponding to the mask region on the image of the product, which correspond to a plurality of rotation angles at which the product is rotated when photographed for inspection. The inspection system 30 then rotates a sample equivalent to the product at a plurality of rotation angles to generate a plurality of sample images corresponding to the plurality of rotation angles, and applies a plurality of mask images corresponding to the plurality of rotation angles to the plurality of sample images corresponding to the plurality of rotation angles, thereby generating a plurality of learning images corresponding to the plurality of rotation angles. The inspection system 30 uses the plurality of learning images to learn an inference model that is configured to input an image and detect whether a product included in the input image is a good product or a defective product. The inspection system 30 then rotates the product at a plurality of rotation angles, photographs the product at each of the plurality of rotation angles, thereby generating a plurality of photographed images corresponding to the plurality of rotation angles, and applies a plurality of mask images corresponding to the plurality of rotation angles to the plurality of photographed images corresponding to the plurality of rotation angles, thereby generating a plurality of inference images corresponding to the plurality of rotation angles. The inspection system 30 uses the learned inference model to determine whether the product is a good product or a defective product from the plurality of inference images.

[0073] According to the above configuration, even if the surface of the product includes a portion that causes specular reflection, the contrast does not change significantly between the learning images. As a result, it is possible to generate an inference model with high recognition accuracy, and the accuracy of the inspection using the inference model is improved. Furthermore, the contrast does not change significantly between the captured images used for product inspection, the occurrence of erroneous judgment is suppressed, and the inspection accuracy is improved. Furthermore, since the mask image used for creating the inference model is used in the inspection to perform mask processing on the captured images, the time required for mask processing during the inspection can be shortened. That is, according to the inspection system 30 having the above configuration, even if the product includes a portion that causes specular reflection on the surface, it is possible to perform a high-precision inspection in a short time.

[0074] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0075] 10...product, 11...unit, 12...lighting device, 13...camera, 14...computer, 30...inspection system, 31...mask image generation unit, 32...mask image memory unit, 33...learning image generation unit, 34...learning image memory unit, 35...learning unit, 36...model memory unit, 37...imaging unit, 38...inference image generation unit, 39...inference unit, 311...3D model memory unit, 312...mask area setting unit, 313...mask processing unit, 314...mask image generation unit, 1010...imaging device, 1020...computer, 1021...processor, 1022...RAM, 1023...program memory, 1024...storage device, 1025...input / output interface, 1026...bus.

Claims

1. a mask image generating unit that generates a plurality of mask images for masking pixel groups corresponding to a mask region set for the object on an image of the object, the mask images corresponding to a plurality of orientations of the object at the time of photographing the object for inspection; a learning unit that uses a plurality of learning images generated using the plurality of mask images corresponding to the plurality of postures to learn an inference model configured to input an image and detect whether an object included in the input image is a non-defective product or a defective product; an inference image generating unit that generates a plurality of inference images including the object corresponding to the plurality of poses, using the plurality of mask images corresponding to the plurality of poses; an inference unit that uses the trained inference model to determine whether the object is a non-defective or defective product from the plurality of inference images; An inspection system comprising:

2. 2. The inspection system of claim 1, further comprising a training image generation unit that generates a plurality of sample images corresponding to the plurality of poses by rotating a sample equivalent to the object, and generates the plurality of training images corresponding to the plurality of poses by applying the plurality of mask images corresponding to the plurality of poses to the plurality of sample images corresponding to the plurality of poses.

3. 2. The inspection system of claim 1, wherein the inference image generation unit generates a plurality of captured images corresponding to the plurality of poses by rotating the object and photographing the object in each of the plurality of poses, and generates the plurality of inference images corresponding to the plurality of poses by applying the plurality of mask images corresponding to the plurality of poses to the plurality of captured images corresponding to the plurality of poses.

4. The inspection system according to claim 1 , further comprising a mask area setting unit that sets a portion of the surface of the object that causes specular reflection as a mask area.

5. The inspection system of claim 1 , wherein pixels corresponding to the mask regions in the plurality of mask images have pixel values ​​based on a color of a platform on which the object rests during inspection.

6. 2. The inspection system of claim 1, wherein the mask image generator generates the plurality of mask images by identifying a point cloud on a 3D model of the object that corresponds to the mask region, rotating the 3D model while a viewpoint for rendering the 3D model of the object is fixed with respect to the 3D model, and rendering the 3D model in the plurality of poses.

7. 1. A computer implemented inspection method, comprising: generating a plurality of mask images for masking pixel groups corresponding to mask regions set for the object on the image of the object, the mask images corresponding to a plurality of orientations of the object at the time of photographing the object for inspection; training an inference model using a plurality of training images generated using the plurality of mask images corresponding to the plurality of poses, the inference model being configured to input an image and detect whether an object included in the input image is a non-defective product or a defective product; generating a plurality of inference images including the object corresponding to the plurality of poses using the plurality of mask images corresponding to the plurality of poses; determining whether the object is a good or bad product from the plurality of inference images using the trained inference model; An inspection method comprising:

Images