Appearance inspection system and appearance inspection method

Abstract

To provide an appearance inspection device capable of detecting minute defects that cannot be detected by conventional photometric stereo methods.SOLUTION: An appearance inspection system includes an inference unit. The inference unit performs inference on an inspection target by using an image obtained by imaging the inspection target with a lighting device that has deviation in one direction relative to an optical axis of an imaging unit with an inference model learned in advance. The inference model is learned by using images captured under the same lighting conditions as those used to capture the images used for the inference.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to an appearance inspection system and an appearance inspection method. [Background technology]

[0002] Photometric stereo is used when performing visual inspections to detect defects with subtle unevenness (see, for example, Patent Document 1). In photometric stereo, three or more images captured using illumination light from three or more directions are synthesized, and a normal image representing the slope of the normal to the surface is generated by determining the normal to the surface from the synthesized image. The normal image is then used for AI (Artificial Intelligence) learning and inference.

[0003] In conventional photometric stereo, an image for detecting defects is created by combining three or more images captured by illuminating the surface with light from three or more directions, making it difficult to detect defects with slight features in a single direction. The technology disclosed in Patent Document 1 uses normal images created by photometric stereo for learning and inference, but it is difficult to detect defects when the normal to the surface of the defect has a slight tilt. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6650986 Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide a technique capable of detecting defects having subtle features that cannot be detected by conventional photometric stereo methods. [Means for solving the problem]

[0006] According to one embodiment, the visual inspection system includes an inference unit. The inference unit performs inference on an inspection object using an image obtained by capturing an image of the inspection object using an illumination device that is biased in one direction relative to the optical axis of the imaging unit, using a pre-trained inference model. The inference model is trained using an image captured under the same illumination conditions as those used to capture the image used in the inference. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique that can detect defects with subtle features that cannot be detected by conventional photometric stereo methods. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a visual inspection system according to an embodiment. [Figure 2] FIG. 2 is a perspective view showing an imaging unit and an illumination unit shown in FIG. [Figure 3] 1A to 1C are diagrams for explaining a visual inspection method according to an embodiment. [Figure 4] 1A to 1C are diagrams for explaining a visual inspection method according to an embodiment. [Figure 5] FIG. 2 is a block diagram showing in detail a learning unit and a model storage unit included in the learning device shown in FIG. [Figure 6] 2 is a block diagram showing in detail an inference unit and a model storage unit included in the image processing unit shown in FIG. 1. [Figure 7] FIG. 2 is a block diagram showing the hardware configuration of a computer that can realize the information processing unit shown in FIG. [Figure 8] 1 is a flowchart showing a visual inspection method according to an embodiment. [Figure 9] FIG. 10 is a perspective view showing an imaging unit and an illumination unit according to another embodiment. [Figure 10] FIG. 10 is a perspective view showing an imaging unit and an illumination unit according to a further embodiment. [Figure 11] FIG. 10 is a block diagram illustrating a learning unit according to a further embodiment. [Figure 12]FIG. 10 is a block diagram illustrating an inference unit according to a further embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] The embodiments relate to a technique applicable to visual inspections performed in factories, etc. Visual inspections are performed, for example, to ensure the quality of a product (also called a workpiece), and involve inspecting the appearance of the product to determine whether the product is good or defective. For example, a product is determined to be good if it has no defects such as scratches or stains, or if any defects exist but the defects are within an acceptable range, and is determined to be defective if it has defects outside the acceptable range.

[0011] FIG. 1 is a functional block diagram that schematically illustrates an appearance inspection system 100 according to one embodiment, and FIG. 2 is a perspective view that schematically illustrates a portion of the appearance inspection system 100. As shown in FIG. 1, the appearance inspection system 100 includes an appearance inspection device 10 and a learning device 20. The appearance inspection device 10 performs an appearance inspection of an inspection target object and includes an imaging unit 11, an illumination unit 12, and an information processing unit 13. The learning device 20 is used to learn an inference model used in the appearance inspection by the appearance inspection device 10 and includes an image processing unit 21 and a communication unit 22. The learning device 20 is a computer such as a server, and communicates with the appearance inspection device 10 via a network such as the Internet. Learning of the inference model may be implemented in the form of a so-called cloud service.

[0012] As shown in FIG. 2, the imaging unit 11 captures an image of a product 19, which is an object to be inspected, from a certain direction to generate an image. In the example shown in FIG. 2, the imaging unit 11 is oriented vertically downward, and an image is captured by the imaging unit 11 with the product 19 positioned directly below the imaging unit 11. In this case, the top surface of the product 19 is imaged by the imaging unit 11. In this example, the top surface of the product 19 is shown as a flat surface, but this is not limiting. The imaging unit 11 can be, for example, a digital camera having a CCD (charge coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor. The image obtained by the imaging unit 11 is output to the information processing unit 13.

[0013] The lighting unit 12 is configured to illuminate the product 19 from four different lighting directions. Specifically, the lighting unit 12 includes four light sources (also referred to as lighting devices) 121, 122, 123, and 124 that irradiate the top surface of the product 19 with illumination light from the four different lighting directions. The light sources 121 and 123 face each other in the left-right direction (horizontal direction), and the light sources 122 and 124 face each other in the front-to-back direction (vertical direction). The front-to-back direction is perpendicular to the left-to-right direction. Therefore, an imaginary line segment connecting the light source 121 and the light source 123 is perpendicular to an imaginary line segment connecting the light source 122 and the light source 124. The light sources 121 to 124 are arranged at the same height so as to surround the imaging unit 11. The light sources 121 to 124 are arranged at equal intervals on the circumference of an imaginary circle that is centered on a point on the optical axis of the imaging unit 11 and is perpendicular to the optical axis of the imaging unit 11. Each of the light sources 121 to 124 is oriented obliquely downward so that its optical axis intersects with the optical axis of the imaging unit 11. In this arrangement, the pair of light sources 121 and 123 facing each other in the left-right direction has a left-right bias with respect to the optical axis of the imaging unit 11, and the pair of light sources 122 and 124 facing each other in the front-rear direction has a front-rear bias with respect to the optical axis of the imaging unit 11. As the light sources 121 to 124, for example, light emitting diodes (LEDs) or incandescent bulbs can be used.

[0014] 2, the illumination unit 12 further includes a ring-shaped support member 129, and light sources 121 to 124 are provided on the support member 129. The imaging unit 11 and the illumination unit 12 are arranged so that the central axis of the support member 129 coincides with the optical axis of the imaging unit 11. As an example, the light sources 121 to 124 can be realized by providing a large number of LEDs along the circumferential direction of the support member 129, defining four equal regions on the support member 129, and controlling the operation of the LEDs for each region.

[0015] 1 again, the information processing unit 13 includes a control unit 14 that controls the imaging unit 11 and the lighting unit 12, an image processing unit 15 that performs an appearance inspection of the product 19 based on the image obtained by the imaging unit 11, and a communication unit 16 that communicates with the learning device 20. The information processing unit 13 can be implemented by a computer such as a personal computer (PC) or a server, for example.

[0016] The control unit 14 controls the imaging unit 11 and the lighting unit 12 so that the light sources 121 to 124 are turned on one by one in a predetermined order and the imaging unit 11 captures an image of the product 19. For example, the control unit 14 turns on the light source 121 to capture an image of the product 19 with the imaging unit 11, turns off the light source 121 and turns on the light source 123 to capture an image of the product 19 with the imaging unit 11, turns off the light source 123 and turns on the light source 122 to capture an image of the product 19 with the imaging unit 11, and turns off the light source 122 and turns on the light source 124 to capture an image of the product 19 with the imaging unit 11, thereby obtaining an image set including four images of the product 19 captured with illumination light shining on it from each of the four lighting directions. Hereinafter, the image obtained by the imaging unit 11 when only light source 121 of light sources 121 to 124 is turned on will be called Image 1, the image obtained by the imaging unit 11 when only light source 122 of light sources 121 to 124 is turned on will be called Image 2, the image obtained by the imaging unit 11 when only light source 123 of light sources 121 to 124 is turned on will be called Image 3, and the image obtained by the imaging unit 11 when only light source 124 of light sources 121 to 124 is turned on will be called Image 4.

[0017] The appearance inspection is performed based on a composite image obtained by combining two images obtained by using two opposing light sources as a pair. In this embodiment, two pairs of light sources are provided, so two composite images are generated. Specifically, a composite image is generated by combining image 1 and image 3 obtained by using light source 121 and light source 123 opposing each other in the left-right direction, and a composite image is generated by combining image 2 and image 4 obtained by using light source 122 and light source 124 opposing each other in the front-back direction. Hereinafter, the composite image generated by combining image 1 and image 3 will be referred to as composite image 1, and the composite image generated by combining image 2 and image 4 will be referred to as composite image 2.

[0018] As shown in the following equation (1), composite image 1 is generated by the linear sum of the absolute value of the difference between image 1 and image 3 and the sum of image 1 and image 3. As shown in the following equation (2), composite image 2 is generated by the linear sum of the absolute value of the difference between image 2 and image 4 and the sum of image 2 and image 4. Composite image 1 = k1 × | image 1 - image 3 | + k2 × ( image 1 + image 3 ) (1) Composite image 2 = k1 × |image 2 - image 4| + k2 × (image 2 + image 4) (2) Here, k1 and k2 are coefficients that are determined in advance.

[0019] The absolute value of the difference between the two images (|Image 1 - Image 3|, |Image 2 - Image 4|) has the effect of showing the unevenness of the object being inspected as light and dark in the image, and the sum of the two images (Image 1 + Image 3, Image 2 + Image 4) has the effect of showing the pattern of the object being inspected.

[0020] To express the subtle unevenness of the object under inspection, increase the coefficient k1. If the object under inspection is dark overall, increase k2, and conversely, if it is bright, decrease k2 to improve visibility.

[0021] In this way, the coefficients k1 and k2 are set in accordance with the characteristics of the upper surface to be inspected of the product 19, which is the object to be inspected, thereby improving the inspection accuracy.

[0022] Figure 3 shows a schematic diagram of a visual inspection of a product 19 that has a defect (in this example, a dent) that is elongated in the left-right direction. As shown in Figure 3, the defect does not appear clearly in composite image 1. On the other hand, the defect appears clearly in composite image 2.

[0023] 4 is a schematic diagram showing a visual inspection of a product 19 having a defect (in this example, a dent) that is elongated in the front-to-rear direction. As shown in FIG. 4, the defect does not appear clearly in composite image 2. On the other hand, the defect appears clearly in composite image 1.

[0024] In this way, by using two composite images obtained using two pairs of light sources, each pair containing two light sources facing each other, it is possible to detect defects that are elongated in the left-right direction as well as defects that are elongated in the front-to-back direction.

[0025] In this embodiment, AI (Artificial Intelligence) is used for image processing in the image processing unit 15. Specifically, the image processing unit 15 performs inference using two inference models prepared corresponding to two pairs of light sources.

[0026] The image processing unit 15 includes a model storage unit 152 and an inference unit 153. The model storage unit 152 stores a trained inference model. The inference unit 153 performs a visual inspection of each of a plurality of products using the inference model stored in the model storage unit 152. The inference model is trained in advance by the learning device 20, and the control unit 14 acquires the trained inference model from the learning device 20 using the communication unit 16 and stores the acquired inference model in the model storage unit 152. The trained inference model can be commonly used for visual inspections of a plurality of products. The inference model is retrained when it needs to be updated.

[0027] First, the learning phase in which the inference model is learned will be described with reference to FIGS.

[0028] 5 schematically shows the functional configuration of the learning unit 211 and model storage unit 212 included in the image processing unit 21 of the learning device 20. As shown in Fig. 5, the learning unit 211 includes a first learning image generation unit 2101, a second learning image generation unit 2102, a first learning image storage unit 2103, a second learning image storage unit 2104, a first learning unit 2105, and a second learning unit 2106. The model storage unit 212 includes a first model storage unit 2107 and a second model storage unit 2108.

[0029] In this embodiment, the inference model is configured to receive an image containing a product as input and detect whether the product is of good quality. The inference model outputs, for example, a score indicating the likelihood that the product is of good quality. For example, the score is defined so that the value ranges from 0 to 1, and the closer the value is to 1, the more likely the product is of good quality. As the inference model, for example, a deep neural network (DNN) such as R-CNN (Region Convolutional Neural Network) or SSD (Single Shot MultiBox Detector) can be used. The inference model is trained, for example, by supervised learning.

[0030] In the training phase, an inference model is trained using multiple image sets corresponding to multiple samples. A sample may refer to a product used to create training images. A plurality of non-defective samples (i.e., samples without defects) and defective samples (i.e., samples with defects) are prepared in advance, and an image set is generated for each of these samples. Alternatively, an image set corresponding to a non-defective sample may be processed using computer graphics (CG) to generate an image set corresponding to a defective sample. In the example shown in FIG. 1 , multiple image sets corresponding to multiple samples are generated using the imaging unit 11 and the illumination unit 12 of the visual inspection device 10, and the communication unit 22 of the training device 20 acquires the multiple image sets from the visual inspection device 10. Images 1 and 3 included in each image set are input to a first training image generation unit 2101, and images 2 and 4 included in each image set are input to a second training image generation unit 2102.

[0031] The first training image generation unit 2101 acquires image 1 and image 3 included in the image set, and generates composite image 1 by combining image 1 and image 3 according to the above formula (1). The first training image generation unit 2101 stores composite image 1 as a training image in the first training image storage unit 2103. The training image is assigned either a label indicating a good product or a label indicating a defective product. The labeling is performed, for example, manually.

[0032] The first training image generation unit 2101 generates a training image (synthetic image 1) for each of a plurality of image sets corresponding to a plurality of samples. Therefore, the first training image storage unit 2103 stores a plurality of training images corresponding to the plurality of samples, respectively.

[0033] The first learning unit 2105 learns an inference model using a plurality of learning images stored in the first learning image storage unit 2103. Since learning of the inference model can be performed using well-known techniques, detailed explanations will be omitted. The first learning unit 2105 stores the learned inference model in the first model storage unit 2107.

[0034] The second training image generation unit 2102, the second training image storage unit 2104, and the second training unit 2106 operate in the same manner as the first training image generation unit 2101, the first training image storage unit 2103, and the first training unit 2105, respectively, and therefore detailed explanations thereof will be omitted.

[0035] The second training image generation unit 2102 generates a composite image 2 by combining images 2 and 4 included in the image set according to the above formula (2), and stores the composite image 2 as a training image in the second training image storage unit 2104. The second training unit 2106 trains an inference model using the multiple training images stored in the second training image storage unit 2104, and stores the trained inference model in the second model storage unit 2108.

[0036] In this way, two inference models corresponding to two pairs of light sources are learned. The inference model corresponding to the pair of light sources 121 and 123 facing each other in the left-right direction is also called the inference model for left-right light sources, and the inference model corresponding to the pair of light sources 122 and 124 facing each other in the front-back direction is also called the inference model for front-back light sources.

[0037] The communication unit 22 transmits the learned inference model for the light source in the left and right directions and the learned inference model for the light source in the front and rear directions to the appearance inspection device 10.

[0038] Next, the inference phase in which inference is performed using the trained inference model will be described with reference to FIGS.

[0039] 6 schematically shows the functional configuration of the inference unit 153 and model storage unit 152 included in the image processing unit 15. As shown in Fig. 6, the inference unit 153 includes a first inference-use image generation unit 1509, a second inference-use image generation unit 1510, a first inference unit 1511, a second inference unit 1512, and an inference result display unit 1513. The model storage unit 152 includes a first model storage unit 1507 and a second model storage unit 1508.

[0040] The first model storage unit 1507 stores the trained inference model for the light source in the left and right direction. Specifically, the communication unit 16 shown in FIG. 1 receives the trained inference model for the light source in the left and right direction from the learning device 20 and stores it in the first model storage unit 1507. The second model storage unit 1508 stores the trained inference model for the light source in the front and rear direction. Specifically, the communication unit 16 shown in FIG. 1 receives the trained inference model for the light source in the front and rear direction from the learning device 20 and stores it in the second model storage unit 1508.

[0041] In the inference phase, the control unit 14 captures images of the product 19 to be inspected using the imaging unit 11 and the lighting unit 12 to obtain an image set. Images 1 and 3 included in the image set are input to a first inference image generation unit 1509, and images 2 and 4 included in the image set are input to a second inference image generation unit 1510.

[0042] The first inference image generation unit 1509 generates a composite image 1 by combining images 1 and 3 included in the image set in accordance with the above formula (1). The first inference image generation unit 1509 sends the composite image 1 to the first inference unit 1511 as an inference image.

[0043] The first inference unit 1511 receives an inference image from the first inference image generation unit 1509 and acquires an inference model for a horizontal light source from the first model storage unit 1507. The first inference unit 1511 inputs the inference image to the inference model and obtains a score output from the inference model. The first inference unit 1511 determines whether a product is good or bad based on the score output from the inference model. For example, the first inference unit 1511 determines a product as good if the score value exceeds a predetermined threshold, and determines a product as defective if the score value is equal to or less than the predetermined threshold. The first inference unit 1511 notifies the inference result display unit 1513 of the inference result indicating whether the product is good or defective.

[0044] The second inference image generation unit 1510 and the second inference unit 1512 perform the same processing as the first inference image generation unit 1509 and the first inference unit 1511, respectively, and therefore detailed explanations thereof will be omitted.

[0045] The second inference image generation unit 1510 generates a composite image 2 by combining images 2 and 4 included in the image set according to the above formula (2), and sends the composite image 1 to the second inference unit 1512 as an image for inference. The second inference unit 1512 receives the image for inference from the second inference image generation unit 1510 and acquires an inference model for a front-to-rear light source from the second model storage unit 1508. The second inference unit 1512 inputs the image for inference into the inference model and obtains a score output from the inference model. The second inference unit 1512 determines whether the product is good or bad based on the score output from the inference model, and notifies the inference result display unit 1513 of the inference result indicating whether the product is good or bad.

[0046] In this way, for learning by the first learning unit 2105, an image obtained under the same shooting conditions as the image used in the inference by the first inference unit 1511 is used, and for learning by the second learning unit 2106, an image obtained under the same shooting conditions as the image used in the inference by the second inference unit 1512 is used. Specifically, for learning by the first learning unit 2105 and the inference by the first inference unit 1511, light sources 122 and 124 that are biased in the left-right direction with respect to the optical axis of the imaging unit 11 are used, and for learning by the second learning unit 2106 and the inference by the second inference unit 1512, light sources 122 and 124 that are biased in the front-to-back direction with respect to the optical axis of the imaging unit 11 are used.

[0047] The inference result display unit 1513 receives inference results from the first inference unit 1511 and the second inference unit 1512. When both the inference result from the first inference unit 1511 and the inference result from the second inference unit 1512 indicate a good product, the inference result display unit 1513 displays an inference result indicating that the product is a good product on a display device (not shown). When either the inference result from the first inference unit 1511 or the inference result from the second inference unit 1512 indicates a defective product, the inference result display unit 1513 displays an inference result indicating that the product is a defective product on a display device.

[0048] 7 schematically shows a computer 60 that can realize the information processing unit 13. As shown in Fig. 7, the computer 60 includes, as hardware components, a processor 61, a RAM (Random Access Memory) 62, a program memory 63, a storage device 64, an input / output interface 65, a communication interface 66, and a bus 67. The processor 61 exchanges signals with the RAM 62, the program memory 63, the storage device 64, the input / output interface 65, and the communication interface 66 via the bus 67.

[0049] The processor 61 typically includes general-purpose circuits such as a central processing unit (CPU) and a graphics processing unit (GPU). The RAM 62 is used by the processor 61 as a working memory. The RAM 62 includes a volatile memory such as a synchronous dynamic random access memory (SDRAM). The program memory 63 stores programs executed by the processor 61, such as an appearance inspection program. For example, a read-only memory (ROM) is used as the program memory 63. Alternatively, a partial area of the storage device 64 may be used as the program memory 63. The processor 61 loads the program stored in the program memory 63 into the RAM 62, and interprets and executes the program. When executed by the processor 61, the appearance inspection program causes the processor 61 to perform the processes described with respect to the control unit 14, the first inference-use image generation unit 1509, the second inference-use image generation unit 1510, the first inference unit 1511, the second inference unit 1512, and the inference result display unit 1513. In other words, the processor 61 functions as a control unit 14, a first inference image generation unit 1509, a second inference image generation unit 1510, a first inference unit 1511, a second inference unit 1512, and an inference result display unit 1513 in accordance with the appearance inspection program.

[0050] A program such as the visual inspection program may be provided to the computer 60 in a state where it is stored on a computer-readable storage medium. In this case, for example, the computer 60 is provided with a drive that reads data from the storage medium and acquires the program from the storage medium. Examples of storage media include magnetic disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. Alternatively, the program may be stored on a server on a network, and the computer 60 may download the program from the server.

[0051] The storage device 64 stores data. The storage device 64 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 64 functions as a first model storage unit 1507 and a second model storage unit 1508.

[0052] The input / output interface 65 is an interface for connecting peripheral devices. The input / output interface 65 is connected to the imaging unit 11 and the lighting unit 12 via cables. The processor 61 controls the imaging unit 11 and the lighting unit 12 via the input / output interface 65. The processor 61 acquires images from the imaging unit 11 via the input / output interface 65. Furthermore, the input / output interface 65 can be connected to an input device and an output device such as a display device. Note that the input / output interface 65 may include a wireless module, and the computer 60 may communicate with the imaging unit 11 and the lighting unit 12 wirelessly.

[0053] The communication interface 66 is an interface for communicating with an external device such as the learning device 20 shown in FIG. 1. The processor 61 uses the communication interface 66 to transmit a set of images acquired using the imaging unit 11 and the lighting unit 12 to the learning device 20. The processor 61 uses the communication interface 66 to receive a trained inference model from the learning device 20. The communication interface 66 functions as the communication unit 16.

[0054] In addition, the processor 61 may include a dedicated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) instead of or in addition to the general-purpose circuit, and at least a part of the series of processes may be realized by the dedicated circuit.

[0055] The learning device 20 can be implemented by a computer having a hardware configuration similar to that of the computer 60 shown in Fig. 7. For example, the processor functions as a first learning image generation unit 2101, a second learning image generation unit 2102, a first learning unit 2105, and a second learning unit 2106 by executing a learning program stored in a program memory.

[0056] 8 is a diagram illustrating an example of the procedure of the visual inspection process according to the embodiment. The visual inspection process illustrated in FIG. 8 is executed by the inference unit 153 illustrated in FIGS.

[0057] In step S71 of FIG. 8, the first inference image generation unit 1509 generates composite image 1 by combining image 1 and image 3 obtained using light sources 121 and 123 that face each other in the left-right direction. For example, the control unit 14 turns on light source 121 and causes the imaging unit 11 to capture an image of product 19, thereby obtaining image 1. Subsequently, the control unit 14 turns off light source 121 and turns on light source 123 and causes the imaging unit 11 to capture an image of product 19, thereby obtaining image 3. The first inference image generation unit 1509 generates composite image 1 by combining image 1 and image 3 according to the above formula (1).

[0058] In step S72, the first inference unit 1511 performs inference based on the composite image 1 obtained in step S71 and the inference model for left-right light sources stored in the first model storage unit 1507. For example, the first inference unit 1511 inputs the composite image 1 to the inference model and obtains a score output from the inference model.

[0059] In step S73, the first inference unit 1511 determines whether the score obtained in step S72 is higher than a predetermined threshold. If the score is equal to or lower than the threshold (step S73; No), the flow proceeds to step S74. In step S74, the inference result display unit 1513 displays that the product 19 is defective, and the visual inspection of the product 19 is completed.

[0060] If the score is higher than the threshold (Step S73; Yes), the flow proceeds to Step S75. In Step S75, the second inference image generation unit 1510 generates a composite image 2 by combining images 2 and 4 obtained using light sources 122 and 124 facing each other in the front-to-back direction. For example, the control unit 14 turns on the light source 122 and captures an image of the product 19 with the imaging unit 11, thereby obtaining image 2. Subsequently, the control unit 14 turns off the light source 122 and turns on the light source 124 and captures an image of the product 19 with the imaging unit 11, thereby obtaining image 4. The second inference image generation unit 1510 generates a composite image 2 by combining images 2 and 4 according to the above formula (2).

[0061] In step S76, the second inference unit 1512 performs inference based on the composite image 2 obtained in step S75 and the inference model for the front and rear light source stored in the second model storage unit 1508. For example, the second inference unit 1512 inputs the composite image 2 to the inference model and obtains a score output from the inference model.

[0062] In step S77, the second inference unit 1512 determines whether the score obtained in step S76 is higher than a predetermined threshold. If the score is equal to or lower than the threshold (step S77; No), the flow proceeds to step S78. In step S78, the inference result display unit 1513 displays that the product 19 is defective, and the visual inspection of the product 19 is completed.

[0063] If the score is higher than the threshold value (step S77; Yes), the flow proceeds to step S79. In step S79, the inference result display unit 1513 displays that the product 19 is a non-defective product, and the visual inspection of the product 19 is completed.

[0064] As described above, the appearance inspection device 10 includes the imaging unit 11 that images the product 19, the light sources 121 to 124 that irradiate the product 19 with illumination light from different illumination directions, the illumination unit 12 having the light source 121 and the light source 123 facing each other in the left-right direction and the light source 122 and the light source 124 facing each other in the front-back direction, a first inference image generation unit 1509 that generates a first inference image by combining image 1 obtained by the imaging unit 11 with the illumination light from the light source 121 irradiating the product 19 and image 3 obtained by the imaging unit 11 with the illumination light from the light source 123 irradiating the product 19, and a first inference image generation unit 1509 that inputs the first inference image to an inference model prepared corresponding to the pair of light sources 121 and 123, obtains a score output from the inference model, and determines whether the product 19 is a good or bad based on the score. The system includes a first inference unit 1511 that determines whether the product is good or defective, a second inference image generation unit 1510 that generates a second inference image by combining image 2 obtained by the imaging unit 11 with illumination light from light source 122 shining on the product 19 and image 4 obtained by the imaging unit 11 with illumination light from light source 124 shining on the product 19, a second inference unit 1512 that inputs the second inference image to an inference model prepared corresponding to the pair of light sources 122 and 124, obtains a score output from the inference model, and determines whether the product 19 is good or defective based on the score, and an inference result display unit 1513 that displays the results of the appearance inspection of the product 19 based on the judgment results by the first inference unit 1511 and the second inference unit 1512. For example, the first inference image generation unit 1509 generates the first inference image as the linear sum of the absolute value of the difference between image 1 and image 3 and the sum of image 1 and image 3, and the second inference image generation unit 1510 generates the second inference image as the linear sum of the absolute value of the difference between image 2 and image 4 and the sum of image 2 and image 4.

[0065] According to the above configuration, if the product 19 has a defect extending in the front-to-back direction, the defect can be detected based on the first inference image, and if the product has a defect extending in the left-to-right direction, the defect can be detected based on the second inference image. In other words, it is possible to detect defects with subtle features in a single direction. Therefore, it is possible to detect defects with subtle features that cannot be detected by conventional photometric stereo.

[0066] If the product 19 is determined to be defective in an inspection using one pair of light sources (e.g., light beams 121 and 123), the product 19 may be determined to be defective without performing an inspection using the other pair of light sources (e.g., light beams 122 and 124), thereby reducing processing time and power consumption.

[0067] In the above embodiment, an example in which two pairs of light sources are used has been described, but three or more pairs of light sources may be used.

[0068] FIG. 9 is a schematic diagram illustrating an imaging unit 11 and an illumination unit 12 included in a visual inspection apparatus 10 according to another embodiment. In the example illustrated in FIG. 9, the illumination unit 12 includes four pairs of light sources. Specifically, the illumination unit 12 includes light sources 121 to 128, where light source 121 and light source 123 face each other in the left-right direction, light source 122 and light source 124 face each other in the front-rear direction, light source 125 and light source 127 face each other in a first oblique direction, and light source 126 and light source 128 face each other in a second oblique direction. The first oblique direction is perpendicular to the second oblique direction, and the first oblique direction and the second oblique direction intersect with the left-right direction and the front-rear direction at 45 degrees. The light sources 121 to 128 are arranged at equal intervals on the circumference of an imaginary circle centered on a point on the optical axis of the imaging unit 11 and perpendicular to the optical axis of the imaging unit 11. Each pair of light sources is biased in one direction with respect to the optical axis of the imaging unit 11.

[0069] If the illumination unit 12 has four pairs of light sources, four inference models are prepared corresponding to the four pairs of light sources. In the learning phase, the inference model corresponding to each pair of light sources is learned using a composite image obtained by combining images obtained by the imaging unit 11 using that pair of light sources. In the inference phase, the composite image obtained by combining images obtained by the imaging unit 11 using that pair of light sources is input to the inference model corresponding to that pair of light sources, thereby performing an appearance inspection. That is, the appearance inspection system 100 has four systems of processing blocks, each including a learning image generation unit, a learning image storage unit, a learning unit, a model storage unit, an inference image generation unit, and an inference unit, corresponding to the four pairs of light sources.

[0070] In cases where it is known in advance that the shape of the defect to be detected is elongated in one direction, one of the above-described processing blocks is sufficient. For example, when detecting a defect that is elongated in the front-rear direction, visual inspection can be performed using only the inference model for the left-right light source described above. In this case, as shown in FIG. 10, the illumination unit 12 includes a pair of light sources 121 and 123 facing each other in the left-right direction. As shown in FIG. 11, the learning unit 211 includes a first learning image generation unit 2101, a first learning image storage unit 2103, and a first learning unit 2105. The inference unit 153 includes a first inference image generation unit 1509, a first inference unit 1511, and an inference result display unit 1513. In FIGS. 10, 11, and 12, components similar to those shown in FIGS. 1, 5, and 6 are denoted by the same reference numerals, and detailed descriptions of these components will be omitted. The inference result display unit 1513 receives the inference results from the first inference unit 1511. When the inference result from the first inference unit 1511 indicates a good product, the inference result display unit 1513 displays, on the display device, an inference result indicating that the product is a good product. When the inference result from the first inference unit 1511 indicates a defective product, the inference result display unit 1513 displays, on the display device, an inference result indicating that the product is a defective product.

[0071] In the above-described embodiment, the learning device 20 uses a set of images acquired using the imaging unit 11 and the illumination unit 12 in the visual inspection device 10 to train an inference model. The image set used to train the inference model only needs to be acquired under the same lighting conditions as those used in the visual inspection, and does not have to be acquired by the visual inspection device 10. For example, the learning device 20 may be provided with an imaging unit and an illumination unit arranged in the same manner as the imaging unit 11 and the illumination unit 12 shown in FIG. 2, and the imaging unit and the illumination unit may be used to capture images of a sample in the same manner as described above, and the inference model may be trained using the image set obtained thereby.

[0072] In the above-described embodiment, the learning of the inference model and the inference using the inference model are performed by separate devices. In other embodiments, the learning of the inference model may be performed by the visual inspection device 10.

[0073] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0074] 10...appearance inspection device, 11...imaging unit, 12...illumination unit, 13...information processing unit, 14...control unit, 15...image processing unit, 16...communication unit, 20...learning device, 21...image processing unit, 22...communication unit, 60...computer, 61...processor, 62...RAM, 63...program memory, 64...storage device, 65...input / output interface, 66...communication interface, 67...bus, 100...appearance inspection system, 121-128...light source, 129...support member, 152...model memory unit, 153...inference unit, 211...learning unit, 212...model Model memory unit, 1507...first model memory unit, 1508...second model memory unit, 1509...first inference image generation unit, 1510...second inference image generation unit, 1511...first inference unit, 1512...second inference unit, 1513...inference result display unit, 2101...first learning image generation unit, 2102...second learning image generation unit, 2103...first learning image memory unit, 2104...second learning image memory unit, 2105...first learning unit, 2106...second learning unit, 2107...first model memory unit, 2108...second model memory unit.

Claims

1. an inference unit that performs inference on an inspection object using an image obtained by imaging the inspection object using a pre-trained inference model using an illumination device that is biased in one direction relative to the optical axis of the imaging unit; An appearance inspection system in which the inference model is trained using images captured under the same lighting conditions as those under which the images used in the inference were captured.

2. The inference model includes a first inference model configured to receive an image as input and determine whether an inspection object included in the image is a non-defective product; The inference unit a first inference image generating unit that generates a first inference image by combining a first image obtained by the imaging unit in a state where illumination light from a first illumination device is irradiated on the inspection object and a third image obtained by the imaging unit in a state where illumination light from a third illumination device that faces the first illumination device in a first direction is irradiated on the inspection object; a first inference unit that performs an appearance inspection of the inspection object by inputting the first inference image into the first inference model; an output unit that outputs a result of the appearance inspection of the inspection object based on the result of the appearance inspection by the first inference unit; Equipped with The first inference model is trained using images captured under the same lighting conditions as those under which the first image and the third image were captured. The visual inspection system according to claim 1 .

3. a first learning image generating unit that generates a first learning image by combining an image obtained by the imaging unit in a state where illumination light from the first illumination device is applied to a sample corresponding to the inspection object and an image obtained by the imaging unit in a state where illumination light from the third illumination device is applied to the sample; a first learning unit that learns the first inference model using the first learning image; The visual inspection system according to claim 2 , further comprising:

4. The first inference image is a linear sum of an absolute value of a difference between the first image and the third image and a sum of the first image and the third image. The visual inspection system according to claim 2 .

5. the first inference image is a sum of a value obtained by multiplying an absolute value of a difference between the first image and the third image by a first coefficient and a value obtained by multiplying a sum of the first image and the third image by a second coefficient; the first coefficient and the second coefficient are set according to characteristics of a surface to be inspected of the inspection object. The visual inspection system according to claim 4 .

6. The inference model further includes a second inference model configured to receive an image as input and determine whether an inspection object included in the image is a good product; The inference unit a second inference image generating unit that generates a second inference image by combining a second image obtained by the imaging unit in a state where illumination light from a second illumination device is irradiated on the inspection object and a fourth image obtained by the imaging unit in a state where illumination light from a fourth illumination device that faces the second illumination device in a second direction is irradiated on the inspection object; a second inference unit that performs a visual inspection of the inspection object by inputting the second inference image into the second inference model; Furthermore, the second direction is perpendicular to the first direction, the output unit outputs a result of the appearance inspection of the inspection object based on a result of the appearance inspection by the first inference unit and a result of the appearance inspection by the second inference unit; The second inference model is trained using images captured under the same lighting conditions as those under which the second image and the fourth image were captured. The visual inspection system according to claim 2 .

7. a first learning image generating unit that generates a first learning image by combining an image obtained by the imaging unit in a state where illumination light from the first illumination device is applied to a sample corresponding to the inspection object and an image obtained by the imaging unit in a state where illumination light from the third illumination device is applied to the sample; a first learning unit that learns the first inference model using the first learning image; a second learning image generating unit that generates a second learning image by combining an image obtained by the imaging unit in a state where illumination light from the second illumination device is irradiated on the sample and an image obtained by the imaging unit in a state where illumination light from the fourth illumination device is irradiated on the sample; a second learning unit that learns the second inference model using the second learning image; The visual inspection system according to claim 6 , further comprising:

8. the first inference image is a linear sum of an absolute value of a difference between the first image and the third image and a sum of the first image and the third image; The second inference image is a linear sum of an absolute value of a difference between the second image and the fourth image and a sum of the second image and the fourth image. The visual inspection system according to claim 6 .

9. the first inference image is a sum of a value obtained by multiplying an absolute value of a difference between the first image and the third image by a first coefficient and a value obtained by multiplying a sum of the first image and the third image by a second coefficient; the second inference image is a sum of a value obtained by multiplying an absolute value of a difference between the second image and the fourth image by the first coefficient and a value obtained by multiplying a sum of the second image and the fourth image by the second coefficient; the first coefficient and the second coefficient are set according to characteristics of a surface to be inspected of the inspection object. The visual inspection system according to claim 8 .

10. When the result of the appearance inspection by the first inference unit indicates that the inspection target is a defective product, the appearance inspection by the second inference unit is not performed. The visual inspection system according to claim 6 .

11. The inference model further includes a third inference model configured to receive an image as input and determine whether an object to be inspected included in the image is a good product or not, and a fourth inference model configured to receive an image as input and determine whether an object to be inspected included in the image is a good product or not, The inference unit a third inference image generating unit that generates a third inference image by combining a fifth image obtained by the imaging unit in a state where illumination light from a fifth illumination device is irradiated on the inspection object and a seventh image obtained by the imaging unit in a state where illumination light from a seventh illumination device that faces the fifth illumination device in a third direction is irradiated on the inspection object; a fourth inference image generating unit that generates a fourth inference image by combining a sixth image obtained by the imaging unit in a state where illumination light from a sixth illumination device is irradiated on the inspection object and an eighth image obtained by the imaging unit in a state where illumination light from an eighth illumination device that faces the sixth illumination device in a fourth direction is irradiated on the inspection object; a third inference unit that performs an appearance inspection of the inspection object by inputting the third inference image into the third inference model; a fourth inference unit that performs an appearance inspection of the inspection object by inputting the fourth inference image into the fourth inference model; Furthermore, the third direction intersects with the first direction at 45 degrees, and the fourth direction is perpendicular to the third direction; the output unit outputs a result of the appearance inspection of the inspection object based on a result of the appearance inspection by the first inference unit, a result of the appearance inspection by the second inference unit, a result of the appearance inspection by the third inference unit, and a result of the appearance inspection by the fourth inference unit; the third inference model is trained using images captured under the same lighting conditions as those under which the fifth image and the seventh image were captured; The fourth inference model is trained using images captured under the same lighting conditions as those under which the sixth image and the eighth image were captured. The visual inspection system according to claim 6 .

12. Using a pre-trained inference model, an inference is made on the inspection object using an image obtained by capturing an image of the inspection object using an illumination device having a unidirectional bias with respect to the optical axis of the imaging unit; training the inference model using images captured under the same lighting conditions as those under which the images used in the inference were captured; A visual inspection method comprising:

Images