Image recognition method, image recognition device, inspection method, inspection device, program, and recording medium

Abstract

To provide an image recognition method capable of accurately identifying the reflection on an object from among captured images, even when an object having a complicated shape is captured.SOLUTION: An image recognition method is characterized in that a plurality of images is captured with a camera while changing the relative position between an object and a light source, the plurality of images are input to a trained model, then a static reflection image reflected on the object is extracted from the plurality of images, the extracted static reflection image is removed from the plurality of images, and a dynamic reflection image whose position changes when the relative positions between the object and the light source is changed is extracted from the plurality of images.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image recognition method for recognizing, from an image of an object, reflections of illumination light or the like reflected on the object, and further to an inspection method for inspecting an article by applying the image recognition method. [Background technology]

[0002] In recent years, image recognition, which recognizes objects from images captured by cameras, has been utilized in various fields. For example, in the field of in-vehicle cameras, people on the road are recognized from captured images. In the field of factory automation (FA), parts are recognized from captured images.

[0003] In order to reliably capture an image of an object to be recognized, it is necessary to illuminate the object using a lighting device or increase the imaging sensitivity of the camera. However, this makes it easier for the illumination light, illumination light reflected from the surroundings of the object, and illumination light that has undergone multiple reflections inside the object to be reflected on the surface of the object. If such reflections exist on the surface of the object, the accuracy of the recognition process will decrease when recognizing the object using the captured image.

[0004] Patent Document 1 addresses the problem of a device that uses a camera to capture an image of the area ahead of a vehicle through the windshield to detect people and other objects outside the vehicle, where fixed objects on the vehicle are captured in the windshield.Patent Document 1 therefore discloses a method of pre-storing an image of the windshield under conditions in which fixed objects on the vehicle are reflected as a reflected image, and generating a difference image between the image to be processed and the reflected image.

[0005] Furthermore, Patent Document 2 addresses the problem of the occurrence of reflections (ghosts) due to multiple reflections of illumination light in an apparatus that uses light interference to inspect for streak-like unevenness defects on glass substrates having periodic patterns such as color filters. Therefore, Patent Document 2 utilizes the fact that when the angle or position of the illumination is changed, the position of the reflections (ghosts) moves but the position of the defects does not, and separates the reflections (ghosts) from the streak-like unevenness defects using multiple inspection images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-79706 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-139209 Summary of the Invention [Problem to be solved by the invention]

[0007] However, with conventional methods, there are cases where the accuracy of the recognition process cannot be sufficiently increased when glare is present on the surface of the object. The method disclosed in Patent Document 1 does not address the issue of reflections on the object to be detected (such as a person outside the vehicle), but rather on the windshield between the camera and the object, and therefore does not provide a direct solution. Furthermore, Patent Document 1 compares the luminance values ​​between pixels of a stored reflected image and an image to be processed. Therefore, if the luminance of an object contained in the image to be processed is similar to the luminance of a fixed object on the vehicle contained in a previously stored reflected image, the object may be mistakenly recognized as a reflection.

[0008] Furthermore, in the method disclosed in Patent Document 2, a device that uses light interference to inspect periodic patterns creates a template image for reflections and tracks the reflections between multiple images by pattern matching. The technology in Patent Document 2 is effective when the inspection object is a flat plate such as a color filter, but is difficult to apply to inspection objects with complex shapes. If the inspection object has a complex shape with curved surfaces or unevenness, it is difficult to inspect periodic patterns using light interference in the first place, and there are cases where streak-like unevenness defects and illumination reflections cannot be properly recognized. Therefore, there is a need for an image recognition method that can accurately identify reflections on objects in captured images, even when the object has a complex shape. [Means for solving the problem]

[0009] In a first aspect of the present invention, a plurality of images are captured by a camera while changing the relative position of an object and a light source, a static reflection image reflected in the object is removed from the plurality of images, and a dynamic reflection image whose position changes when the relative position of the object and the light source is changed is extracted from the plurality of images from which the static reflection image has been removed. death , removing the extracted dynamic reflection images from the plurality of images from which the static reflection images have been removed; identifying a defect candidate area from each of the plurality of images when removing the dynamic reflection images; extracting feature points from the identified defect candidate areas; performing an expansion process for each of the plurality of images from which the feature points have been extracted to expand an image area of ​​the feature points; and not extracting, as the dynamic reflection image, any defect candidate area that has an overlapping area with the image of the expanded feature points in each of the plurality of images from which the expanded feature points have been extracted. The image recognition method is characterized by the above.

[0010] A second aspect of the present invention includes a light source that illuminates an object, a camera that can capture an image of the object, and a computer, wherein the computer captures a plurality of images with the camera while changing the relative position of the object and the light source, removes static reflection images that are reflected in the object from the plurality of images, and extracts dynamic reflection images whose positions change when the relative position of the object and the light source is changed from the plurality of images from which the static reflection images have been removed. death , removing the extracted dynamic reflection image from the plurality of images from which the static reflection image has been removed; Execute the process death , When removing the dynamic reflected image, a defect candidate area is identified from each of the plurality of images, feature points are extracted from the identified defect candidate area, and an expansion process is performed on each of the plurality of images from which the feature points have been extracted to expand an image area of ​​the feature points, and in each of the plurality of images from which the expanded feature points have been extracted, the defect candidate area having an overlapping area with the image of the expanded feature points is not extracted as the dynamic reflected image. The image recognition device is characterized by the above.

[0011] In addition, a third aspect of the present invention is a method for detecting defects in an object to be inspected by a camera while changing the relative position of the object to be inspected and a light source, removing from the plurality of images a static reflection image that is reflected in the object to be inspected, and removing a dynamic reflection image that changes position when the relative position of the object to be inspected and the light source is changed from the plurality of images from which the static reflection image has been removed, and inspecting the object to be inspected for defects using the plurality of images from which the static reflection image and the dynamic reflection image have been removed. death , When removing the dynamic reflected image, a defect candidate area is identified from each of the plurality of images, feature points are extracted from the identified defect candidate area, and an expansion process is performed on each of the plurality of images from which the feature points have been extracted to expand an image area of ​​the feature points, and in each of the plurality of images from which the expanded feature points have been extracted, the defect candidate area having an overlapping area with the image of the expanded feature points is not extracted as the dynamic reflected image. The inspection method is characterized by the following.

[0012] A fourth aspect of the present invention is an inspection apparatus including a light source that illuminates an inspection object, a camera that can capture an image of the inspection object, and a computer, wherein the computer performs the following process: capturing a plurality of images with the camera while changing the relative position of the inspection object and the light source; removing from the plurality of images a static reflection image that is reflected in the inspection object; removing from the plurality of images from which the static reflection image has been removed a dynamic reflection image whose position changes when the relative position of the inspection object and the light source is changed; and inspecting the inspection object for defects using the plurality of images from which the static reflection image and the dynamic reflection image have been removed. death , When extracting the dynamic reflected image, a defect candidate area is identified from each of the plurality of images, feature points are extracted from each of the identified defect candidate areas, and an expansion process is performed on each of the plurality of images from which the feature points have been extracted to expand an image area of ​​the feature points, and in each of the plurality of images from which the expanded feature points have been extracted, the defect candidate area having an overlapping area with the image of the expanded feature points is not extracted as the dynamic reflected image. The inspection device is characterized by the above. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide an image recognition method that can accurately identify reflections on an object from a captured image, even when the object has a complex shape. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of an inspection apparatus according to an embodiment. [Figure 2] 1 is a flowchart showing a method for creating a trained model according to an embodiment. [Figure 3](a) An example of a model image according to the embodiment. (b) An example of automatic annotation according to the embodiment. (c) An example of annotation region correction according to the embodiment. (d) An example of a composite image of annotation regions after correction according to the embodiment. [Figure 4] 1 is a flowchart showing an inspection method according to an embodiment. [Figure 5] (a) An example of the captured image of the Nth frame. (b) An example of the captured image of the N+1th frame. [Figure 6] (a) An example of the Nth frame test image. (b) An example of the N+1th frame test image. (c) An example of the Nth frame test image after expansion processing of the feature points. (d) An example of the N+1th frame test image after expansion processing of the feature points. (e) An example showing the product of the feature points after expansion. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the present invention will be described with reference to the drawings. First, an inspection device according to the embodiment and a method for creating a trained model included in the inspection device will be described, followed by an inspection method (image recognition method) according to the embodiment. In the drawings referred to in the following description of the embodiments and examples, elements denoted by the same reference numerals have the same functions unless otherwise specified.

[0016] [Inspection equipment] 1 is a schematic diagram showing the general configuration of an inspection device according to an embodiment of the present invention. The inspection device 101 includes a base 102 that holds a workpiece W (an object to be inspected), a light source 103, a moving device 104 that moves the light source 103, a camera 105 as an imaging device, a computer 106, and a display device 107.

[0017] In this embodiment, the workpiece W is a lens, an optical component, and the inspection device 101 inspects the lens for scratches, dirt, and the like on its surface based on its appearance. The lens is an example of an object to be inspected, and the workpiece W may be another optical component or other component. The inspection device of this embodiment can perform highly reliable inspection processing on objects whose reflection state changes when the illumination position is changed, such as objects with curved or uneven surfaces. Here, reflections on the object to be inspected include the reflection of the illumination light source itself, reflections caused by illumination light reflected by surrounding structures, and images caused by multiple reflections of illumination light inside the object to be inspected. Furthermore, the illumination light referred to here includes not only the light from the light source 103 provided in the inspection device 101, but also light from external light sources such as fluorescent lamps. The workpiece W is held on the base 102 in a position and orientation that allows an image to be captured by the camera 105. The base 102 is fixed to a stand or the like (not shown).

[0018] The light source 103 is an illumination device that irradiates illumination light onto the workpiece W. The light source 103 irradiates the workpiece W with an appropriate amount of light so that the camera 105 can capture an image of the workpiece W with the brightness and contrast required for inspection. The type of light-emitting device used for the light source 103 is not particularly limited, and for example, an LED or a halogen lamp can be used. The light source 103 may have any appropriate light-emitting shape, such as a ring light or a bar light. In addition, an optical element such as a diffuser or a lens may be provided in part of the light source 103 to adjust the orientation characteristics of the illumination light.

[0019] The moving device 104 is a device that movably supports the light source 103, and includes a base that directly supports the light source 103, and a moving mechanism (such as a drive motor) that moves the base. Here, it is desirable that the moving device 104 be capable of moving while supporting the light source 103 so that the optical axis of the illumination light maintains a constant angle with respect to the workpiece W. The moving device 104 can be configured, for example, by a linear stage or an articulated robot.

[0020] The direction in which the moving device 104 moves the light source 103 is preferably a direction in which defects such as scratches on the inspection object can be more easily detected by moving the light source 103, but it may also be a direction in which the movement changes the reflection on the object. When the shape of the workpiece W is highly symmetric, such as the workpiece W illustrated in FIG. 1 , if a bar light with a line-shaped light-emitting element is used as the light source 103, it is preferable to move the light source 103 along a direction intersecting the axis of symmetry of the workpiece W (the vertical direction in FIG. 1 ) (the horizontal direction in FIG. 1 ). Furthermore, when a ring light with a ring-shaped light-emitting element is used as the light source 103, it is preferable to move the light source 103 along a direction parallel to the axis of symmetry of the workpiece W (the vertical direction in FIG. 1 ). In this way, by appropriately setting the direction in which the light source 103 is moved depending on the shape of the workpiece W, the shape of the light-emitting element of the light source used, the shape of the reflections and defects observed on the workpiece, etc., the discernibility of reflections in the inspection process described below can be improved.

[0021] The camera 105 is a digital camera that captures an image of the workpiece W and outputs image data. For example, a digital camera equipped with an area sensor such as a CMOS image sensor or a CCD image sensor is preferably used.

[0022] The computer 106 controls the operation of the inspection apparatus 101 and includes a CPU as a processing unit, a storage unit capable of reading and writing data, and various input / output interfaces and man-machine interfaces. The computer 106 may also include a GPU dedicated to image processing as a processing unit. The computer 106 may include a storage unit such as a flexible disk, optical disk, magneto-optical disk, magnetic tape, non-volatile memory such as a USB memory, or a recording medium such as an SSD. The computer 106 may also include input devices such as a keyboard, jog dial, mouse, pointing device, and voice input device as a man-machine interface. The computer 106 not only controls the operation of each unit of the inspection apparatus 101, but also performs image processing on images captured by the camera 105. The computer 106 can create training data for deep learning using images captured by the camera 105, or perform image processing on images captured for inspection to detect defects in the object. A trained model can also be created in a partial area of ​​the computer 106.

[0023] The display device 107 can operate as a display unit of the inspection device 101. For example, it displays images captured by the camera 105 and images processed by the computer 106, allowing an operator to easily set coordinate values, parameters, etc. The display device 107 is configured using a display device such as a liquid crystal display or an organic EL display.

[0024] In this embodiment, the camera 105 is fixed to a stand or other structure so that the relative position with respect to the workpiece W set on the pedestal 102 does not change. The inspection device 101 can change the relative position between the light source 103 and the workpiece W by moving the light source 103. In other words, by moving the light source 103 with the moving device 104, the irradiation position at which the light source 103 irradiates the workpiece W can be relatively changed.

[0025] As a method for changing the light irradiation position, a configuration in which the light source 103 is moved using the moving device 104 will be described, but the present invention is not limited to this. A configuration in which multiple light sources are arranged at different positions and each light source is turned on in sequence to irradiate the workpiece W from each irradiation position in turn may also be used. 1 illustrates a configuration in which the light source 103 is installed on the opposite side of the workpiece W from the camera 105 to provide transmitted illumination, but the light source 103 may also be installed on the same side of the workpiece W as the camera 105 to provide direct illumination. The layout of the light sources can be set appropriately depending on the properties of the workpiece W (transparent, opaque, etc.).

[0026] [How to create a trained model] Next, a method for creating a trained model according to an embodiment will be described. Fig. 2 is a flowchart illustrating the method for creating a trained model according to an embodiment. In this embodiment, a trained model that extracts static reflection areas contained in an image by inference is created using a deep learning technique. Here, a static reflection area refers to an area in an image reflected on a defect-free object where there is a (static) reflection that does not change much even when the light source 103 is moved (the lighting position is changed).

[0027] Images taken to inspect an object for scratches or other defects contain images of the defects (if any) and the reflection of the lighting on the object. To accurately determine the presence or absence of scratches or other defects, it is important to be able to properly distinguish between scratches and other defects and the reflection of the lighting. Images of defects such as scratches and images of reflected lighting have the following properties. If the relative position of the object and the camera is fixed, when multiple images are captured by changing the lighting position, the position of the image of the defect such as a scratch is unlikely to change between images. On the other hand, when images of reflected lighting on an object are captured in the same manner as described above, there are dynamic reflected images whose position changes between images, and static reflected images whose position does not change. In particular, when the object is a part with a complex shape and a roughened surface, such as an aspherical lens, both types of images are prominent.

[0028] For dynamic reflection images, moving reflection parts can be identified by performing correlation analysis using multiple captured images, making it possible to distinguish them from images of defects such as scratches. On the other hand, for static reflection images, even if multiple inspection images are captured by changing the lighting position, the position of the static reflection image does not change between images, just like images of defects such as scratches, making it difficult to distinguish them from images of defects such as scratches.

[0029] In general, images of defects such as scratches vary greatly depending on the type and shape of the defect, so creating a trained model using machine learning requires the collection of a huge amount of training data, making it difficult to create a trained model.In contrast, static reflection images change little in the image even when the lighting position is changed, so it is relatively easy to collect training data for machine learning. Therefore, in this embodiment, a trained model is created that extracts, by inference, an area in an image where a static reflection image exists (static reflection area) according to the flowchart in FIG.

[0030] First, in step S201, a teacher workpiece for creating a trained model is set on the pedestal 102 of the inspection device 101. Here, a workpiece (sample) without defects such as scratches is selected in advance from among workpieces of the same type as the workpiece W to be inspected and used as the teacher workpiece. It is desirable to prepare multiple teacher workpieces, taking into consideration variations in the shape of the workpiece W to be inspected. The teacher workpiece is set on the pedestal 102 in the same position and orientation as the workpiece W is set at the time of inspection. In other words, it is a position and orientation that reflects the inspected surface of the workpiece when it is imaged by the camera 105.

[0031] Next, in step S202, the computer 106 sends a command to the moving device 104 to move the light source 103 to a predetermined lighting position. As will be described later, during inspection, the light source 103 is moved sequentially to a plurality of lighting positions, and an inspection image is captured by the camera 105 at each lighting position. In step S202, the light source 103 is moved to one of the plurality of lighting positions used during inspection. Next, in step S203, the computer 106 causes the camera 105 to capture an image of the teacher workpiece illuminated by the light source 103, and stores the image data in the storage unit of the computer 106 as a model image.

[0032] Next, in step S204, the computer 106 determines whether or not the imaging of the teacher workpiece set on the base 102 has been completed for all of the multiple lighting positions used during the inspection. If imaging has not been completed for all of the multiple illumination positions used during the inspection (step S204: NO), the process returns to step S202, and the computer 106 moves the light source 103 to a position among the multiple illumination positions to which it has not yet been moved. Thereafter, steps S202 to S204 are repeatedly executed for the set teacher workpiece until imaging has been completed for all of the multiple illumination positions used during the inspection.

[0033] If imaging of the set teacher work has been completed for all of the multiple lighting positions used during inspection (step S204: YES), proceed to step S205 and determine whether imaging of all of the multiple prepared teacher work has been completed. If imaging of all the teacher workpieces has not been completed (step S205: NO), the process returns to step S201, and teacher workpieces that have not yet been imaged are set on the base 102, and the same steps as those described above are repeated. Furthermore, if there is variation in the position and posture of the workpiece each time it is set on the base 102, in order to enrich the training data for machine learning, the same training workpiece may be reset on the base 102 multiple times, and the above-mentioned imaging may be performed each time.

[0034] If the necessary imaging has been completed for all teaching works (step S205: YES), the process proceeds to step S206, where the computer 106 automatically performs annotation processing on the stored model images. In the automatic annotation process, feature amounts are calculated based on preselected feature amounts, and a predetermined feature amount threshold is used to perform binarization so that the reflection area becomes a bright pixel. Feature amounts that can extract only reflections are selected, and the threshold value used in the binarization process is set to a value that can sufficiently extract the reflection area. In this embodiment, a standard deviation is used as the feature amount.

[0035] The standard deviation of all pixels within a certain distance from the pixel of interest is calculated, and this is set as the standard deviation value of the pixel of interest. The standard deviation value is then calculated for each pixel in the model image. If the lighting is bright enough, the brightness on the inspection image is saturated, and the edge pixels of the reflection are extracted. Since the edge part of the reflection becomes a closed area, the closed area is filled in and used as the annotation area.

[0036] For the edge portion of the reflection, neighboring pixels of the pixel in the model image are examined, and pixels with a certain brightness or higher are reclassified as edge pixels. This process allows the closed region to be filled in. A general closed region filling algorithm can be used for this purpose. The feature value is selected from among the average value calculated using a method similar to standard deviation, and the feature value calculated by dividing the model image into a certain number of sections, calculating a co-occurrence matrix for each section, and then calculating the contrast and entropy from the co-occurrence matrix. Alternatively, binarization may be performed using a brightness threshold without calculating the feature value. Furthermore, to incorporate more information about the edge portion of the reflection generated by automatic annotation into the model, the annotation region may be expanded to create an annotation image that is several pixels wider than the actual reflection.

[0037] Once the automatic annotation process has been executed, the process proceeds to step S207, where the computer 106 displays the results of the automatic annotation process on the display device 107. The results of the automatic annotation process are displayed so that the operator (model creator) can easily compare the model image and the annotation area. For example, the model creator can arbitrarily switch between the model image and an annotation area composite image in which the annotation area of ​​the model image is filled in on the screen. Alternatively, an annotation image in which the annotation area is visualized may be created, and the model image and the annotation image may be displayed side by side, or the model image and the annotation area may be layered and composited together and displayed so that they can be compared in a single image.

[0038] Next, in step S208, the model creator checks the results of the automatic annotation process displayed on the display device 107, and if the automatic annotation results differ from the actual reflection area, corrects the annotation area. The model creator performs necessary editing work, such as removing or re-entering the reflection area, using, for example, a mouse. Note that the annotation area may also be corrected using a device such as a touch panel.

[0039] 3(a) to 3(d) are schematic diagrams for explaining a series of steps from displaying the annotation region in step S207 to the correction process in step S208. FIG. 3(a) is an example of a captured model image, showing a teacher's workpiece 301 and a reflection image 302 reflected in the teacher's workpiece 301. Fig. 3(b) shows the automatic annotation region 303 in the automatic annotation process of step S206. By filling the automatic annotation region 303 with any color and embedding it in the model image, the annotation region composite image shown in Fig. 3(c) can be created.

[0040] If the model creator recognizes from the annotation region composite image displayed on the display device 107 that there is a difference between the actual reflected image 302 and the automatic annotation region 303, the model creator corrects it in step S208. The model creator corrects the area where there is a difference by filling it with the same color as the automatic annotation region, and creates a corrected annotation region composite image shown in Figure 3(d) as training data. Note that 306 is the corrected annotation region.

[0041] Next, proceeding to step S209, deep learning is performed using the training data to create a trained model. Using the training data in which model images and annotation regions corresponding to the model images are associated, machine learning is performed on, for example, a U-NET, a type of full-layer convolutional network. Then, machine learning is performed on areas where static reflection images exist, which are reflections whose position does not change even when the lighting position is changed, to create a trained model. The trained model is stored in a memory unit within computer 106 or in another memory unit. Note that, in order to reflect more reflection states in the model, image transformation processing such as affine transformation may be used to translate, rotate, enlarge, or reduce the set of model images and annotation regions, thereby enriching the training data and performing deep learning. In this way, a trained model is formed that can identify areas in a captured image where static reflection images exist.

[0042] [Inspection method] Next, an inspection method (image recognition method) according to an embodiment will be described. A workpiece W as an object to be inspected is illuminated from multiple lighting positions and captured by a camera to obtain multiple inspection images. Then, static reflection images in the inspection images are extracted (estimated) using a trained model and excluded from the inspection images. Furthermore, feature points are extracted from the images contained in the multiple inspection images from which static reflections have been excluded, and the presence or absence of movement of each feature point between the multiple inspection images is detected. If a feature point moves, the image is estimated to be a dynamic reflection image, and if a non-moving feature point is detected, the image is extracted as a defect (candidate).

[0043] FIG. 4 is a flowchart showing an inspection method according to the embodiment. First, in step S501, the workpiece W, which is the object to be inspected, is set on the pedestal 102 of the inspection device 101. The workpiece W is set on the pedestal 102 in the same position and orientation as the teacher workpiece was set when the trained model was created. In other words, the position and orientation are such that when the workpiece W is imaged by the camera 105, the surface to be inspected of the workpiece W is reflected in the captured image.

[0044] Next, in step S502, the computer 106 sends a command to the moving device 104 to move the light source 103 to a predetermined lighting position. During inspection, the light source 103 is moved sequentially to a plurality of lighting positions, and an inspection image is captured by the camera 105 at each lighting position. In step S502, the light source 103 is moved to one of the plurality of lighting positions.

[0045] Next, in step S503, the computer 106 causes the camera 105 to capture an image of the workpiece W illuminated by the light source 103, and stores the image in a storage unit within the computer 106 as inspection image data.

[0046] Next, in step S504, the computer 106 determines whether or not images of the workpiece W set on the base 102 have been taken at all of the multiple illumination positions. If imaging has not been completed for all of the plurality of illumination positions (step S504: NO), the process returns to step S502, and the computer 106 moves the light source 103 to a position among the plurality of illumination positions to which it has not yet been moved. Thereafter, steps S502 to S504 are repeatedly executed for the workpiece W to be inspected until imaging has been completed for all of the plurality of illumination positions.

[0047] If imaging of the set workpiece W to be inspected has been completed for all of the multiple illumination positions (step S504: YES), the process proceeds to step S505. In step S505, the trained model processes the inspection image and infers the area in the inspection image where a static reflection image exists (static reflection area). Then, image data of a defect candidate extraction area, excluding the static reflection area, is extracted from the inspection image. The defect candidate extraction area may contain a dynamic reflection image and, if present, a defect image.

[0048] Next, in step S506, the image data of the defect candidate extraction region is binarized using a preset brightness threshold. This is to extract pixels (defect candidate pixels) that may contain defects from within the defect candidate extraction region. The threshold used in the binarization process is set to a brightness value that is sufficient to extract defects. Note that instead of extracting defect candidate pixels using brightness as the threshold, defect candidate pixels may be extracted by detecting specific spatial frequencies using a band-pass filter. For convenience, the binarization results can be displayed as, for example, defect candidate pixels as bright pixels and pixels other than defect candidates as dark pixels.

[0049] Next, in step S507, feature points are extracted from the defect candidate pixels (defect candidate areas). First, a labeling process is performed to assign the same label to pixels constituting one image among a group of defect candidate pixels, and the number of defect candidate pixels with the same label is counted to calculate the area of ​​the defect candidate. Labels existing within a circular area equidistant from the center of any label are linked to integrate defect candidates with the same label. Note that instead of using the area of ​​the defect candidate, defect candidates may be integrated using, for example, the length and width of the defect candidate, the average brightness, maximum brightness, or minimum brightness value of the point cloud on the coordinates of the defect candidate. Note that at this stage, the integrated defect candidate may be a dynamic reflection image or a defect image.

[0050] Next, the center of gravity of the integrated defect candidates is extracted as a feature point. Here, a case where there is one feature point for the integrated defect candidates is described, but there may be multiple feature points. Also, in step S506, a case where feature points are extracted from an image obtained by performing image processing (binarization) on the inspection image captured by camera 105 is described, but the present invention is not limited to this. That is, feature points may be extracted directly from the inspection image, or, for example, corner coordinate values ​​derived by Harris corner detection may be calculated as feature points.

[0051] Next, in step S508, it is determined whether the feature points of the integrated defect candidates have moved in the images captured by changing the illumination position. By determining whether the feature points have moved, it is possible to extract dynamic reflection images from the defect candidates. With reference to Figures 5(a) and 5(b), it will be explained that dynamic reflection images can be extracted by determining whether the feature points have moved.

[0052] 5(a) is an image captured in the Nth frame of images captured continuously while moving the light source 103, and shows images of a first defect candidate 601 and a second defect candidate 602. Reference numeral 603 denotes a feature point (e.g., the center of gravity) of the first defect candidate 601, and 604 denotes a feature point (e.g., the center of gravity) of the second defect candidate 602. It is not possible to distinguish from FIG. 5(a) alone whether each defect candidate is a defect image or a dynamic reflection image.

[0053] 5(b) is an image captured in the (N+1)th frame among images captured continuously while moving the light source 103, and shows a first defect candidate 601 and a second defect candidate 602. Reference numeral 603 denotes a feature point (e.g., a center of gravity) of the first defect candidate 601, and 604 denotes a feature point (e.g., a center of gravity) of the second defect candidate 602.

[0054] Comparing Figures 5(a) and 5(b), it can be seen that the feature point 603 of the first defect candidate 601 has moved, but the feature point 604 of the second defect candidate 602 has not moved. This allows the first defect candidate 601 to be determined to be a dynamic reflection image. Here, two consecutive frames have been used as an example, but by tracking feature points using a larger number of captured images, it is possible to improve the accuracy of distinguishing between dynamic reflection images and defect images. For example, if the same feature point does not move in a predetermined number or more of consecutive images captured continuously, the image is determined not to be a dynamic reflection image.

[0055] Note that the positions of feature points may change slightly between images due to, for example, quantization errors in binarization processing or the effects of errors that occur in the processing of integrating defect candidate pixels. If the positions of feature points move only slightly, and this is determined to be a dynamic reflection, there is a possibility that the defect image may be erroneously determined to be a dynamic reflection image. Therefore, in the embodiment, even if the positions of feature points change between images, if the amount of change is small, processing may be used to determine that the image is not a dynamic reflection image.

[0056] This process will be described with reference to FIGS. 6(a) to 6(e). FIG. 6(a) shows the inspection image for the Nth frame, and FIG. 6(b) shows the inspection image for the N+1th frame. In this example, the positions of feature points 701 and 702 for the same defect candidate are shifted by several pixels. To absorb this slight shift of several pixels, expansion processing (enlargement of the image region) is performed. Region 703 in FIG. 6(c) and region 704 in FIG. 6(d) are the result of expansion processing of the feature points for the inspection images for the Nth and N+1th frames, respectively. FIG. 6(e) shows the logical product of the feature points for the same defect candidate after expansion processing in FIGS. 6(c) and 6(d). When an intersection region 705 exists, feature points 701 and 702 are considered to have not moved, and this defect candidate is estimated not to be a dynamic reflection. Although Figures 6(a) to 6(e) show examples in which expansion processing is performed within a range of one pixel around a feature point, the range to be expanded is not limited to one pixel, and it is possible to set an appropriate range.

[0057] Returning to FIG. 4, in step S509, those determined to be dynamic reflection images in step S508 are excluded from the defect candidates. Then, in step S510, the remaining defect candidates are determined to be defects. However, if product standards or the like specify the shape or size of scratches that do not need to be treated as defects, the corresponding remaining defect candidates may be excluded from the defect determination. Then, in step S511, the computer 106 compares the number and size of defects determined to be defects in step S510 with the pass / fail criteria registered in advance for the workpiece, and determines whether the workpiece W to be inspected is pass or fail.

[0058] As described above, according to the image recognition method of the embodiment, even if the object to be inspected is a workpiece with a complex shape, such as an aspherical lens, it is possible to automatically identify the image of the workpiece that is reflected in the image with high accuracy. Therefore, it is possible to easily remove the reflected image from the image of the object to be inspected, and it is possible to inspect the presence or absence of defects such as scratches with high accuracy based on the image.

[0059] [Other embodiments] The present invention is not limited to the above-described embodiments and examples, and many modifications are possible within the technical concept of the present invention. A control program capable of executing the above-described trained model creation method and inspection method, and a computer-readable non-transitory recording medium storing the control program are also included in embodiments of the present invention.

[0060] Furthermore, the inspection device and inspection method according to the embodiment can be suitably applied to the inspection of objects with complex shapes that are prone to glare from lighting, such as optical components such as aspherical lenses and decorative exterior components.

[0061] The present invention can also be realized by supplying a program that realizes one or more functions of the embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0062] 101···Inspection device / 102···Base / 103···Light source / 104···Moving device / 105···Camera / 106···Computer / 107···Display device / 301···Teacher's workpiece / 302···Reflected image / 303···Automatic annotation area / 306···Corrected annotation area / 601···First defect candidate / 602···Second defect candidate / 603···Feature points of the first defect candidate / 604···Feature points of the second defect candidate / 701, 702···Feature points / 703, 704···Dilated feature points / 705···Intersection area / W···Workpiece

Claims

1. The camera captures multiple images while changing the relative position of the object and the light source. removing a static image reflected in the object from the plurality of images; extracting a dynamic reflection image whose position changes when the relative position between the object and the light source is changed from the plurality of images from which the static reflection image has been removed; removing the extracted dynamic reflection image from the plurality of images from which the static reflection image has been removed; When removing the dynamic reflection image, Identifying a defect candidate area from each of the plurality of images, and extracting feature points from the identified defect candidate area; performing an expansion process for expanding an image region of the feature points on each of the plurality of images from which the feature points have been extracted; In each of the plurality of images from which the dilation-processed feature points have been extracted, the defect candidate area in which an overlapping area exists in the image of the dilation-processed feature points is not extracted as the dynamic reflection image. An image recognition method comprising:

2. extracting a static reflection image reflected in the object from each of the plurality of images based on a previously prepared identification method, and removing the extracted static reflection image from the plurality of images; 2. The image recognition method according to claim 1.

3. The identification method removes the static reflection image from the plurality of images by inputting the plurality of images into a trained model; The trained model is a trained model generated using as training data annotation images obtained by performing annotation processing on a plurality of images captured by the camera while changing the relative position of a sample of the same type as the target object and the light source.

3. The image recognition method according to claim 2.

4. a light source for illuminating an object; a camera capable of capturing an image of the object; a computer; The computer capturing a plurality of images with the camera while changing the relative position of the object and the light source; removing a static reflection image of the object from the plurality of images; extracting a dynamic reflection image whose position changes when the relative position between the object and the light source is changed from the plurality of images from which the static reflection image has been removed; performing a process of removing the extracted dynamic reflection image from the plurality of images from which the static reflection image has been removed; When removing the dynamic reflection image, Identifying a defect candidate area from each of the plurality of images, and extracting feature points from the identified defect candidate area; performing an expansion process for expanding an image region of the feature points on each of the plurality of images from which the feature points have been extracted; In each of the plurality of images from which the dilation-processed feature points have been extracted, the defect candidate area in which an overlapping area exists in the image of the dilation-processed feature points is not extracted as the dynamic reflection image. An image recognition device characterized by:

5. extracting a static reflection image reflected in the object from each of the plurality of images based on a previously prepared identification method, and removing the extracted static reflection image from the plurality of images; 5. The image recognition device according to claim 4.

6. The identification method removes the static reflection image from the plurality of images by inputting the plurality of images into a trained model; The trained model is a trained model generated using as training data annotation images obtained by performing annotation processing on a plurality of images captured by the camera while changing the relative position of a sample of the same type as the target object and the light source.

6. The image recognition device according to claim 5.

7. The camera captures multiple images while changing the relative position of the object to be inspected and the light source. removing a static reflection image of the object to be inspected from the plurality of images; removing a dynamic reflection image whose position changes when the relative position between the inspection object and the light source is changed from the plurality of images from which the static reflection image has been removed; Inspecting the object for defects using the plurality of images from which the static reflection image and the dynamic reflection image have been removed; When removing the dynamic reflection image, Identifying a defect candidate area from each of the plurality of images, and extracting feature points from the identified defect candidate area; performing an expansion process for expanding an image region of the feature points on each of the plurality of images from which the feature points have been extracted; In each of the plurality of images from which the dilation-processed feature points have been extracted, the defect candidate area in which an overlapping area exists in the image of the dilation-processed feature points is not extracted as the dynamic reflection image. An inspection method characterized by:

8. When removing the dynamic reflection image, Identifying a defect candidate area from each of the plurality of images, and extracting feature points from each of the identified defect candidate areas; extracting the defect candidate areas in which the extracted feature points are present at different positions as the dynamic reflection image; 8. The inspection method according to claim 7.

9. a light source for illuminating an object to be inspected; a camera capable of capturing an image of the inspection object; a computer; The computer Taking a plurality of images with the camera while changing the relative position of the inspection object and the light source; removing a static reflection image of the object to be inspected from the plurality of images; removing a dynamic reflection image whose position changes when the relative position between the inspection object and the light source is changed from the plurality of images from which the static reflection image has been removed; performing a process of inspecting the object to be inspected for defects using the plurality of images from which the static reflection image and the dynamic reflection image have been removed; When extracting the dynamic reflection image, Identifying a defect candidate area from each of the plurality of images, and extracting feature points from each of the identified defect candidate areas; performing an expansion process for expanding an image region of the feature points on each of the plurality of images from which the feature points have been extracted; In each of the plurality of images from which the dilation-processed feature points have been extracted, the defect candidate area in which an overlapping area exists in the image of the dilation-processed feature points is not extracted as the dynamic reflection image. An inspection device characterized by:

10. The computer The dynamic reflection image is Identifying a defect candidate area from each of the plurality of images, and extracting feature points from each of the identified defect candidate areas; extracting the defect candidate areas in which the extracted feature points are present at different positions as the dynamic reflection image; 10. The inspection device according to claim 9.

11. A program for causing the computer included in the inspection device according to claim 9 to execute the processing.

12. A computer-readable non-transitory recording medium on which the program according to claim 11 is recorded.

Images