Image processing apparatus, image processing method, and program

Abstract

To improve detection accuracy of defects when an inspection object has features such as a complicated three-dimensional shape or special surface characteristics, in a technique for performing surface inspection of the inspection object based on a captured image.SOLUTION: A plurality of captured images of an inspection object that are captured while switching illumination directions is acquired. A first inspection for inspecting a surface shape of the inspection object is performed based on a normal line image indicating a normal line direction of each position of the inspection object generated from the plurality of captured images. A second inspection for inspecting the surface shape of the inspection object is performed on the basis of the plurality of captured images, which is different from the first inspection.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to an image processing technique for detecting defects in an object to be inspected. [Background technology]

[0002] There is known a technique for detecting defects on an object surface (inspection surface) based on an image of the object to be inspected. Such a technique is applied to the visual inspection of industrial products. For example, Patent Document 1 describes measuring the surface gradient of the object to be inspected and judging the quality of the object based on the measurement result. The measurement of the surface gradient is performed based on a plurality of images captured by a camera while each of a plurality of lights is transferred in sequence. Specifically, a method of calculating the surface gradient using photometric stereo has been proposed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-240539 A Summary of the Invention [Problem to be solved by the invention]

[0004] When using the method described in Patent Document 1, the accuracy of defect detection may decrease depending on the shape and surface characteristics of the object to be inspected. For example, when the object to be inspected has a concave portion, light may not reach a part of the surface to be inspected. In this case, a black shadow area occurs in the captured image. In addition, the photometric stereo method is used on the premise that the surface to be inspected is a Lambertian surface. However, the surface to be inspected may have a glossy surface. In this case, a white surface reflection area may occur in the captured image due to the influence of surface reflected light from a part of the surface to be inspected. These shadow areas and surface reflection areas may cause a decrease in the accuracy of estimating the surface gradient (or normal), and in this case, the accuracy of defect detection decreases.

[0005] The present invention aims to improve the accuracy of defect detection in a technique for performing surface inspection of an object to be inspected based on captured images when the object to be inspected has features such as a complex three-dimensional shape or special surface characteristics. [Means for solving the problem]

[0006] An image processing apparatus according to an embodiment of the present invention includes the following configuration. An acquisition means for acquiring a plurality of captured images of an object to be inspected, the captured images being captured while switching the illumination direction; a first inspection means for performing a first inspection for inspecting a surface shape of the object to be inspected based on a normal image showing a normal direction at each position of the object to be inspected, the normal image being generated from the plurality of captured images; a second inspection means for performing a second inspection, which is different from the first inspection, for inspecting a surface shape of the inspection target object based on the plurality of captured images; and Equipped with. Effect of the Invention

[0007] In a technique for performing surface inspection of an object to be inspected based on a captured image, it is possible to improve the accuracy of defect detection when the object to be inspected has features such as a complex three-dimensional shape or special surface characteristics. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration example of an inspection system according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the arrangement of an imaging device and an image processing device according to an embodiment. [Diagram 3] FIG. 1 is a conceptual diagram illustrating imaging of an object to be inspected. [Figure 4] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to an embodiment. [Diagram 5] 1 is a flowchart showing a processing flow of an image processing method according to an embodiment. [Figure 6]13 is a flowchart of a process for extracting candidate regions for surface reflection regions and shadow regions. [Figure 7] 13 is a flowchart of a process for determining a surface reflection region and a shadow region. [Figure 8] 6A and 6B are diagrams showing examples of detection results of surface reflection regions and shadow regions. [Figure 9] FIG. 11 is a diagram showing an example of a filter used for inspection. [Figure 10] 13A and 13B are diagrams showing examples of results of defect detection using normal images. [Figure 11] 11A and 11B are diagrams showing examples of results of defect detection using reflected images. [Figure 12] 1 is a flowchart showing a processing flow of an image processing method according to an embodiment. [Figure 13] FIG. 11 is a diagram showing an example of a user interface used for inspection. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] FIG. 1 is a block diagram showing a schematic configuration of an inspection system according to an embodiment. This inspection system includes an imaging device 101 and an image processing device 102. The image processing device 102 according to an embodiment inspects the shape of an object to be inspected using a plurality of captured images of the object to be inspected, the captured images being captured while switching the illumination direction. The imaging device 101 can capture such a plurality of captured images. The imaging device 101 shown in the following example captures a plurality of captured images by capturing an image of the object to be inspected while illuminating it from different directions at separate times.

[0011] The imaging device 101 has an imaging control unit 1011, an imaging unit 1012, and an illumination unit 1013. The imaging control unit 1011 controls the imaging timing of the imaging unit 1012 and the light emission timing of the illumination unit 1013. The illumination unit 1013 has a plurality of light sources arranged at different positions. The imaging control unit 1011 transmits a signal to the illumination unit 1013 to cause a specific light source to emit light at an appropriate timing. Then, the illumination unit 1013 causes the specified light source to emit light in response to this signal.

[0012] The imaging unit 1012 is a two-dimensional imaging device. The imaging unit 1012 may be, for example, a general digital camera or digital video camera. The imaging control unit 1011 transmits a light emission signal to the illumination unit 1013, and then transmits a signal to the imaging unit 1012 to perform imaging. The imaging unit 1012 can perform imaging in response to this signal. The imaging control unit 1011 also transmits a turn-off signal to the illumination unit 1013 at the timing when the imaging unit 1012 completes imaging. In response to this signal, the illumination unit 1013 turns off the light source that is turned on. The imaging control unit 1011 sequentially executes such an operation for each light source according to a preset setting. In this way, the imaging unit 1012 captures a plurality of images of the inspection object illuminated from different directions.

[0013] The image processing device 102 includes an image acquisition unit 1021, an image processing unit 1022, and a detection unit 1023. The image acquisition unit 1021 acquires a plurality of captured images of an object to be inspected, captured while switching the illumination direction. The image acquisition unit 1021 can acquire such captured images from the imaging device 101. The image processing unit 1022 performs image processing using the plurality of captured images. Furthermore, the detection unit 1023 can inspect the object to be inspected based on the processing result by the image processing unit 1022. In this way, the detection unit 1023 can detect defects in the object to be inspected. The image processing unit 1022 and the detection unit 1023 will be described in detail later.

[0014] 2(A) and (B) show the appearance of the imaging device 101 according to one embodiment. FIG. 2(A) shows the illumination unit 1013 and the imaging unit 1012 as viewed from the side. FIG. 2(B) shows the illumination unit 1013 and the imaging unit 1012 as viewed from above. The illumination unit 1013 is a dome-shaped multi-lamp illumination. The illumination unit 1013 shown in FIG. 2(B) has eight light sources (light source 0 to light source 7). The imaging unit 1012 is a camera. These are connected to the imaging control unit 1011. The number of light sources is not limited to eight. In this embodiment, the number of light sources is three or more.

[0015] As described above, the imaging control unit 1011 individually controls the light source 0 to the light source 7 to irradiate the inspection surface of the inspection object 204 with light. The illumination directions from the light source 0 to the light source 7 to the inspection object 204 are different from each other. The imaging control unit 1011 also controls the exposure, focus, and shooting timing of the camera, which is the imaging unit 1012, to capture the inspection surface illuminated with light. For example, the imaging device 101 individually turns on the eight light sources in the order of light source 0, light source 1, ..., and light source 7, and obtains eight captured images of the inspection surface, namely image 0, image 1, ..., and image 7. Image 0 is an image captured when only light source 0 is turned on. Image 1 is an image captured when only light source 1 is turned on.

[0016] The image processing device 102 can detect defects on the inspection surface of the object to be inspected 204 by using the eight captured images thus obtained. The image capture control unit 1011 can be realized by using a PC 201 as shown in FIG. 2(C). The image processing device 102 can also be realized by using a PC 201 as shown in FIG. 2(C). The image capture control unit 1011 and the image processing device 102 may be the same device. For example, the PC 201 functioning as the image processing device 102 may have the function of the image capture control unit 1011. On the other hand, the image capture control unit 1011 and the image processing device 102 may be separate devices.

[0017] Fig. 2(C) shows an example of the hardware configuration of the image processing device 102 in one embodiment. The image processing device 102 can be realized by a computer including a processor and a memory. Fig. 2(C) shows the configuration of a PC 201 that can function as the image processing device 102 as described above. The PC 201 includes a RAM 205, a CPU 206, a ROM 207, and an interface 208. These elements are connected via an internal bus 209.

[0018] In this way, a processor such as CPU 206 executes a program stored in a memory such as RAM 205 or ROM 207, thereby realizing the functions of each unit shown in Fig. 4 etc., which will be described later. For example, the processes shown in the flowcharts of Fig. 5 etc. may be stored as program code in ROM 207. This program code is expanded in RAM 205 and executed by CPU 206.

[0019] In addition, a display 211, a mouse 212, and a keyboard 213 are connected to the PC 201 through an external bus 210 via an interface 208. The display 211 presents information to a user. In addition, the mouse 212 and the keyboard 213 accept input from the user. The above-mentioned lighting unit 1013 and imaging unit 1012 are also connected to the PC 201 through the external bus 210.

[0020] 4 and the like can be realized by a computer, but some or all of the functions of the image processing device 102 may be realized by dedicated hardware. Also, an image processing device according to an embodiment of the present invention may be configured by a plurality of information processing devices connected via a network, for example.

[0021] 3(A)-(B) are conceptual diagrams showing the state in which the inspection target object 204 is imaged by the imaging device 101 shown in FIG. 2(A). In FIG. 3, a workpiece manufactured by injecting resin or the like into a case is shown as the inspection target object 204. FIGS. 3(A) and 3(B) are a cross-sectional view and a top view (i.e., an imaged image) of the inspection target object 204, respectively. When the inspection target object 204 is irradiated with light from diagonally above, depending on the shape of the case, a black-crushed shadow area 301 caused by the light not hitting the inspection surface as shown in FIG. 3(A) may occur in the imaged image. In addition, the inspection target object 204 may have a glossy surface. In this case, depending on the angle of the light from the illumination, a white-out reflection area 302 may occur in the imaged image due to the influence of the surface reflection light on a part of the inspection surface. In general, defects can be detected with a relatively high accuracy by a method of calculating the normal line at each position of the inspection surface and inspecting the surface shape based on the normal line. On the other hand, the presence of these shadow regions and surface reflection regions may reduce the accuracy with which the image processing device 102 estimates the normal to the inspection surface, and in this case, the accuracy with which defects are detected decreases.

[0022] [Embodiment 1] An image processing device 102 according to an embodiment has a first detection unit 403 that performs a first inspection to inspect the shape of an object to be inspected. The image processing device 102 also has a second detection unit 405 that performs a second inspection to inspect the shape of an object to be inspected. The first inspection is performed based on a normal image that indicates the normal direction (or surface gradient) of each position of the object to be inspected. The second inspection is different from the first inspection.

[0023] Specific examples of the first inspection and the second inspection are described below. Also, an example of a method of switching between the first inspection and the second inspection is described. In the present embodiment described below, an area that satisfies a predetermined condition (for example, a shadow area and a surface reflection area) is detected from a captured image of the inspection surface of the object to be inspected. Then, a first inspection based on a normal image is performed on an area that does not meet such a condition. Also, a second inspection is performed on an area that meets such a condition. In this embodiment, an inspection based on a reflection image generated based on a plurality of captured images is performed as the second inspection.

[0024] Fig. 4 is a block diagram showing an example of the functional configuration of the image processing device 102 according to this embodiment. As shown in Fig. 4, in this embodiment, the image processing unit 1022 has a processing condition determination unit 401, a normal image generation unit 402, and a reflected image generation unit 404. In addition, the detection unit 1023 has a first detection unit 403 and a second detection unit 405.

[0025] The processing condition determination unit 401 determines a first region to be the object of the first inspection and a second region to be the object of the second inspection based on the captured image acquired by the image acquisition unit 1021. Here, the processing condition determination unit 401 can determine a region having a predetermined characteristic on the surface of the object to be inspected. In this case, the processing condition determination unit 401 can determine a region not having the predetermined characteristic as the first region, and a region having the predetermined characteristic as the second region. This predetermined characteristic may be a characteristic that reduces the estimation accuracy of the normal. In addition, examples of the region having such a predetermined characteristic include a shadow region and a surface reflection region. The region having such a predetermined characteristic may be a region having a complex three-dimensional shape (e.g., a recess) or a special surface characteristic (e.g., a glossy surface). In this embodiment, the processing condition determination unit 401 determines at least one of the shadow region and the surface reflection region as a region having the predetermined characteristic.

[0026] The normal image generating unit 402 generates a normal image based on a plurality of captured images acquired by the image acquiring unit 1021. This normal image indicates the normal direction of each position of the object to be inspected. The normal image generating unit 402 can generate a normal image for a first region. In addition, the reflected image generating unit 404 generates a composite image of the plurality of captured images acquired by the image acquiring unit 1021. For example, the reflected image generating unit 404 can generate a composite image for a second region. In the following example, the reflected image generating unit 404 generates a reflected image based on a plurality of captured images.

[0027] The first detection unit 403 performs a first inspection based on the normal image generated by the normal image generation unit 402. For example, the first detection unit 403 can detect defects in the surface shape of the object to be inspected by filtering the normal image. The second detection unit 405 performs a second inspection based on the composite image generated by the reflected image generation unit 404. Here, the second detection unit 405 inspects the surface shape of the object to be inspected without using the normal image. For example, the second detection unit 405 can detect defects in the surface shape of the object to be inspected by filtering the composite image.

[0028] Fig. 5 is a flowchart showing the flow of an image processing method performed by the image processing device 102. In this specification, each process (step) in the flowchart is assigned a reference number beginning with S. First, an overview of each process shown in Fig. 5 will be described. Details of these processes will be described later.

[0029] In S500, the image acquisition unit 1021 acquires a plurality of captured images of the inspection target object. As described above, the plurality of captured images are captured while switching the illumination state. The image acquisition unit 1021 can acquire such captured images from the imaging device 101.

[0030] In S501, the processing condition determination unit 401 detects the first region and the second region as described above. For example, the processing condition determination unit 401 can detect a shadow region and a surface reflection region using the captured image group acquired in S500. As already described, the first inspection is performed on the first region, and the second inspection is performed on the second region. That is, the processing of S501 corresponds to selecting an inspection method to be used for inspecting the surface shape for each position on the inspection surface from the first inspection and the second inspection.

[0031] In S502, the normal image generating unit 402 generates a normal image. The normal image generating unit 402 can generate a normal image for the first region. In S503, the first detection unit 403 performs a first inspection based on the normal image generated in S502. In this way, the first detection unit 403 can inspect the inspection surface for defects.

[0032] In S504, the reflected image generating unit 404 generates reflected images of the multiple captured images. For example, the reflected image generating unit 404 can generate a reflected image for a second region. Next, in S505, the second detecting unit 405 performs a second inspection based on the reflected image generated in S504. In this way, the second detecting unit 405 can inspect the inspection surface for defects.

[0033] The detection process of the shadow region and the surface reflection region in S501 will be described in detail below. The processing condition determination unit 401 can detect the second region based on the pixel value of each pixel of the captured image. For example, the processing condition determination unit 401 can determine a region having a pixel value included in a predetermined range in the captured image. The processing condition determination unit 401 can determine such a region as the second region. In particular, the processing condition determination unit 401 can determine a region having a size equal to or larger than a predetermined area in such a region as the second region. As a specific example, the processing condition determination unit 401 can detect pixels brighter than a threshold value or darker than a threshold value, and can detect the second region based on the detection result of such pixels. In this example, the first region refers to a region that is not the second region. In one embodiment, the imaging device 101 captures a color captured image. In this case, the following process can be performed using a grayscale image obtained by image processing on the captured image. For example, Img(k) described below may be a luminance image indicating the luminance value of each pixel obtained based on the color captured image.

[0034] In S501, first, a candidate region of the second region is detected. FIG. 6 shows an example of a flowchart of this process. According to the process shown in FIG. 6, pixels having a pixel value equal to or greater than a first threshold Th1 set in advance are extracted as a candidate region of the surface reflection region. Also, pixels having a pixel value equal to or less than a second threshold Th2 set in advance are extracted as a candidate region of the shadow region. That is, in this example, a region having a pixel value equal to or greater than Th1 or equal to or less than Th2 is extracted as a candidate region. Here, in at least one of the multiple captured images, the pixel values ​​of pixels in the candidate region of the second region (and the second region detected according to FIG. 7) are within a predetermined range (in this example, equal to or greater than Th1 or equal to or less than Th2).

[0035] In S601, the processing condition determination unit 401 generates a binary image Img_buf having the same number of vertical and horizontal pixels as the captured image, and all pixels having a pixel value of "0." Note that in this embodiment, the multiple captured images have the same number of vertical and horizontal pixels. In S602, the processing condition determination unit 401 sets an initial value k to k=0.

[0036] In S603, the processing condition determination unit 401 obtains Img(k) and stores it in memory. Img(k) is the captured image obtained in S500. Eight images Img(0) to Img(7) are obtained in S500. In S604, the processing condition determination unit 401 sets the initial values ​​of pixel position i in the horizontal direction and pixel position j in the vertical direction to i=0 and j=0, respectively.

[0037] In S605, the processing condition determination unit 401 determines whether the pixel value P(i,j) of Img(k) is equal to or greater than a first threshold Th1 set in advance or equal to or less than a second threshold Th2 set in advance. That is, the processing condition determination unit 401 determines whether the pixel (i,j) of Img(k) is in either the candidate region of the surface reflection region (P(i,j)≧Th1) or the candidate region of the shadow region (P(i,j)≦Th2). If neither condition is satisfied, the process proceeds to S606. If either the condition P(i,j)≧Th1 or P(i,j)≦Th2 is satisfied, the pixel (i,j) is included in the candidate region of the reflection region or the shadow region. In this case, the process proceeds to S607. In S607, the processing condition determination unit 401 updates the value of Img_buf(i,j) to "1", and then the process proceeds to S606.

[0038] In S606, the processing condition determination unit 401 determines whether the processing in S605 has been completed for all i. If the processing has not been completed, i is incremented in S609, and the processing then returns to S605. If the processing has been completed, the processing proceeds to S608. In S608, i is initialized to 0, and the processing then proceeds to S610.

[0039] In S610, the processing condition determination unit 401 determines whether the processing in S605 has been completed for all j. If the processing has not been completed, j is incremented in S612, and the processing then returns to S605. If the processing has been completed, the processing proceeds to S611. At this time, in Img_buf, the pixel values ​​in the candidate regions for the surface reflection region and shadow region detected from the image Img(k) have been updated to "1". In S611, j is initialized to 0, and the processing then proceeds to S613.

[0040] In S613, the processing condition determination unit 401 determines whether or not the processing has been completed for all k. If the processing has not been completed, k is incremented in S614, and the processing then returns to S603. If the processing has been completed, the processing in Fig. 6 ends. At this time, the pixel values ​​in the candidate regions for the surface reflection region and the shadow region detected in at least one captured image in Img_buf have been updated to "1".

[0041] Next, among the candidate regions for the second region, a region having a predetermined size is detected as the second region. Fig. 7 shows an example of a flowchart of this process. According to the process shown in Fig. 7, among the candidate regions for the surface reflection region and the shadow region, a region having an area equal to or smaller than a preset threshold value Th_S is excluded.

[0042] In S701, the processing condition determination unit 401 inputs the binary image Img_buf obtained by the processing according to Fig. 6 into the memory. In S702, the processing condition determination unit 401 labels an area having a pixel value of "1" in Img_buf. For example, the processing condition determination unit 401 searches for a pixel having a pixel value of "1" in the binary image, and assigns a label number to this pixel. Then, if there is another pixel having a pixel value of "1" around the pixel thus found, the processing condition determination unit 401 assigns the same label number to this pixel. In this way, a group of adjacent pixels having a pixel value of "1" is recognized as a single object.

[0043] In S703, the processing condition determination unit 401 determines whether the area of ​​each object labeled in S702 is equal to or less than a preset threshold value Th_S. The area of ​​an object can be expressed by the number of pixels to which the same label number is assigned. Then, the processing condition determination unit 401 sets a pixel value of "0" for each pixel in Img_buf that constitutes an object whose area is equal to or less than Th_S. In other words, such pixels are excluded from the candidate areas for the surface reflection area and the shadow area. The processes of S703 and S704 are performed for each object.

[0044] FIG. 8 is a conceptual diagram showing the results of processing according to the flowcharts of FIG. 6 and FIG. 7. FIG. 8(A) shows images 0 to 7 captured by the imaging unit 1012. Images 0 to 7 are captured images captured while being illuminated by light sources 0 to 7 from different angles. The object to be inspected is as shown in FIG. 3. In FIG. 8(A), a shadow area 811 is shown in black, and a surface reflection area 812 is shown in white. Also, as shown in FIG. 8(A), a defect 813 exists on the inspection surface. Since the object to be inspected is illuminated from different directions in each captured image, the generated shadow area and surface reflection area differ between the captured images.

[0045] Fig. 8(B) shows Img_buf obtained by processing according to the flowchart of Fig. 6. In Fig. 8(B), pixels having a pixel value "0" are shown in black, and pixels having a pixel value "1" are shown in white. That is, in any of images 0 to 7, pixels whose pixel values ​​are equal to or greater than the first threshold value Th1 or equal to or less than the second threshold value Th2 are shown in white as candidate regions for surface reflection regions or shadow regions.

[0046] Fig. 8(C) shows Img_buf obtained by processing according to the flowchart of Fig. 7. Compared with the white regions shown in Fig. 8(B), white regions having an area smaller than the threshold value Th_S are excluded. The white regions shown in Fig. 8(C) are reflective regions or shadow regions.

[0047] Furthermore, an inspection area to be inspected may be set in the captured image. FIG. 8(D) shows an example of the inspection area. In FIG. 8(D), the inspection area is shown in white. The processing condition determination unit 401 may extract an area included in the inspection area from the area (black area in FIG. 8(C)) determined to be neither a shadow area nor a surface reflection area shown in FIG. 8(C). A first inspection is performed on the area extracted in this way. The first detection unit 403 described later can generate a normal image for this area. Also, the processing condition determination unit 401 may extract an area included in the inspection area from the area (white area in FIG. 8(C)) determined to be a shadow area or a surface reflection area shown in FIG. 8(C). For example, the processing condition determination unit 401 can generate an image showing the logical product of each pixel between Img_buf shown in FIG. 8(C) and an image showing the inspection area shown in FIG. 8(D). A second inspection is performed on the area extracted in this way (an area having a pixel value "1" in the image showing the logical product). The reflected image generating section 404, which will be described later, can generate a reflected image for this area.

[0048] Next, the details of the normal image generation process in S502 will be described. The normal image generation unit 402 can generate a normal image for an area determined not to be a shadow area or a surface reflection area. The normal image generation unit 402 generates a normal image indicating the normal direction at each position of the object to be inspected based on a plurality of captured images. The normal image generation unit 402 can generate a normal image using a photometric stereo method. In the following example, a case will be described where the normal image generation unit 402 generates a normal image based on eight captured images.

[0049] In photometric stereo, it is assumed that the reflected luminance of the inspection surface is proportional to the dot product of the vector indicating the direction of the light source and the normal vector. In other words, it is assumed that formula (1) holds at each position (x, y) on the inspection surface. Hereinafter, the inspection surface is assumed to be an xy plane consisting of the x-axis and the y-axis. The direction perpendicular to the inspection surface is assumed to be the z-axis. As mentioned above, the inspection surface may have a recess, but in this example it is approximated to a flat surface. In the following example, the position (x, y) of the inspection surface corresponds to the pixel (x, y) of the captured image.

[0050] However, each position on the inspection surface may be represented by three-dimensional coordinates. Even in this case, the following light source matrix L can be obtained based on a light source vector previously measured for each position on the inspection surface. The relationship between each position on the inspection surface and a position on the captured image can also be obtained in advance. Then, the following brightness vector I can be obtained based on such positional relationship.

[0051]

number

[0052] Equation (1) holds for each of light source 0 to light source 7. Summarizing these relationships, equation (2) is obtained. Furthermore, by denoting the vector on the left side of equation (2) as I and the 8-row, 3-column matrix on the right side as L, equation (3) is obtained. Hereinafter, I will be called the luminance vector, and L will be called the light source matrix.

number

[0053] The luminance vector I shown in formula (3) can be calculated from eight captured images. The light source matrix L can be obtained by prior measurement. Therefore, formula (3) can be calculated by the pseudo-normal vector N ρ In the case of three light sources, the light source matrix L is a square matrix with 3 rows and 3 columns. If L is a regular matrix, the inverse matrix L of L is -1 This L -1 Applying to the luminance vector I gives the solution to equation (3).

[0054] On the other hand, when there are four or more light sources, equation (3) becomes an overdetermined system of simultaneous equations. In that case, as shown in equation (4), the pseudo-inverse matrix (L T L) -1 L T By applying to the luminance vector I, a least-squares solution to equation (3) can be found.

[0055]

number

[0056] In this case, the albedo ρ is calculated by the pseudo-normal vector N ρSince it corresponds to the length of the normal vector N, it can be calculated by the formula (5). ρ Since the length of is a unit vector normalized to 1, it can be calculated using equation (6).

[0057] By performing the calculations according to the above formulas (4) to (6) for each position (x, y), a normal image showing the spatial distribution of the normal vector N on the inspection surface can be obtained. Note that an albedo map (reflectance map) showing the spatial distribution of the albedo on the inspection surface can also be obtained by formula (5).

[0058] FIG. 8(E) shows an example of a normal image thus generated. In the normal image shown in FIG. 8(E), the darker the color, the closer the normal is to the vertical direction, and the lighter the color, the further away the normal is from the vertical direction. The area shown in white in FIG. 8(E) is not the target for generating a normal image.

[0059] Next, the first inspection process in S503 will be described. The first detection unit 403 performs a first inspection on an area that is determined not to be a shadow area or a surface reflection area. A method of detecting defects by the first detection unit 403 will be described with reference to FIGS. 9 and 10.

[0060] 9(A) to (D) show examples of detection filters used in the first inspection. In this embodiment, four filters in directions perpendicular to the light source directions are used as the detection filters. The reason for using such filters is that a normal image of a concave-convex defect on the inspection surface can be calculated with high accuracy when the defect is perpendicular to the light source direction. By using these filters, it is possible to detect a region in which the normal direction is disturbed and which is continuous in the filter direction.

[0061] The first detection unit 403 performs filtering using the detection filters shown in Fig. 9(A)-(D) on the two-dimensional distribution of Nx, which is the x-component of the normal vector N, and the two-dimensional distribution of Ny, which is the y-component of the normal vector N, shown in the normal image. Fig. 10(A)-(D) show the average values ​​for each pixel of the results of filtering on Nx and the results of filtering on Ny. Fig. 10(A)-(D) correspond to the filters shown in Fig. 9(A)-(D), respectively.

[0062] 10(E)-(H) show the results of excluding the areas other than the first area, which is the target of the first inspection, from FIG. 10(A)-(D) (i.e., the pixel values ​​in this area are set to 0). As described above, the shadow area and surface reflection area, which are excluded from the target of the first inspection, are shown as white areas in FIG. 8(C). Note that the boundaries between the shadow area and the surface reflection area and other areas may be further excluded from the target of the first inspection. In this case, the white area in FIG. 8(C) can be expanded by performing morphological transformation or the like. The area corresponding to the white area after such expansion may be excluded from the detection results shown in FIG. 10(A)-(D).

[0063] The detection result shown in FIG. 10(I) is obtained from the detection results shown in FIG. 10(E) to (H). Each pixel in the detection result shown in FIG. 10(I) indicates the maximum value of each pixel in the detection results shown in FIG. 10(E) to (H). FIG. 10(I) shows the detection result in the first inspection. That is, the pixels in the defective area detected in the target area of ​​the first inspection have pixel values ​​different from those in other areas. In this way, the image shown in FIG. 10(I) shows the distribution of defects detected by the second inspection.

[0064] Next, the generation process of the reflected image in S504 will be described. The reflected image generating unit 404 generates a reflected image for an area determined to be a shadow area or a surface reflection area. In the following example, a case where the reflected image generating unit 404 generates a reflected image based on eight captured images will be described. The reflected image can be obtained by combining a plurality of captured images. In this embodiment, each pixel of the reflected image generated by the reflected image generating unit 404 has the maximum value of the pixel values ​​of the corresponding pixels in the eight captured images. For example, the reflected image generating unit 404 can compare the pixel values ​​of the pixels at the same location in the eight captured images. Then, the reflected image generating unit 404 can use the maximum value of these pixel values ​​for the eight captured images as the pixel value of the pixel at the same location in the reflected image. FIG. 8(F) shows an example of the reflected image thus generated. In FIG. 8(F), the darker the color, the lower the reflection luminance. The area shown in white in FIG. 8(F) is not a target for generating a reflected image.

[0065] In one embodiment, the imaging device 101 captures a color image. In this case, the reflection image generating unit 404 can generate a reflection image for each plane (e.g., R, G, and B). However, in general, the presence of a defect is rarely indicated only in one of the R, G, or B planes. For this reason, a reflection image may be generated based on only the G component, and a second inspection may be performed based on this reflection image. As another example, a reflection image may be generated based on an image in another color space (e.g., XYZ or L*a*b*), and a second inspection may be performed based on this reflection image. Furthermore, a reflection image may be generated based on only the Y component or the L* component, and a second inspection may be performed based on this reflection image.

[0066] Next, the second inspection process in S505 will be described. The second detection unit 405 performs a second inspection on the area determined to be a shadow area or a surface reflection area. A method of detecting defects by the second detection unit 405 will be described with reference to FIGS. 9 and 11.

[0067] In this embodiment, the second detection unit 405 inspects the surface shape of the object to be inspected based on the reflected image generated in S504. The second detection unit 405 can also detect defects in the surface shape of the object to be inspected by filtering the reflected image. The second detection unit 405 can use the detection filter shown in FIG. 9 as in the first detection unit 403. This is because, like the normal image, when an uneven defect on the inspection surface is perpendicular to the light source direction, the pixel value of the position of the defect in the captured image becomes large, making it easier to detect the defect. However, the normal image used by the first detection unit 403 for detection and the reflected image used by the second detection unit 405 for detection each represent different physical characteristics. For this reason, the shapes of the filters used by the first detection unit 403 and the second detection unit 405 do not need to be the same. For example, the filters of the first detection unit 403 and the second detection unit 405 may be adjusted so that the first detection unit 403 and the second detection unit 405 can perform appropriate detection processing.

[0068] The second detection unit 405 can perform filtering using the detection filters shown in Fig. 9 on each of the two-dimensional distributions of the R, G, and B components shown in the reflected image generated in S504. Figs. 11(A) to (D) show the results of filtering the reflected image shown in Fig. 8(F). Figs. 11(A) to (D) show the average values ​​for each pixel of the results of filtering the R, G, and B components of the reflected image. Each of Figs. 11(A) to (D) corresponds to each of the filters shown in Figs. 9(A) to (D).

[0069] 11(E)-(H) show the results of excluding the areas other than the second area, which is the target of the second inspection, from FIG. 11(A)-(D) (i.e., the pixel values ​​in this area are set to 0). As described above, the shadow area and surface reflection area, which are the target of the second inspection, are shown as white areas in FIG. 8(C). Note that the boundaries between the shadow area and the surface reflection area and the other areas may be further excluded from the target of the second inspection. In this case, the black area in FIG. 8(C) can be expanded by performing morphological transformation or the like. The area corresponding to the black area after such expansion may be excluded from the detection results shown in FIG. 11(A)-(D).

[0070] The detection result shown in FIG. 11(I) is obtained from the detection results shown in FIG. 11(E)-(H). Each pixel of the detection result shown in FIG. 11(I) indicates the maximum value of each pixel of the detection result shown in FIG. 11(E)-(H). FIG. 11(I) shows the detection result in the second inspection. That is, the pixels of the defective area detected in the target area of ​​the second inspection have pixel values ​​different from those of other areas. In this way, the image shown in FIG. 11(I) shows the distribution of defects detected by the second inspection.

[0071] As described above, the first detection unit 403 can output the result of the first inspection in the first region (in this example, the region other than the shadow region and the surface reflection region). Furthermore, the second detection unit 405 can output the result of the second inspection in the second region (in this example, the shadow region and the surface reflection region). The detection unit 1023 may have an output unit (not shown) that outputs these pieces of information. Furthermore, this output unit may combine the result of the first inspection with the result of the second inspection and output the combined result. For example, the output unit may combine an image showing a defect detected in the first region (for example, FIG. 10(I)) with an image showing a defect detected in the second region (for example, FIG. 11(I)) and output the combined image.

[0072] As described above, according to this embodiment, in order to inspect the surface shape of the object to be inspected, the image processing device can perform both the first inspection based on the normal image and the second inspection different from the first inspection. Therefore, it is possible to improve the inspection accuracy when the surface of the object to be inspected has a feature that reduces the estimation accuracy of the normal. For example, an inspection based on the reflection image is performed for the shadow area and the surface reflection area, and an inspection based on the normal image is performed for the other areas. Therefore, defects can be detected in both the shadow area or the surface reflection area and the other areas. In particular, according to this method, in areas other than the shadow area and the surface reflection area, it is expected that the inspection accuracy will be improved by performing the first inspection based on the normal image compared to the case of performing the second inspection. On the other hand, it is expected that the inspection accuracy will be improved by performing the second inspection compared to the case of performing the first inspection in the shadow area and the surface reflection area.

[0073] [Embodiment 2] In the following, a method of switching between the first inspection and the second inspection, and another example of the method of the second inspection will be described. In the second embodiment, when a shadow area or a surface reflection area is not detected from the inspection surface, the first inspection based on the normal image is performed. On the other hand, when a shadow area or a surface reflection area is detected from the inspection surface, the normal image is not generated and the second inspection is performed. Also, in the second embodiment, instead of performing the second inspection based on the reflection image, the second inspection based on the filter processing of the captured image group is performed. The differences from the first embodiment will be described below.

[0074] 12 is a flowchart showing the flow of the image processing method performed by the image processing device 102. S500 and S501 are performed in the same manner as in the first embodiment.

[0075] In S1201, the processing condition determination unit 401 determines whether to perform the first inspection or the second inspection. That is, in S1201, an inspection method used to inspect the surface shape of the inspection target object is selected from the first inspection and the second inspection. If an area having a predetermined characteristic is not detected from the surface of the inspection target object, the processing condition determination unit 401 determines to perform the first inspection in S1202 to S1203. On the other hand, if an area having a predetermined characteristic is detected from the surface of the inspection target object, the processing condition determination unit 401 determines to perform the second inspection in S1205. In this embodiment as well, the area having the predetermined characteristic may be at least one of a shadow area and a surface reflection area.

[0076] In the present embodiment, the processing condition determination unit 401 first determines whether or not a shadow area or a surface reflection area has been detected in S501. For example, the processing condition determination unit 401 can generate an image showing the logical product of each pixel between Img_buf shown in FIG. 8(C) and an image showing the inspection area shown in FIG. 8(D). Then, the processing condition determination unit 401 can determine that a shadow area or a surface reflection area has been detected from the inspection surface if this image has an area having a pixel value of "1" (i.e., a shadow area or a surface reflection area in the inspection area). Also, the processing condition determination unit 401 can determine that a shadow area or a surface reflection area has not been detected from the inspection surface if the entire pixel value of this image is "0". If a shadow area or a surface reflection area has not been detected, the process proceeds to S1202. If a shadow area or a surface reflection area has been detected, the process proceeds to S1205.

[0077] In S1202, the normal image generating unit 402 generates a normal image in the same manner as in S502. In addition, in S1203, the first detecting unit 403 performs a first inspection based on the normal image in the same manner as in S503. In this manner, the first detecting unit 403 can perform the first inspection in response to the fact that a region having a predetermined characteristic is not detected from the surface of the object to be inspected.

[0078] In S1205, the second detection unit 405 performs the second inspection. In this way, the second detection unit 405 can perform the second inspection in response to the detection of an area having a predetermined characteristic from the surface of the object to be inspected. In this embodiment, the second detection unit 405 inspects the surface shape of the object to be inspected based on each of the multiple captured images without using a normal image. In this way, in the second inspection in this embodiment, the captured images acquired by the image acquisition unit 1021 are used. In addition, the second detection unit 405 can detect defects in the surface shape of the object to be inspected by filtering each of the multiple captured images. In this embodiment, the same reflected image as in the first embodiment is not used. Therefore, in this embodiment, the image processing unit 1022 does not need to have the reflected image generation unit 404.

[0079] Specifically, the second detection unit 405 can perform filter processing using any one of the filters shown in Fig. 9(A) to (D) on each of the eight captured images. As described above, defects extending in a direction perpendicular to the light source direction are noticeable in the captured images. For this reason, in this embodiment, instead of applying all types of filters to all captured images, a filter perpendicular to the light source direction is applied to each captured image. This method can achieve high-speed processing. For example, only the filter shown in Fig. 9(A) can be applied to image 0, which is a captured image of the inspection surface illuminated by light source 0 on the left side. Also, only the filter shown in Fig. 9(B) can be applied to image 1, which is a captured image of the inspection surface illuminated by light source 1 diagonally above to the left.

[0080] Based on the results of the filter processing for each of these eight captured images, a detection result by the second processing can be generated. The detection result may be a two-dimensional image showing the maximum value of each pixel in the results of the eight filter processing, as in the first embodiment. The image thus obtained shows the distribution of defects detected by the second inspection. Also, as in the first embodiment, the inspection area can be limited by using the mask image shown in FIG. 8(D). In this way, only defects present on the detection surface can be detected.

[0081] As described above, according to this embodiment, in order to inspect the surface shape of an object to be inspected, the image processing device can perform both a first inspection based on a normal image and a second inspection different from the first inspection. Therefore, appropriate inspection can be performed both in cases where the surface of the object to be inspected has a feature that reduces the estimation accuracy of the normal and in cases where the surface does not have such a feature. For example, when the object to be inspected has a shadow area or a surface reflection area, an inspection based on a captured image is performed, and when the object to be inspected does not have these areas, an inspection based on a normal image is performed. Therefore, appropriate inspection can be performed for both an object to be inspected that has a shadow area or a surface reflection area and an object to be inspected that does not have these areas.

[0082] [Embodiment 3] Another example of a method for switching between the first and second inspections will be described below. In this embodiment, the first or second inspection is selected according to a user input. Differences from the second embodiment will be described below.

[0083] 13 shows a user interface used for user input. This user interface can be used by the user to input information about the object to be inspected. The processing condition determination unit 401 can display such a user interface on a display device such as the display 211. In addition, the processing condition determination unit 401 can obtain the user input made on the user interface via the mouse 212 or the keyboard 213.

[0084] Toggle button 1301 is used to set detailed conditions. Toggle button 1302 is used to set whether or not the inspection surface of the object to be inspected (workpiece) has a recess. Toggle button 1303 is used to set whether or not the inspection surface of the workpiece is glossy. Toggle button 1304 is used to set the movement of the workpiece. The user can input that the workpiece to be photographed is moving or stationary. For example, the object to be inspected may be photographed while moving on a conveyor belt or the like. If "Photograph while moving" is selected, the user can further input the movement speed of the workpiece in the text box.

[0085] In this way, the user can set conditions related to the inspection, such as information about the object to be inspected. As shown in Fig. 13, the information about the object to be inspected can indicate at least one of the shape characteristics of the object to be inspected (e.g., the presence or absence of a recess) and the surface characteristics of the object to be inspected (e.g., the presence or absence of gloss). The information about the object to be inspected can also indicate at least one of whether the object to be inspected is moving and the moving speed of the object to be inspected.

[0086] Then, the processing condition determination unit 401 can determine whether to perform the first inspection or the second inspection according to the information on the object to be inspected set by the user. For example, the processing condition determination unit 401 can select the second inspection when the inspection surface is likely to include a shadow area or a surface reflection area. The processing condition determination unit 401 can determine that the inspection surface is likely to include a shadow area or a surface reflection area when the inspection surface has a recess or gloss. In the example of FIG. 13, when "has recess" is selected with the toggle button 1302, or when "has gloss" is selected with the toggle button 1303, the processing condition determination unit 401 can select the second inspection. In this way, the processing condition determination unit 401 can determine to perform the second inspection when the information on the object to be inspected satisfies at least one of the conditions that the object to be inspected has a recess and that the object to be inspected has gloss.

[0087] Furthermore, the processing condition determination unit 401 can select the second inspection when the object to be inspected is moving. As another example, the processing condition determination unit 401 can select the second inspection when the object to be inspected is moving faster than a predetermined threshold. This is because, when a normal image is generated using an image of the object to be inspected captured while moving, the accuracy of estimating the normal may decrease due to a shift in the object position due to a difference in image capturing timing and blurring due to movement. In the example of FIG. 13, when "take a picture while moving" is selected with the toggle button 1304, the processing condition determination unit 401 can select the second inspection. In another example, in the example of FIG. 13, when "take a picture while moving" is selected with the toggle button 1304 and the input moving speed of the work is faster than a predetermined threshold, the processing condition determination unit 401 can select the second inspection. In this way, the processing condition determination unit 401 can determine to perform the second inspection when the information on the object to be inspected satisfies at least one of the conditions that the object to be inspected is moving and that the moving speed of the object to be inspected exceeds a threshold.

[0088] Furthermore, the processing condition determination unit 401 can determine to perform the first inspection when the information on the object to be inspected does not satisfy the above conditions. For example, the processing condition determination unit 401 can select the first inspection based on the normal image when the object to be inspected is photographed in a stationary state. Furthermore, the processing condition determination unit 401 can select the first inspection when the inspection surface does not include a shadow area and a surface reflection area. The processing condition determination unit 401 can determine that the inspection surface does not have a shadow area when the inspection surface does not have a recessed portion. Furthermore, the processing condition determination unit 401 can determine that the inspection surface does not have a surface reflection area when the inspection surface does not have a glossy surface. In the example of FIG. 13, the processing condition determination unit 401 can select the first inspection when "no recessed portion" is selected with the toggle button 1302, "no gloss" is selected with the toggle button 1303, and "photographing a stationary object" is selected with the toggle button 1304.

[0089] Note that when "Auto" is selected with the toggle button 1301, since no conditions are set, the possibility cannot be denied that the estimation accuracy of the normal line may decrease. For this reason, the processing condition determination unit 401 may decide to perform the second inspection without generating a normal line image. Also, for the same reason, when "Not Designated" is selected with any of the toggle buttons 1302, 1303, or 1304, the processing condition determination unit 401 may decide to perform the second inspection.

[0090] The processing according to this embodiment can be performed according to the flowchart shown in FIG. 12. In this embodiment, S501 can be omitted. In addition, in S1201, the processing condition determination unit 401 selects the first inspection or the second inspection according to the user input as described above. If the first inspection is selected, the processing proceeds to S1202. If the second inspection is selected, the processing proceeds to S1205. By performing the first inspection or the second inspection in this manner, the inspection of the inspection target object in this embodiment is performed.

[0091] As described above, according to this embodiment, it is possible to inspect the surface shape of an object to be inspected by an appropriate inspection method based on the inspection conditions set by the user.

[0092] [Variations] So far, various specific examples of the second inspection have been described. Also, various methods of switching between the first inspection and the second inspection have been described. These methods can be used in any combination. For example, in the first embodiment, the second inspection based on the reflected image was performed on the shadow area and the surface reflection area, which are part of the inspection surface. Also, in the second embodiment, when the shadow area or the surface reflection area is detected, the second inspection based on the captured image was performed on the entire inspection surface. On the other hand, the second inspection based on the captured image may be performed on the shadow area and the surface reflection area, which are part of the inspection surface, as in the second embodiment. Also, when the shadow area or the surface reflection area is detected, the second inspection based on the reflected image may be performed on the entire inspection surface as in the first embodiment.

[0093] Furthermore, instead of switching between the first inspection and the second inspection depending on the region or the object to be inspected as in the above embodiment, both the first inspection and the second inspection may be performed. In this case, defects can be detected by combining the results of the first inspection and the results of the second detection. For example, the results of the first inspection and the results of the second detection may be weighted averaged. In this case, the weights used in the combination can be adjusted so that the weight of the result of the first inspection in the shadow region or the surface reflection region is smaller than the weight of the result of the first inspection in the other regions. In addition, the weights used in the combination may be adjusted so that the weight of the result of the first inspection in the case where the inspection surface has a shadow region or a surface reflection region is smaller than the weight of the result of the first inspection in the case where the inspection surface does not have a shadow region or a surface reflection region.

[0094] The shape and number of filters used in the first inspection and the second inspection are not limited to the above examples. For example, in order to improve the inspection accuracy, the number of filters used in the second inspection can be increased. For example, the number of filters used in the second inspection may be greater than the number of filters used in the first inspection. In addition, the appropriate shape (width and size) of the filter varies depending on the size of the defect occurring on the inspection surface. Therefore, when the size or type of the defect occurring is unknown, the variation of the filters used can be increased. Also, FIG. 9 shows an anisotropic filter that is suitable for detecting linear defects. However, an isotropic filter can also be used as the filter. An isotropic filter is suitable for detecting granular defects.

[0095] In addition, in the first inspection and the second inspection, it is not essential to detect defects using a filter. For example, a reference normal image, a reflected image, or a captured image may be prepared in advance. In this case, the first detection unit 403 may inspect the surface shape of the object to be inspected based on a comparison between the reference reflected image or captured image and the reflected image generated by the normal image generation unit 402 or the captured image acquired by the image acquisition unit 1021. In addition, the second detection unit 405 may inspect the surface shape of the object to be inspected based on a comparison between the reference normal image and the normal image generated by the reflected image generation unit 404.

[0096] In the above embodiment, a map indicating the positions of the detected defects is output as the inspection result. However, the format of the output data is not limited to this. For example, a quantitative evaluation value indicating the number or size of the defects may be output as the inspection result. For example, the quantitative evaluation value can be calculated based on the map. In this case, the average value or integral value of the values ​​for the entire map can be calculated.

[0097] In the above embodiment, the inspection target object 204 is illuminated by the multiple light sources sequentially emitting light. However, the configuration of the imaging device 101 is not limited to this. For example, multiple light sources that are relatively close to each other may emit light simultaneously. Also, multiple light emitting elements may be densely arranged, and these light emitting elements may emit light simultaneously as a single light source.

[0098] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0099] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) An acquisition means for acquiring a plurality of captured images of an object to be inspected, the captured images being captured while switching the illumination direction; a first inspection means for performing a first inspection for inspecting a surface shape of the object to be inspected based on a normal image showing a normal direction at each position of the object to be inspected, the normal image being generated from the plurality of captured images; a second inspection means for performing a second inspection, which is different from the first inspection, for inspecting a surface shape of the inspection target object based on the plurality of captured images; and An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the first inspection means detects defects in the surface shape of the object to be inspected by filtering the normal image. (Item 3) The image processing device according to any one of items 1 to 2, characterized in that the second inspection means inspects the surface shape of the object to be inspected based on each of the multiple captured images or a composite image of the multiple captured images without using the normal image. (Item 4) The image processing device according to any one of items 1 to 3, characterized in that the second inspection means detects defects in the surface shape of the object to be inspected by filtering each of the plurality of captured images or a composite image of the plurality of captured images. (Item 5) 5. The image processing device according to any one of items 1 to 4, further comprising a determination unit for determining an area having a predetermined characteristic on the surface of the object to be inspected. (Item 6) 6. The image processing device according to item 5, wherein the region having the predetermined characteristic is at least one of a shadow region and a surface reflection region. (Item 7) 7. The image processing device according to any one of items 1 to 6, further comprising a determination unit for determining an area of ​​the captured image having pixel values ​​that fall within a predetermined range. (Item 8) 8. The image processing device according to item 7, characterized in that in at least one of the plurality of captured images, pixel values ​​of pixels within the region are within the predetermined range. (Item 9) 9. The image processing device according to any one of items 1 to 8, further comprising a determination unit for determining each of the area to be subjected to the first inspection and the area to be subjected to the second inspection. (Item 10) 10. The image processing device according to item 9, wherein an area of ​​the surface of the object to be inspected that does not have a predetermined feature is an area that is the subject of the first inspection, an area that has the predetermined feature is an area that is the subject of the second inspection, and the area that has the predetermined feature is at least one of a shadow area and a surface reflection area. (Item 11) 9. The image processing device according to any one of items 1 to 8, further comprising a determination unit that determines whether the first inspection or the second inspection is to be performed. (Item 12) the determination means determines to perform the first inspection in response to a region having a predetermined characteristic not being detected on the surface of the object to be inspected, and determines to perform the second inspection in response to a region having the predetermined characteristic being detected on the surface of the object to be inspected, Item 12. The image processing device according to item 11, wherein the region having the predetermined characteristic is at least one of a shadow region and a surface reflection region. (Item 13) Item 12. The image processing device according to item 11, wherein the determination means determines whether to perform the first inspection or the second inspection depending on information about the object to be inspected set by a user. (Item 14) Item 14. The image processing device according to item 13, characterized in that the information relating to the object to be inspected indicates at least one of a shape characteristic of the object to be inspected and a surface characteristic of the object to be inspected. (Item 15) The image processing device described in any one of items 13 to 14, characterized in that the determination means determines to perform the second inspection when the information about the object to be inspected satisfies a condition including at least one of the following: the object to be inspected has a recess and the object to be inspected is glossy, and determines to perform the first inspection when the information about the object to be inspected does not satisfy the condition. (Item 16) 16. The image processing device according to any one of items 13 to 15, characterized in that the information about the object to be inspected indicates at least one of whether the object to be inspected is moving or not and a moving speed of the object to be inspected. (Item 17) 17. The image processing device according to any one of items 13 to 16, characterized in that the determination means determines to perform the second inspection when the information on the object to be inspected satisfies a condition including at least one of the following: the object to be inspected is moving and the moving speed of the object to be inspected exceeds a threshold value; and determines to perform the first inspection when the information on the object to be inspected does not satisfy the condition. (Item 18) 18. The image processing device according to any one of items 13 to 17, wherein the determination means displays on a display device a user interface used by a user to input information about the object to be inspected. (Item 19) An image processing method performed by an image processing device, comprising: acquiring a plurality of captured images of an object to be inspected, the captured images being captured while switching the illumination direction; selecting an inspection method used for inspecting the surface shape of the inspection target object from among a first inspection for inspecting the surface shape of the inspection target object based on a normal image that indicates a normal direction at each position of the inspection target object generated from the multiple captured images, and a second inspection that is different from the first inspection and inspects the surface shape of the inspection target object based on the multiple captured images; Inspecting a surface shape of the object to be inspected according to a selected inspection method; 13. An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing device according to any one of items 1 to 18.

[0100] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0101] 101: imaging device, 102: image processing device, 401: processing condition determination unit, 402: normal image generation unit, 403: first detection unit, 404: reflected image generation unit, 405: second detection unit, 1011: imaging control unit, 1021: image acquisition unit, 1022: image processing unit

Claims

1. Acquisition means for acquiring multiple images of an object to be inspected, while switching the direction of illumination, A determination means for determining the area to be inspected in the first inspection and the area to be inspected in the second inspection, A first inspection means that performs the first inspection of the surface shape of the object to be inspected based on a normal image that shows the normal direction of each position of the object to be inspected, generated from the plurality of captured images, A second inspection means that performs a second inspection, which is different from the first inspection, and which inspects the surface shape of the object to be inspected based on the plurality of captured images, Equipped with, An image processing apparatus characterized in that a region of the surface of the object to be inspected that does not have a predetermined feature is the region to be inspected in the first inspection, a region that has the predetermined feature is the region to be inspected in the second inspection, and the region that has the predetermined feature is at least one of a shadow region and a surface reflection region.

2. The image processing apparatus according to claim 1, wherein the first inspection means is characterized by detecting defects in the surface shape of the object to be inspected by filtering the normal image.

3. The image processing apparatus according to claim 1, wherein the second inspection means inspects the surface shape of the object to be inspected based on each of the plurality of captured images or a composite image of the plurality of captured images, without using the normal image.

4. The image processing apparatus according to claim 1, wherein the second inspection means is characterized by detecting defects in the surface shape of the object to be inspected by filtering each of the plurality of captured images or a composite image of the plurality of captured images.

5. The image processing apparatus according to claim 1, wherein the determination means determines a region having predetermined characteristics on the surface of the object to be inspected.

6. The image processing apparatus according to claim 1, wherein the determination means determines a region having pixel values ​​that are included in a predetermined range of the captured image.

7. The image processing apparatus according to claim 6, characterized in that the pixel values ​​of pixels within the region are included in the predetermined range in at least one of the plurality of captured images.

8. The image processing apparatus according to claim 1, characterized in that the determination means determines whether to perform the first inspection or the second inspection.

9. The image processing apparatus according to claim 1, characterized in that the determination means determines to perform the first inspection in response to the detection of no region having predetermined characteristics from the surface of the object to be inspected, and determines to perform the second inspection in response to the detection of a region having predetermined characteristics from the surface of the object to be inspected.

10. The image processing apparatus according to claim 8, characterized in that the determination means determines whether to perform the first inspection or the second inspection according to the information of the object to be inspected set by the user.

11. The image processing apparatus according to claim 10, characterized in that the information relating to the object to be inspected includes at least one of the shape characteristics of the object to be inspected and the surface characteristics of the object to be inspected.

12. The image processing apparatus according to claim 10, characterized in that the determination means determines to perform the second inspection if the information relating to the object to be inspected satisfies the condition that the object to be inspected has a recess and the object to be inspected has gloss, and determines to perform the first inspection if the information relating to the object to be inspected does not satisfy the condition.

13. The image processing apparatus according to claim 10, characterized in that the information relating to the object to be inspected indicates at least one of whether or not the object to be inspected is moving and the speed at which the object to be inspected is moving.

14. The image processing apparatus according to claim 10, characterized in that the determination means determines to perform the second inspection when the information relating to the object to be inspected satisfies a condition including at least one of the conditions that the object to be inspected is moving and the movement speed of the object to be inspected exceeds a threshold, and determines to perform the first inspection when the information relating to the object to be inspected does not satisfy the condition.

15. The image processing apparatus according to claim 10, wherein the determination means displays a user interface on a display device for the user to input information about the object to be inspected.

16. An image processing method performed by an image processing device, A process of acquiring multiple images of the object to be inspected, while switching the direction of illumination, A process for determining the area to be inspected in the first inspection and the area to be inspected in the second inspection, The first inspection step involves inspecting the surface shape of the object to be inspected based on a normal image that shows the normal direction of each position of the object to be inspected, generated from the plurality of captured images. A step of performing a second inspection, which is different from the first inspection, and which inspects the surface shape of the object to be inspected based on the plurality of captured images, Includes, An image processing method characterized in that a region of the surface of the object to be inspected that does not have a predetermined feature is the region to be inspected in the first inspection, a region that has the predetermined feature is the region to be inspected in the second inspection, and the region that has the predetermined feature is at least one of a shadow region and a surface reflection region.

17. A program for causing a computer to function as an image processing device according to any one of claims 1 to 15.

Images