Image processing device and image processing method

Abstract

To provide a technique that enables detection of fine linear irregularities even when a posture of an object under inspection changes.SOLUTION: An image processing device provided herein is configured to acquire a normal line map indicating a spatial distribution of normal lines on a surface of an object, acquire a directional vector of a virtual light source corresponding to a change in the posture of the object, and perform detection processing for detecting linear irregularities on the surface based on the normal line map and the directional vector.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for detecting linear irregularities on a surface to be inspected. [Background technology]

[0002] As an appearance inspection technique for industrial products, a technique for detecting unevenness on an inspection surface is known. Patent Document 1 discloses a technique in which the inspection surface is imaged by turning on each light source of a multi-lamp lighting system one by one, and unevenness defects are detected based on the multiple images obtained by imaging. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-191105 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the visual inspection of industrial products, there is a need to detect linear irregularities with a characteristic direction. For example, a scratch parallel to the outer frame of the inspection surface is an example of such linear irregularities. Among such linear irregularities, there is a need to detect fine irregularities in particular.

[0005] According to the technology described in Patent Document 1, light can be irradiated from various directions using multiple lighting. Therefore, if light is irradiated perpendicular to the direction of linear unevenness, even minute unevenness can be detected. For example, when detecting scratches parallel to the outer frame of the inspection surface, light is irradiated perpendicular to the outer frame of the inspection surface.

[0006] However, in industrial production lines, the posture of the inspection object may vary. For example, the rotation angle of the inspection surface around the vertical axis may vary. In a case where light is irradiated perpendicularly to the outer frame of the inspection surface to detect scratches parallel to the outer frame of the inspection surface, if the rotation angle of the inspection surface varies, the light will not hit the scratches perpendicularly. As a result, the shadows caused by the unevenness are weaker than when the light hits perpendicularly, making it difficult to detect fine unevenness. The present invention provides a technology that can detect fine linear unevenness even if the posture of the inspection object varies. [Means for solving the problem]

[0007] One aspect of the present invention is characterized by comprising a first acquisition means for acquiring a normal map indicating a spatial distribution of normals on a surface of an object, a second acquisition means for acquiring a direction vector of a virtual light source corresponding to a posture change of the object, and a detection means for performing a detection process of linear irregularities on the surface based on the normal map and the direction vector. Effect of the Invention

[0008] According to the present invention, even if the posture of the inspection object changes, minute linear irregularities can be detected. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration relating to uneven defect inspection. [Diagram 3] 11 is a flowchart showing an example of a linear concave-convex inspection process. [Figure 4] 10 is a flowchart showing details of the process in step S302. [Diagram 5] 1 is a diagram illustrating a change in the posture of an object to be inspected on the XY plane and the rotation of a virtual light source vector. [Figure 6] 10 is a flowchart showing details of the process in step S304. [Figure 7] 11 is a flowchart showing an example of a linear concave-convex inspection process. [Figure 8] 10 is a flowchart showing details of the process in step S702. [Figure 9] 10 is a flowchart showing details of the process in step S804. [Figure 10] 11 is a flowchart showing an example of a linear concave-convex inspection process. [Figure 11] 10 is a flowchart showing details of the process in step S1001. [Figure 12] 10 is a flowchart showing details of the process in step S1002. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] The image processing device according to this embodiment first acquires a normal map that represents normals at each position on an inspection surface, which is the surface of an object (inspection object) to be inspected. In this embodiment, photometric stereo is used as a method for acquiring the normal map of the inspection surface, but other methods such as a light section method may also be used, and the method for acquiring the normal map of the inspection surface is not limited to a specific method. In the process of acquiring the normal map, the image processing device generates a reflectance map that represents the spatial distribution of reflectance (albedo) on the inspection surface.

[0012] Next, the image processing device acquires, as a reference irradiation direction, a predetermined irradiation direction (a relative direction with respect to the inspection surface) as the irradiation direction of light to be irradiated onto the inspection surface in order to detect linear irregularities (linear irregularity defects) on the inspection surface.

[0013] The image processing device then obtains a posture change of the inspection object, and determines a direction such that the relative direction of the inspection object with respect to the inspection surface after the posture change becomes the reference irradiation direction as a virtual light source direction (direction of light irradiated from the virtual light source).

[0014] The image processing device then estimates the luminance distribution on the inspection surface when light is irradiated from the virtual light source in the virtual light source direction based on the normal map, reflectance map, and virtual light source direction, and performs detection processing for linear irregularities on the inspection surface based on the estimated luminance distribution.

[0015] This makes it possible to obtain the luminance distribution on the inspection surface when light is irradiated from a virtual light source whose relative direction to the inspection surface is the reference irradiation direction, regardless of the posture of the inspection object. For example, when performing an inspection to detect scratches parallel to the outer frame of the inspection surface as linear irregularities, it is possible to obtain the luminance distribution on the inspection surface when light is always irradiated in a direction perpendicular to the outer frame of the inspection surface, even if the posture of the inspection object changes.

[0016] If light is applied perpendicular to the direction of linear unevenness (e.g., a scratch), the shadow created by the unevenness will be strongest, making it possible to detect even the most minute unevenness. Therefore, according to this embodiment, the luminance distribution when light is applied perpendicular to the direction of the linear unevenness can be estimated regardless of the orientation of the test object, so that even if the orientation of the test object changes, minute linear unevenness can be detected.

[0017] First, a configuration example of a system according to this embodiment will be described with reference to Fig. 1. Fig. 1(a) shows a state in which a light source unit 102 and an image capturing unit 103 are arranged, as viewed from the side, and Fig. 1(b) shows the same state as viewed from above.

[0018] 1(a) and (b), the system according to this embodiment includes light sources 0 to 7 included in a light source unit 102, which is a dome-shaped multi-lamp lighting, an imaging unit 103, and an image processing device 101. The number of light sources used in the photometric stereo method is not limited to eight, and may be three or more. An inspection object 104 is an object to be inspected for linear irregularities by this system, and is, for example, an industrial product such as a home appliance or a cosmetic product.

[0019] The image processing device 101 sequentially lights up the light sources 0 to 7 one by one. The image processing device 101 controls the exposure, focus, shooting timing, etc. of the imaging unit 103, and causes the imaging unit 103 to capture an image of the inspection surface of the object 104 with one light source turned on. For example, the image processing device 101 turns on only the light source 0 and then causes the imaging unit 103 to capture an image of the object 104, and then turns on only the light source 1 and then causes the imaging unit 103 to capture an image of the object 104. In this manner, the image processing device 101 causes the imaging unit 103 to perform an operation of capturing an image of the object 104 with only one light source turned on for each of the light sources 0 to 7, thereby acquiring eight captured images of the object 104. The image processing device 101 generates a normal map according to the photometric stereo method based on the eight captured images, and performs inspection of linear irregularities on the inspection surface of the inspection object 104 based on the normal map and a reflectance map obtained in the process of generating the normal map.

[0020] Next, an example of the hardware configuration of the system according to this embodiment will be described with reference to the block diagram in Fig. 1(c). The light source unit 102 and the image capture unit 103 have been described above, so the remaining functional units will be described.

[0021] The CPU 106 executes various processes using computer programs and data stored in the RAM 105. As a result, the CPU 106 controls the operation of the entire system including the image processing device 101, and executes or controls various processes that will be described as processes performed by the system.

[0022] The RAM 105 has an area for storing computer programs and data loaded from the ROM 107, and an area for storing captured images acquired from the imaging unit 103. The RAM 105 further has a work area used when the CPU 106 executes various processes. In this way, the RAM 105 can provide various areas as needed.

[0023] The ROM 107 stores setting data for the image processing device 101, computer programs and data related to the startup of the image processing device 101, and computer programs and data related to the basic operation of the image processing device 101. Furthermore, the ROM 107 stores computer programs and data for causing the CPU 106 to execute or control various processes described as processes performed by the image processing device 101.

[0024] The interface 108 functions as a communication interface for performing data communication between the image processing apparatus 101 and an external device connected to an external bus 110. The CPU 106, the RAM 105, the ROM 107, and the interface 108 are all connected to a system bus 109.

[0025] The display 111 has a liquid crystal screen or a touch panel screen, and can display the results of processing by the CPU 106 as images, characters, etc. The display 111 may be a projection device such as a projector that projects images and characters.

[0026] The keyboard 113 and the mouse 112 are examples of a user interface, and when operated by the user, various information and instructions can be input to the image processing device 101. In addition to these, for example, a touch panel screen can also be applied to the user interface.

[0027] <Normal map generation using photometric stereo> Next, a method for the image processing device 101 to generate a normal map of the inspection surface of the inspection object 104 according to photometric stereo based on the eight captured images captured by the imaging unit 103 will be described.

[0028] Photometric stereo assumes Lambert's cosine law and assumes that the reflected luminance of the inspection surface is proportional to the dot product of the directional vector indicating the direction of the light source and the normal vector. In other words, it is assumed that the following equation (1) holds true at each position (x, y) on the inspection surface.

[0029]

number

[0030] In formula (1), Ii is the reflected luminance at position (x, y) when only light source i (i=0 to 7) is turned on, and can be obtained from a captured image. Li=(Lix, Liy, Liz) is a unit vector indicating the direction of light source i at position (x, y), and will be referred to as the light source vector hereinafter. FIG. 1(a) illustrates an example of a light source vector indicating the direction of light source 4.

[0031] Nρ = (ρNx, ρNy, ρNz) T is the normal vector N=(Nx, Ny, Nz) at position (x, y). T The pseudo-normal vector is obtained by multiplying the proportionality constant ρ by the normal vector N. Note that T indicates the transpose of the vector. The normal vector N is a unit vector, and the proportionality constant ρ indicates the reflectance (albedo) at the position (x, y). Figure 1(a) shows an example of a normal vector.

[0032] Equation (1) holds for light source 0 to light source 7, and can be compiled to give the following equation (2). Furthermore, if the vector on the left side is written as I and the 8-row, 3-column matrix on the right side is written as L, then we get the following equation (3). Hereinafter, I will be called the luminance vector, and L will be called the light source matrix.

[0033]

number

[0034] In equation (3), the luminance vector I can be calculated from eight captured images, and the light source matrix L can be calculated from prior measurements. Equation (3) can be said to be a simultaneous equation with the three components of the pseudo-normal vector Nρ as unknowns. If there are three light sources, the light source matrix L will be a square matrix with 3 rows and 3 columns. If L is a regular matrix, the inverse matrix L of L will be -1 Applying to the luminance vector I gives the solution to equation (3).

[0035] On the other hand, when there are four or more light sources as in this embodiment, equation (3) becomes an overdetermined simultaneous equation. In this case, as shown in equation (4), the pseudo-inverse matrix (L T L) -1 L T By applying to the luminance vector I, the least squares solution of equation (3) is found.

[0036]

number

[0037] In this case, reflectance (albedo) ρ corresponds to the length of the pseudo-normal vector Nρ, so it can be calculated by formula (5). The normal vector N is a unit vector obtained by normalizing the length of the pseudo-normal vector Nρ to 1, so it can be calculated by formula (6).

[0038] By performing the calculations of the above-described formulas (4) to (6) for each position (x, y), a normal map showing the spatial distribution of normal vectors on the inspection surface can be generated. In addition, in this process, a reflectance map showing the spatial distribution of reflectance (albedo) on the inspection surface can also be generated using formula (5).

[0039] <Inspection of uneven defects using normal maps and virtual light sources> Next, an example of a functional configuration for unevenness defect inspection, which is an example of inspection related to linear unevenness, in the system according to this embodiment will be described with reference to the block diagram of Fig. 2. The acquisition unit 202 acquires surface characteristics of the inspection object 104 as described above by photometric stereo based on eight captured images captured by the imaging unit 103. The surface characteristics are a normal map and a reflectance map of the inspection surface of the inspection object 104.

[0040] The acquiring unit 203 acquires the inclination of the inspection object 104 (the rotation angle of the inspection surface of the inspection object 104 about the vertical axis of the inspection surface of the inspection object 104) as the posture change of the inspection object 104 based on the surface characteristics acquired by the acquiring unit 202. Note that in this embodiment, the rotation angle in the tilt direction is not acquired.

[0041] The setting unit 204 acquires a vector of a virtual light source direction that assumes that light is emitted from a direction that maximizes the shadow (visibility) of linear concaves and convexes having a characteristic direction. The vector acquired by the setting unit 204 corresponds to the light source vector Li in formula (1) and is a unit vector that indicates the virtual light source direction. Hereinafter, this unit vector is called a virtual light source vector.

[0042] In this embodiment, a case will be described in which a scratch that appears along the outer frame of the inspection surface of the inspection object 104 is targeted as a linear unevenness having a characteristic direction. At this time, the setting unit 204 acquires a virtual light source vector (a set of virtual light source vectors) having a direction perpendicular to the outer frame of the inspection surface. For example, when the inspection surface is a rectangle, the setting unit 204 acquires four virtual light source vectors.

[0043] The characteristic linear unevenness that may occur differs depending on the type (product type) of the inspection object 104. Therefore, the setting unit 204 may hold a "set of virtual light source vectors" corresponding to the type (product type) of the inspection object 104. In this case, when the type (product type) of the inspection object 104 is specified, the setting unit 204 acquires a "set of virtual light source vectors" corresponding to the specified type.

[0044] Correction unit 205 obtains a vector (variation direction vector) obtained by varying (correcting) the virtual light source vector in accordance with the posture variation acquired by acquisition unit 203, as a virtual light source vector to be used in subsequent processing. For example, when inspecting scratches that appear along the outer frame of the inspection surface, a virtual light source vector that is perpendicular to the outer frame of the inspection surface is obtained. Note that when multiple virtual light source vectors (a set of virtual light source vectors) are input, correction is performed on all of the virtual light source vectors.

[0045] The estimation unit 206 uses the normal map and the virtual light source vector to estimate the luminance distribution on the inspection surface when light from the virtual light source is irradiated, assuming Lambert's cosine law. More specifically, the estimation unit 206 calculates the inner product of the virtual light source vector corrected by the correction unit 205 and the normal vector corresponding to the pixel position in the normal map for each pixel position in the normal map. As a result, the estimation unit 206 acquires (estimates) the inner product value corresponding to each pixel position in the normal map as the luminance distribution of the inspection surface. Note that reflectance (albedo) may be taken into consideration when estimating the luminance distribution. Hereinafter, the luminance distribution acquired by the estimation unit 206 is referred to as a lighting image.

[0046] The inspection unit 207 detects linear irregularities on the inspection surface based on the lighting image. For example, when there is a pixel in the lighting image having a pixel value exceeding a threshold, the inspection unit 207 determines that linear irregularities have occurred on the inspection surface. On the other hand, when there is no pixel in the lighting image having a pixel value exceeding the threshold, the inspection unit 207 determines that no linear irregularities have occurred on the inspection surface.

[0047] The process for detecting linear irregularities on the inspection surface based on the lighting image is not limited to the above threshold process. For example, the inspection unit 207 may input the lighting image to a machine learning model that has been trained to detect linear irregularities and perform calculations on the machine learning model to determine the presence or absence of linear irregularities in the lighting image or classify the direction of the linear irregularities.

[0048] Furthermore, the inspection unit 207 may obtain a difference image between the lighting image and a reference image, which is a lighting image of an inspection surface having no linear irregularities, and detect linear irregularities based on the difference image.

[0049] Next, an example of the linear unevenness inspection process performed by the image processing device 101 will be described with reference to the flowchart in Fig. 3. In step S301, the acquisition unit 202 acquires surface characteristics (the normal map N and the reflectance map ρ of the inspection surface of the inspection object 104) as described above.

[0050] In step S302, the acquisition unit 203 calculates a posture variation θ in the XY plane as a posture variation of the inspection object 104 based on the surface characteristics acquired in step S301. In this embodiment, the inspection surface is the XY plane, and the axis (height direction axis) perpendicular to the XY plane is the Z axis. That is, as shown in FIG. 5(a), the lower left corner of the inspection surface 501 of the inspection object 104 is set as the origin, two axes perpendicular to the origin are the X axis and the Y axis, and an axis perpendicular to the X axis and the Y axis is set as the Z axis. In this embodiment, the inspection object 104 does not tilt in the height direction, i.e., in the Z axis direction, but tilts only in the XY plane. This tilt in the XY plane is referred to here as the posture variation θ in the XY plane. This posture variation θ corresponds to a rotation angle. A method for calculating the posture variation θ in the XY plane will be described later.

[0051] In step S303, setting unit 204 acquires virtual light source vector L. In step S304, correction unit 205 obtains, as virtual light source vector L', a vector obtained by varying (correcting) virtual light source vector L acquired in step S303 in accordance with posture variation θ acquired in step S302. Details of the process in step S304 will be described later.

[0052] In step S305, the estimation unit 206 generates a lighting image R according to the following formula, using the surface characteristics (normal map N and reflectance map ρ) acquired in step S301 and the virtual light source vector L′ determined in step S304.

[0053]

number

[0054] Here, ρ(x,y) indicates the reflectance at pixel position (x,y), and N(x,y) indicates the normal vector at pixel position (x,y). Note that ρ(x,y) may be a constant value (ρ) regardless of x and y. Also, the reflectance map ρ may be omitted from this formula. In step S306, the inspection unit 207 detects linear unevenness defects on the inspection surface based on the lighting image R generated in step S305.

[0055] <How to calculate posture variation θ> Details of the process in step S302 (processing for calculating the posture variation θ in the XY plane) will be described with reference to the flowchart in Fig. 4. In step S401, the acquisition unit 203 detects a reference line L_target of the inspection object 104 from the reflectance map ρ included in the surface characteristics acquired in step S301. In this embodiment, the horizontal contour line of the outer frame of the inspection surface of the inspection object 104 is set as the reference line L_target. In the example of Fig. 5(a), if the lower right corner of the inspection surface 501 is set as A, the line segment OA is set as the reference line L_target. In this embodiment, the acquisition unit 206 binarizes the values ​​of each element in the reflectance map ρ, and then acquires the line segment OA obtained by performing edge detection processing, which is processing for detecting edges from the reflectance map ρ, as the reference line L_target.

[0056] Alternatively, a method may be adopted in which an image represented by the reflectance map ρ is divided into a plurality of regions using a clustering technique, and the outer frame of the inspection surface is detected from the boundary lines of the regions. Alternatively, a method may be adopted in which feature points are extracted from the reflectance map ρ, and the outer frame of the inspection surface is detected from the feature points. There are various possible methods for acquiring the contour line of the inspection surface in order to obtain the reference line L_target, and the method is not limited to a specific method.

[0057] In step S402, the acquisition unit 203 acquires a reference line L_original when the virtual light source vector is set. This corresponds to a reference coordinate axis of the coordinate system in the inspection system, and is the X-axis in FIG. 5(a), that is, the line segment OX.

[0058] In step S403, the acquisition unit 203 obtains the angle (less than 180 degrees) between the reference line L_target and the reference line L_original as the posture variation θ. The acquisition unit 203 obtains the angle between the reference line L_target and the reference line L_original, for example, as follows. First, the acquisition unit 203 defines a right triangle in which one of the two reference lines is one side of the right triangle excluding the hypotenuse, and the other of the two reference lines is the other side of the right triangle excluding the hypotenuse. Then, the acquisition unit 206 obtains tan -1 The calculation result of (height of right triangle / length of base of right triangle) is obtained as the posture variation θ. In the example of Fig. 5(a), if the position of the lower right corner A is (Xa, Ya), the acquisition unit 206 calculates the posture variation θ by the following formula.

[0059]

number

[0060] The method for determining the orientation variation is not limited to the method based on the surface characteristics. For example, an orientation sensor may be attached to the inspection object 104, and the acquisition unit 206 may acquire the orientation variation of the inspection surface in the XY plane measured by the orientation sensor as the orientation variation θ.

[0061] <Virtual light source vector correction> Details of the process in step S304 above will be described with reference to the flowchart in Fig. 6. In step S601, correction unit 205 acquires the posture variation θ calculated in step S302.

[0062] In step S602, correction unit 205 acquires virtual light source vector L acquired in step S303. For example, virtual light source vector L of a virtual light source that irradiates light perpendicularly to line segment OA in Fig. 5(a) is virtual light source vector 502 in Fig. 5(b).

[0063] In step S603, the correction unit 205 obtains, as a virtual light source vector L', a vector obtained by rotating the virtual light source vector L acquired in step S602 around the origin in the XY plane by the posture variation θ calculated in step S601.

[0064] For example, assume that virtual light source vector L is virtual light source vector 502 in Fig. 5(b). In this case, virtual light source vector 503 obtained by rotating virtual light source vector 502 around origin O by posture variation θ becomes virtual light source vector L'. More specifically, assuming that virtual light source vector L = (Lx, Ly), the virtual light source vector L' = (L'x, L'y) after rotation is calculated by the following formula.

[0065]

number

[0066] Here, a single virtual light source vector perpendicular to the line segment OA in Fig. 5(a) has been described. However, when multiple virtual light source vectors are acquired, the same process may be performed on each virtual light source vector to obtain a virtual light source vector rotated according to the posture variation θ.

[0067] For example, if the upper left corner of inspection surface 501 is designated as B, then the virtual light source vector of a virtual light source that irradiates light perpendicular to line segment OB in Figure 5(a) may also be similarly calculated as a virtual light source vector rotated in accordance with posture variation θ.

[0068] In this way, according to this embodiment, it is possible to estimate the lighting image when light is irradiated from the intended direction regardless of changes in the posture of the object to be inspected, thereby making it possible to detect fine linear irregularities even if the posture of the object to be inspected changes.

[0069] [Second embodiment] In each of the following embodiments including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the embodiments are the same as the first embodiment. In the first embodiment, the virtual light source vector is rotated in response to the change in posture of the object to be inspected, so that the luminance distribution in response to the light from the virtual light source always coming from the same direction is acquired regardless of the change in posture of the object to be inspected.

[0070] In this embodiment, when the setting unit 204 acquires a plurality of virtual light source vectors, the virtual light source vectors are corrected more simply. For example, when the corners of the outer frame of the inspection surface are rounded as shown in Fig. 5(c), the setting unit 204 acquires three virtual light source vectors 504, 505, and 506 shown in Fig. 5(c) in order to detect uneven defects that appear along the round corners. In the following explanation, in order to explain by giving a concrete example, a case will be explained in which the setting unit 204 acquires the virtual light source vectors 504, 505, and 506.

[0071] An example of a linear unevenness inspection process performed by the image processing device 101 will be described with reference to the flowchart of Fig. 7. In Fig. 7, the same process steps as those shown in Fig. 3 are given the same step numbers, and descriptions of those process steps will be omitted.

[0072] In step S701, the setting unit 204 acquires a virtual light source vector group Lg, which is a collection of virtual light source vectors (here, virtual light source vectors 504, 505, and 506 shown in FIG. 5(c)).

[0073] In step S702, correction unit 205 corrects virtual light source vector group Lg. Details of the process in step S702 will be described with reference to the flowchart in Fig. 8. In Fig. 8, the same process steps as those shown in Fig. 6 are assigned the same step numbers, and descriptions of those process steps will be omitted.

[0074] In step S801, correction unit 205 acquires information on virtual light source vector group Lg. The information on virtual light source vector group Lg may include the number of virtual light source vectors included in the virtual light source vector group Lg, the angle between adjacent virtual light source vectors, and the like.

[0075] In step S802, correction unit 205 obtains the number A of virtual light source vectors (the number of vectors) included in the virtual light source vector group Lg from the information obtained in step S801.

[0076] In step S803, the correction unit 205 determines whether the posture change θ is less than the threshold value (less than Th1). If the posture change θ is less than the threshold value Th1 as a result of this determination, the process proceeds to step S804, and if the posture change θ is equal to or greater than the threshold value (equal to or greater than Th1), the process proceeds to step S805. In this embodiment, Th1=10°.

[0077] In step S804, correction unit 205 obtains a virtual light source vector group Lg' by correcting the virtual light source vector group Lg using a simple calculation method. Details of the process in step S804 will be described later.

[0078] In step S805, correction unit 205 initializes the value of counter P to 0. Then, in step S806, correction unit 205 selects (acquires) one of the virtual light source vectors included in virtual light source vector group Lg that has not been selected, as selected virtual light source vector L. For example, when the virtual light source vectors included in virtual light source vector group Lg are arranged in an appropriate order, correction unit 205 selects the virtual light source vector corresponding to the Pth virtual light source vector as selected virtual light source vector L.

[0079] Then, in step S304, correction unit 205, in the same manner as in the first embodiment, obtains, as virtual light source vector L', a vector obtained by varying (correcting) the selected virtual light source vector L selected in step S806 in accordance with the posture variation θ acquired in step S302.

[0080] In step S807, the correction unit 205 increments the value of the counter P by one. Then, in step S808, the correction unit 205 determines whether the value of the counter P is the same as A. If the result of this determination is that the value of the counter P is the same as A, the process proceeds to step S305. On the other hand, if the value of the counter P is not the same as A, the process proceeds to step S806.

[0081] Next, details of the process in step S804 above will be described with reference to the flowchart in Fig. 9. In Fig. 9, the same process steps as those in Fig. 3 and Fig. 8 are given the same step numbers, and descriptions of those process steps will be omitted.

[0082] In step S901, correction unit 205 acquires, from virtual light source vector group Lg, virtual light source vector Lnext adjacent to selected virtual light source vector L in the direction of rotation angle θ. For example, when virtual light source vector 505 is selected as selected virtual light source vector L in FIG. 5C, virtual light source vector 504 adjacent to virtual light source vector 505 in the direction of rotation angle θ, i.e., the counterclockwise direction, is acquired as virtual light source vector Lnext.

[0083] In step S902, correction unit 205 obtains angle φ between selected virtual light source vector L and virtual light source vector Lnext from the information obtained in step S801. Note that angle φ between selected virtual light source vector L and virtual light source vector Lnext may be calculated by correction unit 205.

[0084] In step S903, the correction unit 205 judges whether the angle φ acquired in step S902 is less than the threshold value Th2. If the result of this judgment is that the angle φ acquired in step S902 is less than the threshold value Th2, the process proceeds to step S904. On the other hand, if the angle φ acquired in step S902 is equal to or greater than the threshold value Th2, the process proceeds to step S304. In this embodiment, Th2=10°.

[0085] In step S904, correction unit 205 performs the following interpolation calculation using selected virtual light source vector L, virtual light source vector Lnext, posture variation θ, and angle φ, to obtain a corrected virtual light source vector L′.

[0086]

number

[0087] Note that various methods of interpolation using these pieces of information are possible, and the method is not limited to a specific interpolation method. In addition, in the process according to the flowchart in FIG. 9, there is no virtual light source vector Lnext adjacent to the (A-1)th selected selected virtual light source vector in the posture variation θ direction. However, when the (A-1)th selected virtual light source vector is selected in step S806, the process proceeds to step S304, and correction unit 205 may obtain a vector obtained by varying (correcting) the (A-1)th selected virtual light source vector in accordance with the posture variation θ as virtual light source vector L'. In addition, for example, when correction unit 205 performs the process of step S904 on the (A-2)th selected virtual light source vector, it may output the (A-1)th selected virtual light source vector as the corrected virtual light source vector L'.

[0088] In the simple calculation of the virtual light source vector L' in step S904, a line segment is assumed that connects the tips of the virtual light source vector L and the virtual light source vector Lnext with a straight line. Then, a vector toward a point obtained by dividing the line segment by (1-θ / φ):(θ / φ) is obtained by linear interpolation. If θ is sufficiently small, the magnitude of the virtual light source vector L' obtained by linear interpolation can be regarded as being substantially the same as the virtual light source vector L before correction. For this purpose, the condition is that the angle θ of the posture change and the angle φ formed by the two virtual light source vectors used in the linear interpolation are smaller than thresholds Th1 and Th2. In this embodiment, the above-mentioned linear interpolation formula is used only when the posture change θ and the angle φ are sufficiently small, but the condition may be relaxed by using a spherical linear interpolation formula.

[0089] [Third embodiment] In the first and second embodiments, a case has been described in which the inspection object is not tilted with respect to the Z axis, but is tilted in the XY plane. On the other hand, the inspection object may be tilted with respect to the Z axis due to vibration or the like. In this case, the virtual light source vector is rotated around the Z axis in the XY plane, and then rotated around the X axis (or Y axis) in the YZ plane (or XZ plane). Below, a case in which the inspection object is tilted with respect to the Z axis will be described.

[0090] An example of a linear unevenness inspection process performed by the image processing device 101 will be described with reference to the flowchart of Fig. 10. In Fig. 10, the same process steps as those shown in Fig. 3 are given the same step numbers, and descriptions of those process steps will be omitted.

[0091] In step S1001, acquisition unit 203 calculates posture variation ω of inspection object 104 in the YZ plane. Details of the processing in step S1001 will be described later. In step S1002, correction unit 205 uses the posture variation θ acquired in step S302 and the posture variation ω acquired in step S1001 to obtain a vector obtained by varying (correcting) the virtual light source vector L acquired in step S303 as a virtual light source vector L". Details of the processing in step S1002 will be described later.

[0092] The processing from step S305 onwards is the same as in the first embodiment, except that the virtual light source vector L" is used instead of the virtual light source vector L'. Next, details of the processing in step S1001 above will be described with reference to the flowchart in FIG.

[0093] In step S1101, the acquisition unit 203 specifies a reference plane area F of the inspection object 104 from the reflectance map ρ. In this embodiment, a flat area parallel to the XY plane in the image of the inspection object 104 in the XYZ axes on which the virtual light source vector L is set is set in advance as the reference plane area F. The reference plane area F is defined by a relative position with respect to the reference line of the inspection object 104. In this embodiment, the line segments OA and OB corresponding to the outer frame of the inspection surface 501 shown in FIG. 5(a) are set as the reference lines. Then, in the coordinate system in which the lengths of the line segments OA and OB are normalized to 1, the area surrounded by the four points (0.85, 0.85), (0.85, 0.70), and (0.70, 0.70) is defined as the reference plane area F. That is, the area 507 in FIG. 5(a) is set in advance as the reference plane area F in this embodiment.

[0094] Next, in step S1102, the acquisition unit 203 identifies an area in the normal map N that corresponds to the reference plane area F, and acquires the most frequent normal vector (mode vector) in the YZ plane of the normal vectors in the identified area. Since the normal map N is prone to noise, the most frequent normal vector is identified as the normal vector of the object 104 with respect to the XY plane.

[0095] In step S1103, the acquiring unit 203 calculates a posture variation ω in the YZ plane from the most frequent normal vector acquired in step S1102. For example, the acquiring unit 203 projects the most frequent normal vector onto the YZ plane, and determines the angle of the projected vector with respect to the Z axis as the posture variation ω. Note that the method of acquiring the posture variation ω is not limited to a specific method, and for example, an orientation sensor may be attached to the inspection object 104, and the acquiring unit 206 may acquire the variation in the posture of the inspection object 104 in the YZ plane measured by the orientation sensor as the posture variation ω.

[0096] Next, details of the process in step S1002 above will be described with reference to the flowchart in Fig. 12. In Fig. 12, the same process steps as those shown in Fig. 6 are given the same step numbers, and descriptions of those process steps will be omitted.

[0097] In step S1201, correction unit 205 acquires the posture variation ω calculated in step S1001. In step S1202, correction unit 205 obtains a vector obtained by rotating virtual light source vector L' corrected in step S603 around the origin in the YZ plane by the posture variation ω acquired in step S1201 as virtual light source vector L" after three-axis correction. More specifically, if virtual light source vector L'=(L'x, L'y, L'z), virtual light source vector L"=(L"x, L"y, L"z) after three-axis correction is calculated by the following formula.

[0098]

number

[0099] In this way, in any of the above embodiments, a lighting image can be created by setting a virtual light source that always irradiates the object with light while maintaining a constant angle, regardless of the posture of the object being inspected. Therefore, even if the posture of the object being inspected changes, the detection of linear irregularities on the surface of the object being inspected can always be performed under constant conditions, and fine linear irregularities can be detected.

[0100] In the above embodiments, for simplicity, the lower left corner of the object is used as the origin, but the present invention is not limited to this. That is, the above processes may be performed with the upper right corner or the center of the object as the origin.

[0101] In addition, the origin may not coincide with the outer frame of the object to be inspected, but may be shifted in each axial direction from the outer frame. It goes without saying that, depending on the position of the origin, the formulas used in the explanation of each embodiment above must be transformed to fit the coordinate system to be used.

[0102] Further, in each of the above embodiments, no reference is made to outputting the results of the inspection by the inspection unit 207, but the output form of the results of the inspection by the inspection unit 207 is not limited to a specific output form.

[0103] The numerical values, processing timing, processing order, processing subject, color space, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0104] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0105] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0106] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) A first acquisition means for acquiring a normal map indicating a spatial distribution of normals on a surface of an object; a second acquisition means for acquiring a direction vector of a virtual light source according to a change in posture of the object; a detection means for detecting linear irregularities on the surface based on the normal map and the direction vector; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the first acquisition means acquires the normal map based on a plurality of captured images obtained by sequentially lighting a plurality of light sources and capturing an image of an object. (Item 3) The method further includes a third acquisition means for acquiring a luminance distribution on the surface based on the normal map and the direction vector, 2. The image processing device according to item 1, wherein the detection means performs the detection process based on the luminance distribution. (Item 4) 4. The image processing device according to item 3, wherein the first acquisition means acquires a reflectance map indicating a spatial distribution of reflectance on the surface based on a plurality of captured images obtained by sequentially lighting each of a plurality of light sources and capturing an image of the object. (Item 5) 5. The image processing device according to item 4, wherein the third acquisition means acquires a luminance distribution on the surface based on the normal map, the direction vector, and the reflectance map. (Item 6) The second acquisition means changes a direction vector of the virtual light source in accordance with the posture change when the posture change is equal to or greater than a threshold value. 6. The image processing device according to any one of items 1 to 5, (Item 7) The second acquisition means, when the posture variation is less than a threshold and an angle formed between a direction vector of a first virtual light source and a direction vector of a second virtual light source adjacent to the direction vector of the first virtual light source is less than a threshold, acquires the direction vector of the virtual light source based on the direction vector of the first virtual light source and the direction vector of the second virtual light source. 7. The image processing device according to any one of items 1 to 6, (Item 8) The second obtaining means varies the direction vector of the first virtual light source in response to the posture variation when the posture variation is less than a threshold and an angle formed between a direction vector of a first virtual light source and a direction vector of a second virtual light source adjacent to the first virtual light source is equal to or greater than a threshold. 8. The image processing device according to any one of items 1 to 7, (Item 9) The image processing device according to any one of items 1 to 8, characterized in that the second acquisition means varies a vector of a preset irradiation direction as a direction of light to be irradiated onto the surface in order to detect linear irregularities on the surface, in accordance with the posture variation. (Item 10) 10. The image processing device according to any one of items 1 to 9, wherein the attitude change is a rotation angle of the object in one plane. (Item 11) 10. The image processing device according to any one of items 1 to 9, wherein the posture change includes a rotation angle of the object in a first plane and a rotation angle of the object in a second plane. (Item 12) A first acquisition step of acquiring a normal map indicating a spatial distribution of normals on a surface of an object; a second acquisition step of acquiring a direction vector of a virtual light source according to a posture change of the object; a detection step of detecting linear irregularities on the surface based on the normal map and the direction vector; 13. An image processing method comprising: (Item 13) 12. A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0107] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0108] 202: Acquisition unit 203: Acquisition unit 204: Setting unit 205: Correction unit 206: Estimation unit 207: Inspection unit

Claims

1. A first acquisition means for acquiring a normal map indicating a spatial distribution of normals on a surface of an object; a second acquisition means for acquiring a direction vector of a virtual light source according to a posture change of the object; a detection means for detecting linear irregularities on the surface based on the normal map and the direction vector; An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the first acquisition means acquires the normal map based on a plurality of captured images obtained by sequentially lighting a plurality of light sources and capturing an image of an object.

3. The method further includes a third acquisition means for acquiring a luminance distribution on the surface based on the normal map and the direction vector, 2. The image processing apparatus according to claim 1, wherein the detection means performs the detection process based on the luminance distribution.

4. 4. The image processing device according to claim 3, wherein the first acquisition means acquires a reflectance map indicating a spatial distribution of reflectance on the surface based on a plurality of captured images obtained by sequentially lighting each of a plurality of light sources and capturing an image of the object.

5. The image processing apparatus according to claim 4 , wherein the third acquisition means acquires a luminance distribution on the surface based on the normal map, the direction vector, and the reflectance map.

6. The second acquisition means changes a direction vector of the virtual light source in accordance with the posture change when the posture change is equal to or greater than a threshold value.

2. The image processing device according to claim 1,

7. The second acquisition means, when the posture variation is less than a threshold and an angle formed between a direction vector of a first virtual light source and a direction vector of a second virtual light source adjacent to the direction vector of the first virtual light source is less than a threshold, acquires the direction vector of the virtual light source based on the direction vector of the first virtual light source and the direction vector of the second virtual light source.

2. The image processing device according to claim 1,

8. The second obtaining means varies the direction vector of the first virtual light source in response to the posture variation when the posture variation is less than a threshold and an angle formed between a direction vector of a first virtual light source and a direction vector of a second virtual light source adjacent to the first virtual light source is equal to or greater than a threshold.

2. The image processing device according to claim 1,

9. The image processing device according to claim 1, characterized in that the second acquisition means varies a vector of a predetermined irradiation direction as a direction of light to be irradiated onto the surface in order to detect linear irregularities on the surface, in accordance with the posture variation.

10. 2. The image processing apparatus according to claim 1, wherein the attitude change is a rotation angle of the object in one plane.

11. The image processing apparatus according to claim 1 , wherein the posture change includes a rotation angle of the object in a first plane and a rotation angle of the object in a second plane.

12. A first acquisition step of acquiring a normal map indicating a spatial distribution of normals on a surface of an object; a second acquisition step of acquiring a direction vector of a virtual light source according to a posture change of the object; a detection step of detecting linear irregularities on the surface based on the normal map and the direction vector; 13. An image processing method comprising:

13. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images