Estimation system, estimation device, and estimation method

Abstract

To lighten a burden of a person carrying out inspection of surface abnormality of the object being inspected.SOLUTION: One embodiment of the present invention is an estimation system comprising a radiation device equipped with a plurality of light sources arranged in a row on a plane; an imaging device equipped with a light receiving element, for receiving at least some of a result of light radiated by the radiation device and then reflected or scattered by the object being inspected; and a control unit for estimating surface roughness of the object being inspected, on the basis of the light reception result of the light receiving element and bearing information indicating the bearing of each section of the object being inspected. The light received by the light receiving element is some of the radiated light of the radiation device having been mirror reflected, diffusion reflected, or subsurface scattered by the object being inspected, the polarization direction of which is same as the polarization direction of the light radiated by the radiation device. The distribution of luminance in a plane where the light sources lie in a row satisfies a pattern condition that there is no change of luminance in one direction in the plane, but there is a change of luminance in other directions perpendicular to the one direction, and that luminance in the other directions is represented by a sine wave in which positions in the other directions are assumed to be independent variables.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an estimation system, an estimation device, and an estimation method. [Background technology]

[0002] Inspecting the surface of an inspection object for defects such as scratches and foreign matter is important from the standpoint of ensuring the quality of products to be shipped. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-139822 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, this type of inspection is difficult for those without specialized knowledge, which can lead to a shortage of inspectors, which can result in inspectors having to work longer hours to make up for the shortage.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for reducing the burden on a person who inspects the surface of an inspection object for abnormalities. [Means for solving the problem]

[0006] One aspect of the present invention is an estimation system comprising: an illumination device having a plurality of light sources arranged on a surface; an imaging device having a plurality of light receiving elements arranged on a surface, the imaging device receiving at least a portion of the result of the light irradiated by the illumination device being reflected or scattered by an object to be inspected; and a control unit estimating a surface roughness of the object to be inspected based on the result of the light reception by the light receiving elements and orientation information indicating the orientation of each part of the object to be inspected, the light source emitting light in a predetermined polarization direction, the polarization direction of the light emitted by the light source being the same regardless of the light source, the light received by the light receiving elements being light that is specularly reflected, diffusely reflected, or subsurface scattered by the object to be inspected, the polarization direction of the light being the same as the polarization direction of the light emitted by the illumination device, and the distribution of luminance within the surface on which the light sources are arranged is an estimation system that satisfies the following pattern conditions: there is no change in luminance in one direction within the surface, but there is a change in luminance in the other direction that is a direction perpendicular to the one direction, and further, the luminance in the other direction is expressed as a sine wave with the position in the other direction as an independent variable.

[0007] One aspect of the present invention is an estimation device that includes an imaging device that receives at least a portion of the light irradiated by an illumination device having a plurality of light sources arranged on a surface and reflected or scattered by an object to be inspected, and a control unit that estimates the surface roughness of the object to be inspected based on the light reception result by the imaging device having a plurality of light receiving elements arranged on a surface and orientation information indicating the orientation of each part of the object to be inspected, wherein the light source emits light in a predetermined polarization direction, the polarization direction of the light emitted by the light source is the same regardless of the light source, the light received by the light receiving element is light that is specularly reflected, diffusely reflected, or subsurface scattered by the object to be inspected, and has the same polarization direction as the polarization direction of the light emitted by the illumination device, and the distribution of luminance within the surface on which the light sources are arranged does not change in one direction within the surface, but changes in luminance in the other direction that is perpendicular to the one direction, and further, the luminance in the other direction is expressed as a sine wave with the position in the other direction as an independent variable.

[0008] One aspect of the present invention includes a control step of estimating the surface roughness of an object to be inspected based on a result of light reception by an imaging device having a plurality of light receiving elements arranged on a surface, the imaging device receiving at least a part of the result of light irradiated by an irradiation device having a plurality of light sources arranged on a surface and reflected or scattered by an object to be inspected, and orientation information indicating the orientation of each part of the object to be inspected, the light source emitting light in a predetermined polarization direction, the polarization direction of the light radiated by the light source being the same regardless of the light source, and the light received by the light receiving elements being light that is specularly reflected, diffusely reflected or scattered subsurface by the object to be inspected, the polarization direction of the light being the same as the polarization direction of the light radiated by the irradiation device, This is an estimation method that satisfies a pattern condition that the distribution of luminance within the plane on which the light sources are arranged does not change in one direction within the plane, but changes in luminance in another direction that is perpendicular to the one direction, and further, the luminance in the other direction is expressed as a sine wave with the position in the other direction as an independent variable. Effect of the Invention

[0009] The present invention makes it possible to reduce the burden on a person who inspects the surface of an inspection object for abnormalities. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating an estimation system according to an embodiment. [Diagram 2] 6A and 6B are diagrams showing an example of a luminance distribution that satisfies a pattern condition in the embodiment. [Diagram 3] FIG. 4 is a first explanatory diagram for explaining estimation of surface roughness in the embodiment. [Figure 4] FIG. 2 is a second explanatory diagram illustrating the estimation of surface roughness in the embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation device according to an embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a processing flow executed by the estimation device according to the embodiment. [Figure 7]6A and 6B are diagrams showing examples of color patterns displayed on a display for obtaining orientation information in the embodiment. [Figure 8] FIG. 1 is a first explanatory diagram for explaining the experimental situation in the embodiment. [Figure 9] FIG. 2 is a diagram showing an image of one of two types of inspection targets used in the embodiment. [Figure 10] FIG. 13 is a diagram showing an image of the other of the two types of inspection targets used in the embodiment. [Figure 11] FIG. 1 is a first diagram showing an example of an experimental result in the embodiment. [Figure 12] FIG. 2 is a second diagram showing an example of an experimental result in the embodiment. [Figure 13] FIG. 2 is a second explanatory diagram for explaining the experimental situation in the embodiment. [Figure 14] FIG. 3 is a third diagram showing an example of an experimental result in the embodiment. [Figure 15] FIG. 4 is a fourth diagram showing an example of an experimental result in the embodiment. [Figure 16] FIG. 5 is a fifth diagram showing an example of an experimental result in the embodiment. [Figure 17] FIG. 6 is a sixth diagram showing an example of an experimental result in the embodiment. [Figure 18] FIG. 7 is a seventh diagram showing an example of an experimental result in the embodiment. [Figure 19] FIG. 8 is an eighth diagram showing an example of an experimental result in the embodiment. [Figure 20] FIG. 9 is a ninth diagram showing an example of an experimental result in the embodiment. [Figure 21] FIG. 10 is a tenth diagram showing an example of an experimental result in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] FIG. 1 is an explanatory diagram illustrating an estimation system 100 in an embodiment. The estimation system 100 is a system that supports inspection of an abnormality on the surface of an inspection object. Specifically, the estimation system 100 estimates the surface roughness of the inspection object. Since the surface roughness is estimated by the estimation system 100, the burden of inspection on a person who inspects the surface of the inspection object for abnormalities is reduced. In this way, the estimation system 100 estimates the surface roughness of the estimation object by taking the inspection object as an estimation object.

[0012] The estimation system 100 irradiates the inspection object 9 with light, and estimates the surface roughness of the inspection object 9 based on the light reflected or scattered by the inspection object 9. Specifically, the reflected or scattered light is specular reflection, diffuse reflection, or subsurface scattering light.

[0013] The estimation system 100 includes an irradiation device 1, an imaging device 2, and an estimation device 3. The irradiation device 1 includes a plurality of light sources arranged on a surface. The light sources emit light in a predetermined polarization direction. The polarization direction of the light emitted by the light sources is the same regardless of the light source. The irradiation device 1 is, for example, a display having a polarizing plate on the surface of the display. In such a case, the light emitted from the display is converted into light in a predetermined polarization direction and propagates in the emitted direction. In such a case, the light source is a combination of each pixel of the display and a part of the polarizing plate that is located on the surface of each pixel. The irradiation device 1 may be, for example, a display in which each pixel is a combination of an LED (Light Emitting Diode) and a polarizing element. In such a case, since the display itself emits polarized light, it is not necessary to provide a polarizing plate on the surface of the display. The display may be, for example, a liquid crystal display.

[0014] At least a part of the light emitted by the irradiation device 1 that propagates to the inspection object 9 is reflected or scattered by the surface of the inspection object 9. In the example of FIG. 1, light L1 is an example of light emitted by the irradiation device 1. A part of the reflected or scattered light is incident on the photographing device 2. In the example of FIG. 1, light L2 is an example of light that is reflected or scattered by the inspection object 9 and incident on the photographing device 2. The photographing device 2 receives the incident light. In this way, the photographing device 2 receives at least a part of the result of the light emitted by the irradiation device 1 being reflected or scattered by the inspection object 9. Then, the photographing device 2 generates information indicating the result of the light reception. The information indicating the result of the light reception is image set information described later. The photographing device 2 will be described in more detail.

[0015] The image capturing device 2 is an image capturing device equipped with an optical filter that transmits only a predetermined polarized light and a plurality of light receiving elements arranged on a surface, and the light receiving elements receive the predetermined polarized light. The image capturing device 2 generates a set of image data of a two-dimensional image showing the result of light reception by the light receiving elements (hereinafter referred to as "image set information"), and outputs the generated image set information to the estimation device 3. The image showing the result of light reception by the light receiving elements is an image showing the intensity of light received by each light receiving element for each frequency of light incident on the image capturing device 2. Note that since the image capturing device is equipped with a set of light receiving elements, the result of light reception by the light receiving elements is also the result of light reception by the image capturing device. The image capturing device 2 is, for example, a hyperspectral camera. The image set information is, for example, a data cube.

[0016] The estimation device 3 includes a control unit 31 having a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program.

[0017] The control unit 31 controls the operation of the components of the estimation system 100 by wired or wireless communication via a predetermined communication interface of the estimation device 3, such as a communication interface of an interface unit 32 described later. Thus, the control unit 31 controls the operation of the irradiation device 1, for example.

[0018] The control unit 31 also executes an orientation information acquisition process and a surface roughness estimation process. The orientation information acquisition process is a process for acquiring orientation information, which is information indicating the orientation of each portion of the inspection target 9. The orientation information acquisition process may be any process that can acquire orientation information. The orientation information acquisition process may be, for example, a process for acquiring orientation information stored in a predetermined storage device such as the storage unit 33 described later. Acquiring information stored in the storage device is a process for reading information stored in the storage device from the storage device. Other examples of the orientation information acquisition process will be described later.

[0019] The surface roughness estimation process is a process for estimating a value of a surface roughness parameter representing the surface roughness of the inspection object 9 based on image set information and orientation information. The surface roughness estimation process may be any process that estimates a value of a surface roughness parameter based on image set information and orientation information.

[0020] <An example of surface roughness estimation processing> An example of the surface roughness estimation process will be described. In the surface roughness estimation process, the control unit 31 controls the operation of the irradiation device 1, and causes each light source of the irradiation device 1 to emit light so that the distribution of luminance in the plane where the light sources are arranged satisfies a pattern condition. The pattern condition is a condition that there is no change in luminance in one direction in the plane where the light sources are arranged, there is a change in luminance in the other direction that is a direction perpendicular to the one direction, and further, the luminance in the other direction is expressed by a sine wave with the position in the other direction as an independent variable. For simplicity of explanation, the following description will be given taking the case where the irradiation device 1 is a display as an example. Note that, when the irradiation device 1 is a display, emitting light from each light source so that the distribution of luminance in the plane where the light sources are arranged satisfies the pattern condition means displaying an image with a luminance distribution that satisfies the pattern condition.

[0021] FIG. 2 is a diagram showing an example of a luminance distribution that satisfies a pattern condition in an embodiment. The one labeled with reference numeral 11 in FIG. 2 is an example of a display screen. That is, the one labeled with reference numeral 11 is an example of a set of a plurality of light sources arranged on a surface. Therefore, in the example of FIG. 2, the surface on which the light sources are arranged is parallel to the paper surface of FIG. 2. The x-axis and y-axis in FIG. 2 are orthogonal to each other. In the example of FIG. 2, there is no change in luminance in the y-axis direction. There is a change in luminance in the x-axis direction, and the luminance is represented by a sine wave with the position in the x-axis direction as an independent variable. Therefore, the sine wave pattern shown in FIG. 2 is an example of a luminance distribution that satisfies a pattern condition.

[0022] After each light source of the irradiation device 1 emits light so that the distribution of luminance within the surface satisfies the pattern condition, part of the light is reflected or scattered by the inspection object 9 and enters the photographing device 2. In the surface roughness estimation process, the control unit 31 acquires the image set information output by the photographing device 2 that receives the incident light via a communication interface provided in the estimation device 3.

[0023] In the surface roughness estimation process, the control unit 31 estimates the surface roughness of the inspection object 9 based on the image set information.

[0024] <Estimation of surface roughness> The estimation of surface roughness will be explained. Hereinafter, an image of a luminance distribution that satisfies the pattern conditions is called a sine wave pattern image. When a sine wave pattern image displayed on an LCD monitor is used as a light source to illuminate an object, and an object is photographed while shifting the sine wave reflected in the sine wave pattern image, an object with low surface roughness such as a mirror will have a large change in the luminance of the observed specular reflected light, since the sine wave pattern image is reflected as is. Note that the sine wave in Figure 2, which uses the position in the x-axis direction as an independent variable, is an example of a sine wave reflected in a sine wave pattern image.

[0025] On the other hand, for objects with high surface roughness, the change in brightness of the observed specular reflected light is small due to contributions from light sources in various directions. For this reason, it is possible to estimate the surface roughness from the degree of change in brightness of the specular reflected light observed when the sine wave in the sine wave pattern image is shifted. Note that the integral range of the light source differs depending on the magnitude of the surface roughness. For this reason, sine wave pattern images with multiple different periods are used to estimate the surface roughness.

[0026] This will be described in more detail with reference to the drawings. Figure 3 is a first explanatory diagram for explaining the estimation of surface roughness in the embodiment. Figure 4 is a second explanatory diagram for explaining the estimation of surface roughness in the embodiment.

[0027] The estimation of surface roughness will be explained using a simplified Torrance-Sparrow model (see Reference 1). The luminance i of the specular reflected light observed at a certain point on the surface of the inspection object 9 is expressed by the following equation (1).

[0028] Reference 1: KE Torrance and EM Sparrow, “Theory for off-specular reflection from roughened surfaces”, Journal of the Optical Society of America, Vol. 57, pp. 1104-1114, 1967.

[0029]

number

[0030] ρ represents the specular reflectance. σ is an example of a surface roughness parameter. That is, σ represents the surface roughness. β represents the angle between the normal n and the half vector h that bisects the light source direction l and the line of sight v. The normal n is perpendicular to the surface of the inspection object 9. The direction of the normal n is the direction indicated by the orientation information. Therefore, in estimating the surface roughness, the orientation information is used to use the normal n.

[0031] For simplicity, the following description will be given assuming that the light source direction l, normal n, and line of sight direction v are on the same plane. In Fig. 3, the light source direction l0 is a light source direction in which the line of sight direction v is the specular reflection direction. Fig. 3 shows that the angle α between the light source direction l0 and the light source direction l is expressed as α = 2β. Therefore, the luminance i of the specularly reflected light is expressed by the following equation (2).

[0032]

number

[0033] A scene will be described in which the inspection target 9 is illuminated by light emitted from the display (i.e., the illumination device 1) so that the pattern condition is satisfied. The coordinate x on the x-axis in Fig. 4 indicates a position on the above-mentioned surface of the display. Therefore, the x-axis in Fig. 4 is parallel to the above-mentioned surface of the display. The origin O in Fig. 4 represents the intersection point between the line extending the light source direction l0 and the display.

[0034] The x' axis in Figure 4 is an axis that passes through the origin O and is perpendicular to the light source direction l0. Using the coordinate x' on the x' axis, tan α=x' / d holds. When the surface roughness of the inspection object 9 is small, only the light source with a small α contributes to the brightness of the surface of the inspection object 9. Therefore, in such a case, the approximation tan α ~ α holds. Therefore, when the surface roughness of the inspection object 9 is small, the following equation (3) holds. r represents the observed brightness.

[0035]

number

[0036] The display displays an image of a cosine wave-like luminance distribution with amplitude a, period 1 / u0, and phase 2πu0s expressed by the following formula (4). The image of the cosine wave-like luminance distribution with amplitude a, period 1 / u0, and phase 2πu0s expressed by formula (4) is an example of a sine wave pattern image, and the cosine wave expressed by formula (4) is an example of a sine wave reflected in a sine wave pattern image. As is well known, a cosine wave and a sine wave differ only in phase by 90 degrees. Therefore, a cosine wave is a type of sine wave. u0 is a predetermined value. s is a predetermined value.

[0037]

number

[0038] If the angle between the x-axis and the x'-axis is θ, then the period 1 / u'0 of the cosine wave expressed by the above equation (4) as viewed from a point on the surface of the object 9 is given by 1 / u'0 = (cos θ) / u0, as can be seen from Figure 4, when the distance d between the point on the surface of the object 9 and the display is large (i.e., x' / d << 1).

[0039] Therefore, when the object 9 is illuminated by light emitted from a display (i.e., the illumination device 1) whose luminance distribution is a cosine wave luminance distribution expressed by the above equation (4), the luminance i(s;d,θ,u0) of a point on the surface of the object 9 is expressed by the following equation (5).

[0040]

number

[0041] In other words, when the object 9 is illuminated by light emitted from a display (i.e., the illumination device 1) whose luminance distribution is a cosine wave luminance distribution expressed by the above equation (4), the luminance i of a point on the surface of the object 9 is expressed by the convolution of the luminance r under the point light source shown in equation (3) and the cosine wave of equation (4).

[0042] As shown in equation (3), the luminance r shown in equation (3) is a Gaussian distribution function. Therefore, when the inspection object 9 is illuminated by light emitted from a display (i.e., the illumination device 1) whose luminance distribution is a cosine wave luminance distribution expressed by the above equation (4), the luminance i of a point on the surface of the inspection object is equal to the cosine wave expressed by equation (4) smoothed by a Gaussian filter.

[0043] The Fourier transform of the convolution integral, F{r*l}, is equal to the product of the Fourier transform results of each function to be convolved (i.e., F{r}(u) and F{l}(u)), as expressed in the following equation (6).

[0044]

number

[0045] The results of the Fourier transform of each function are expressed by the following equations (7) and (8).

[0046]

number

[0047]

number

[0048] Here, δ() represents the Dirac delta function. Substituting equations (7) and (8) into equation (5) and performing a Fourier transform, we obtain the following equation (9).

[0049]

number

[0050] The left side of equation (9) represents the observed luminance. Thus, equation (9) indicates that the luminance observed at a point on the surface of the inspection object 9 is expressed by a cosine-like function related to the phase of a cosine wave with amplitude a and period 1 / u0 displayed on the display. Equation (9) contains two unknowns: the amplitude C of the cosine wave and the surface roughness parameter σ. Therefore, the values ​​of these unknowns can be obtained by using two different cosine waves. Therefore, the surface roughness can be estimated by displaying at least two images whose luminance distribution follows a sine wave and whose sine waves have different phases, each image displayed on the display at different timings.

[0051] In this way, in the surface roughness estimation process, the control unit 31 causes the irradiation device 1 to display at least two images whose luminance distribution follows a sine wave and whose sine wave phases are different from each other, at different timings. The display is performed by emitting light, and the emitted light is reflected or scattered by the inspection object 9. A part of the reflected or scattered light is incident on the photographing device 2, and the photographing device 2 obtains image set information. As described above, the irradiation device 1 displays at least two images at different timings, so the photographing device 2 obtains image set information for each image displayed by the irradiation device 1. In the surface roughness estimation process, the control unit 31 estimates the surface roughness of the inspection object 9 using equation (9) based on the at least two pieces of image set information thus obtained by the photographing device 2. Specifically, the estimation of the surface roughness is the acquisition of a value of a surface roughness parameter.

[0052] <Example of Hardware Configuration of Estimation Device 3> 5 is a diagram illustrating an example of a hardware configuration of the estimation device 3 in an embodiment. As described above, the estimation device 3 includes the control unit 31 and executes a program. The estimation device 3 functions as a device including the control unit 31, an interface unit 32, and a storage unit 33 by executing the program.

[0053] More specifically, the processor 91 reads out a program stored in the storage unit 33, and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the estimation device 3 functions as a device including the control unit 31, the interface unit 32, and the storage unit 33.

[0054] The control unit 31 performs, for example, the orientation information acquisition process and the surface roughness estimation process as described above. The control unit 31 controls, for example, the operation of each functional unit included in the estimation device 3. The control unit 31 acquires image set information, for example, via the interface unit 32. The control unit 31 acquires, for example, information stored in the memory unit 33. The process of acquiring the information stored in the memory unit 33 is specifically a read process.

[0055] The interface unit 32 includes a communication interface for connecting the estimation device 3 to an external device. The interface unit 32 communicates with the external device via wired or wireless communication. The external device is, for example, the irradiation device 1. The external device is, for example, the imaging device 2.

[0056] The interface unit 32 includes input devices such as a mouse, a keyboard, and a touch panel. The interface unit 32 may be configured as an interface that connects these input devices to the estimation device 3. In this manner, the interface unit 32 accepts input of various information to the estimation device 3 via the input device, either wired or wirelessly. Note that the image set information does not necessarily need to be input to a communication interface, and may be input to an input device.

[0057] The interface unit 32 outputs various information. The interface unit 32 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 32 may be configured as an interface that connects these display devices to the estimation device 3. The interface unit 32 outputs information input to the interface unit 32, for example.

[0058] The memory unit 33 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The memory unit 33 stores various information related to the estimation device 3. The memory unit 33 stores information required for, for example, an orientation information acquisition process and a surface roughness estimation process. The memory unit 33 stores, for example, orientation information in advance. The memory unit 33 stores, for example, various information generated by the operation of the control unit 31. The memory unit 33 stores, for example, information acquired by the interface unit 32.

[0059] 6 is a diagram showing an example of a flow of processes executed by the estimation device 3 in the embodiment. The control unit 31 executes a direction information acquisition process (step S101). Next, the control unit 31 executes a surface roughness estimation process (step S102).

[0060] <Another example of orientation information acquisition process> Another example of the orientation information acquisition process is shown below. The color of the observed specular reflection has the property of matching the color of the light source (see Reference 2).

[0061] Reference 2: SA Shafer, “Using color to separate reflection components”, Color Research and Application, Vol.10, Issue.4, pp.210-218, 1985.

[0062] Therefore, in the orientation information acquisition process, the light source position may be estimated based on the color pattern displayed on a display (an example of the illumination device 1) and the color of the observed specular reflection component, and the normal direction may be estimated by determining the direction that bisects the obtained light source direction and the line of sight direction.

[0063] Fig. 7 is a diagram showing an example of a color pattern displayed on a display for acquiring orientation information in an embodiment. Fig. 7 shows an example of a color pattern displayed on a screen 11 of a display. In the color pattern shown in Fig. 7, the value of G (i.e., green) among the RGB values ​​increases from position P1 to position P2. In the color pattern shown in Fig. 7, the value of B (i.e., blue) among the RGB values ​​increases from position P1 to position P3. In the color pattern shown in Fig. 7, the value of R (i.e., red) among the RGB values ​​increases from position P4 to position P3.

[0064] The example in FIG. 7 is an example, and the color pattern at least satisfies the condition that the color is different for each position in the plane. Furthermore, the color pattern satisfies the condition that, for a color change from a color at one point in the plane, such as the screen 11, to a color at another point different from the one point, the amount of change in the value representing the density of each primary color of the color is proportional to the distance in the plane connecting the one point and the other point. Therefore, in a color pattern, for example, when a color is represented by a three-dimensional vector of (R, G, B), each value of RGB is proportional to the coordinate of the display. In a color pattern, for example, when a color is represented by a four-dimensional vector of (C, M, Y, K), each value of CMYK is proportional to the coordinate of the display.

[0065] Therefore, the orientation information may be obtained based on the result of irradiating the light emitted by the light source provided in the irradiation device 1 onto the inspection object 9 when the color differs depending on the position within the plane, and the color change from the color at one point within the plane to the color at another point different from the one point is such that the amount of change in the value representing the concentration of each primary color is proportional to the distance within the plane connecting the one point and the other point.

[0066] The RGB values ​​are an example of values ​​that represent the density of each of the primary colors. The values ​​that represent the density of each of the primary colors may be CMYK values ​​or other color code values.

[0067] A more specific example of obtaining orientation information using a color pattern will be described. When a color pattern is displayed on a display and an object is photographed by the photographing device 2, and the specular reflection component is separated by the method of Kurita et al. (see Reference 3), the color vector (i R , i G , i B ) T is parallel to the color vector of a point on the projected image in the specular reflection direction. In other words, the following formula (10) is satisfied. Note that, as is well known, T represents the transpose of a vector or a matrix.

[0068] Reference 3: Teppei Kurita et al., “Real-time reflection component separation and applied signal processing technology using a back-illuminated four-way polarization CMOS image sensor”, Proceedings of SSII2018, IS1-34, 2018

[0069]

number

[0070] Here, H represents the display height and W represents the display width. x represents the coordinate on the x-axis. y represents the coordinate on the y-axis. Equation (10) is solved for x and y using linear least squares to calculate the light source position in the specular direction. After calculating the light source position, the normal direction is estimated by finding the direction that bisects the obtained light source direction and the known line of sight direction.

[0071] Therefore, in the direction information acquisition process, the control unit 31 first acquires the result of the surface of the inspection target 9 being photographed by the photographing device 2 with the color pattern displayed. Next, based on the acquired result, information indicating the position of the light source is acquired using equation (10). Next, based on the acquired information indicating the position, information indicating the light source direction (hereinafter referred to as "light source direction information"). Next, based on the acquired light source direction information and information indicating the line of sight direction previously stored in a predetermined storage device such as the storage unit 33 (hereinafter referred to as "line of sight direction information"), information indicating the light source direction indicated by the light source direction information, the line of sight direction indicated by the line of sight direction information, and the bisection direction (hereinafter referred to as "bisection direction information"). Next, the control unit 31 estimates the normal direction based on the bisection direction information. Specifically, the estimation of the normal direction based on the bisection direction information is a process of estimating the bisection direction as a normal line.

[0072] In addition, the result of photographing the surface of the inspection object 9 by the photographing device 2 with the color pattern displayed is a collection of image data of a two-dimensional image that indicates the result of light reception by the light receiving element, and is therefore an example of image collection information.

[0073] <Experimental Results> This shows the results of an experiment to estimate surface roughness by performing a process to obtain orientation information using a color pattern and a process to estimate surface roughness. In the experiment, two types of glossy objects with unknown shapes were used as inspection targets.

[0074] Fig. 8 is a first explanatory diagram for explaining the situation of an experiment in an embodiment. Fig. 8 shows that a display 111 and a polarization camera 211 were used in the experiment. The display 111 shown in Fig. 8 is an example of the illumination device 1, and the polarization camera 211 is an example of the imaging device 2. Fig. 8 shows a coin 911 as an example of the inspection object 9. In the situation shown in Fig. 8, the coin 911 is illuminated by the display 111, and the illuminated coin 911 is imaged by the polarization camera 211.

[0075] In the orientation information acquisition process in the experiment, the color pattern shown in Fig. 7 was used as the color pattern, and the display in Fig. 8 displayed the color pattern. In the surface roughness estimation process in the experiment, the brightness distribution image in Fig. 2 was displayed as a sine wave pattern image on the display in Fig. 8. In the experiment, one type of color pattern was used, and 180 types of sine wave pattern images were used. The 180 types of sine wave pattern images were as follows: 36 types of sine wave pattern images with a sine wave period of 80px and the phase shifted by 10 degrees increments, 36 types of sine wave pattern images with a sine wave period of 100px and the phase shifted by 10 degrees increments, 36 types of sine wave pattern images with a sine wave period of 120px and the phase shifted by 10 degrees increments, 36 types of sine wave pattern images with a sine wave period of 160px and the phase shifted by 10 degrees increments, and 36 types of sine wave pattern images with a sine wave period of 240px and the phase shifted by 10 degrees increments.

[0076] In the experiment, the display in FIG. 8 displayed an image, and the polarized camera in FIG. 8 received reflected light from an object to be inspected.

[0077] Fig. 9 is a diagram showing an image of one of two types of inspection objects used in the embodiment. Fig. 10 is a diagram showing an image of the other of two types of inspection objects used in the embodiment. The inspection object shown in Fig. 9 is a coin. The inspection object shown in Fig. 10 is an artificial leather sheet.

[0078] FIG. 11 is a first diagram showing an example of an experimental result in the embodiment. FIG. 12 is a second diagram showing an example of an experimental result in the embodiment. More specifically, FIG. 11 is a diagram showing an example of the result of the orientation information acquisition process in which a coin is inspected. FIG. 12 is a diagram showing an example of the result of the orientation information acquisition process in which a synthetic leather sheet is inspected. FIGS. 11 and 12 show that a substantially uniform normal direction is estimated for the smooth part of the coin and the synthetic leather sheet. Also, FIGS. 11 and 12 show that a normal direction different from that of the smooth part is estimated for the edge part of the letters or pattern on the coin. These are substantially the same as the results estimated in advance by a user based on the feel or appearance.

[0079] The average angle between the normal estimated by the orientation information acquisition process using the color pattern and the normal estimated by the Gray code pattern for all pixels was 5.11 degrees for the coin and 6.91 degrees for the synthetic leather sheet. This shows that the estimation results by the orientation information acquisition process using the color pattern are highly accurate. The estimation of the normal using the Gray code pattern is a technology described in Reference 4 below.

[0080] Reference 4: Jie Lu, et al. "Mirror surface reconstruction using polarization field." 2019 IEEE International Conference on Computational Photography (ICCP). IEEE, 2019.

[0081] FIG. 13 is a second explanatory diagram for explaining the state of an experiment in the embodiment. FIG. 14 is a third diagram showing an example of an experiment result in the embodiment. FIG. 15 is a fourth diagram showing an example of an experiment result in the embodiment. FIG. 16 is a fifth diagram showing an example of an experiment result in the embodiment. FIG. 17 is a sixth diagram showing an example of an experiment result in the embodiment. FIG. 18 is a seventh diagram showing an example of an experiment result in the embodiment.

[0082] Fig. 13 shows, among the pixels in an image shown by the result of shooting by the shooting device 2, the pixels evaluated in obtaining the experimental results of Figs. 14 to 16 and the pixels evaluated in obtaining the experimental results of Figs. 17 and 18. The pixel A1 shown in Fig. 13 was used in obtaining the experimental results of Figs. 14 to 16. The pixels A1 and A2 shown in Fig. 13 were used in obtaining the experimental results of Figs. 17 and 18. Pixel A2 has a surface with more projections and recesses than pixel A1.

[0083] 14 to 16 show the results of fitting a sine wave to the change in luminance value of an image captured while shifting a sine wave pattern image of a different period. Note that shifting the sine wave pattern image means continuously changing the phase of the sine wave shown in the sine wave pattern image. Note that fitting a sine wave means evaluating the degree of agreement between the fitting target and the sine wave.

[0084] More specifically, FIG. 14 shows the result when a sine wave pattern image of a sine wave with a period of 80px is displayed on the display, FIG. 15 shows the result when a sine wave pattern image of a sine wave with a period of 120px is displayed on the display, and FIG. 16 shows the result when a sine wave pattern image of a sine wave with a period of 240px is displayed on the display. The horizontal axis of FIG. 14 to FIG. 16 indicates phase, and the vertical axis indicates amplitude. More specifically, amplitude indicates the intensity of the observed luminance. FIG. 14 to FIG. 16 show that the sine wave is fitted with high accuracy, and the amplitude also changes according to the period of the sine wave.

[0085] More specifically, Figures 17 and 18 show the results of fitting a sine wave to the luminance values ​​of an image captured while shifting a sine wave pattern image, plotting the obtained amplitude in frequency space, and fitting a Gaussian distribution. Figure 17 shows the results when the pixel to be evaluated is pixel A1, and Figure 18 shows the results when the pixel to be evaluated is pixel A2, and shows the results when all of the above-mentioned 80px, 100px, 120px, 160px, and 240px are used as the sine wave frequency. Figures 17 and 18 show that the Gaussian function is sharper when the surface is uneven.

[0086] FIG. 19 is an eighth diagram showing an example of an experimental result in the embodiment. FIG. 20 is a ninth diagram showing an example of an experimental result in the embodiment. FIG. 21 is a tenth diagram showing an example of an experimental result in the embodiment. FIG. 19 is a result of estimating the surface roughness for all pixels, and shows an example of the result of estimating the surface roughness for a coin using 180 images. FIG. 20 is a result of estimating the surface roughness for all pixels, and shows an example of the result of estimating the surface roughness for a coin using 6 images. FIG. 21 is a result of estimating the surface roughness for all pixels, and shows an example of the result of estimating the surface roughness for a synthetic leather sheet using 180 images. In both the case of a coin and the case of a synthetic leather sheet, the 180 images correspond one-to-one to one of the 180 types in total, specifically, 5 types of periods x the above-mentioned 36 types.

[0087] FIG. 19 shows that the smooth portion of the coin has a substantially uniform roughness. FIG. 19 shows that the surface roughness is greater inside the characters and in the patterned portions compared to the smooth portions. The surface roughness is high inside the characters and in the patterned portions because there are many projections and recesses. The results shown in FIG. 19 show that this fact was estimated by estimating the surface roughness by performing an orientation information acquisition process using a color pattern and a surface roughness estimation process. Therefore, the results shown in FIG. 19 show that when estimating the surface roughness by performing an orientation information acquisition process using a color pattern and a surface roughness estimation process, it is possible to estimate the surface roughness with high accuracy.

[0088] Fig. 20 shows that the results estimated from six images are substantially the same as the results in Fig. 19. Therefore, Fig. 20 shows that when estimating surface roughness by performing orientation information acquisition processing using a color pattern and surface roughness estimation processing, it is possible to estimate surface roughness with high accuracy even with a small number of images.

[0089] FIG. 21 shows the result that the surface roughness is greater than the smooth portion of the coin. The synthetic leather sheet has an overall unevenness, and therefore the surface roughness is greater than the smooth portion of the coin. Therefore, the result shown in FIG. 21 shows that this fact was inferred by estimating the surface roughness by performing the orientation information acquisition process using a color pattern and the surface roughness estimation process. Therefore, the result shown in FIG. 21 shows that when estimating the surface roughness by performing the orientation information acquisition process using a color pattern and the surface roughness estimation process, it is possible to estimate the surface roughness with high accuracy.

[0090] The estimation device 3 of the embodiment configured as described above estimates the surface roughness of the inspection object by executing the surface roughness estimation process. As is well known, surface roughness is a quantity that has a strong correlation with surface abnormalities. For example, if there is an area with many scratches on a smooth surface, the surface roughness of that area is a value indicating roughness, and has a large difference in value beyond the error from the smooth area. Therefore, when the estimation device 3 estimates the surface roughness, the effort required for a person to estimate the abnormality of the surface of the inspection object is reduced. Furthermore, since the estimation device 3 estimates the surface roughness, the amount of specialized knowledge required for the inspection required of the person performing the inspection is reduced. As a result, it is possible to reduce the burden on the person performing the inspection for abnormalities on the surface of the inspection object, such as by increasing the number of people performing the inspection.

[0091] The estimation system 100 configured in this manner includes the estimation device 3. Therefore, it is possible to reduce the burden on the person inspecting the surface of the inspection object for abnormalities.

[0092] (Modification) It is not necessary for the control unit 31 to control the operation of the irradiation device 1. For example, if the irradiation device 1 is set in advance to switch and display the sine wave pattern image at a predetermined timing, the control unit 31 does not necessarily have to control the operation of the irradiation device 1. In addition, the control unit 31 does not necessarily have to control the operation of the imaging device 2.

[0093] Therefore, although it has been described above that the control unit 31 controls the operation of the irradiation device 1 in the surface roughness estimation process, this is merely one example, and as long as the surface roughness of the inspection object 9 is estimated based on image set information and orientation information in the surface roughness estimation process, it is not necessarily necessary for the operation of the irradiation device 1 to be controlled in the surface roughness estimation process performed by the control unit 31.

[0094] All or part of the functions of the estimation system 100 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The program may be transmitted via a telecommunications line.

[0095] The estimation device 3 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each functional unit included in the estimation device 3 may be distributed and implemented in the plurality of information processing devices.

[0096] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0097] 100: Estimation system; 1: Irradiation device; 2: Imaging device; 3: Estimation device; 31: Control unit; 32: Interface unit; 33: Storage unit; 91: Processor; 92: Memory

Claims

1. An illumination device having a plurality of light sources arranged on a surface; an imaging device that includes a plurality of light receiving elements arranged on a surface thereof and receives at least a portion of the light irradiated by the irradiating device and reflected or scattered by the inspection object; a control unit that estimates a surface roughness of the inspection object based on a result of light reception by the light receiving element and orientation information indicating an orientation of each portion of the inspection object; Equipped with the light source emits light in a predetermined polarization direction; The polarization direction of the light emitted from the light source is the same regardless of the light source, The light received by the light receiving element is light that is specularly reflected, diffusely reflected, or subsurface scattered by the inspection object from the light emitted by the illumination device, and the polarization direction of the light is the same as the polarization direction of the light emitted by the illumination device; The luminance distribution within the plane in which the light sources are arranged satisfies the pattern condition that there is no change in luminance in one direction within the plane, there is a change in luminance in another direction that is a direction perpendicular to the one direction, and further, the luminance in the other direction is expressed by a sine wave with the position in the other direction as an independent variable. Estimation system.

2. The orientation information is obtained based on the result of irradiating the light emitted by the light source onto the inspection object when the color emitted by the light source is different for each position on the plane on which the light sources are arranged, and when the color at one point on the plane changes to the color at another point different from the one point, the amount of change in the value representing the concentration of each primary color of the color is proportional to the distance on the plane connecting the one point and the other point. The estimation system according to claim 1 .

3. The orientation information indicates the orientation, for example, as a normal orientation. The estimation system according to claim 1 .

4. a control unit that estimates the surface roughness of the inspection object based on a result of light reception by an imaging device that receives at least a part of the result of light irradiated by an illumination device having a plurality of light sources arranged on a surface and reflected or scattered by the inspection object, the imaging device having a plurality of light receiving elements arranged on a surface, and orientation information that indicates the orientation of each part of the inspection object; Equipped with the light source emits light in a predetermined polarization direction; The polarization direction of the light emitted from the light source is the same regardless of the light source, The light received by the light receiving element is light that is specularly reflected, diffusely reflected, or subsurface scattered by the inspection object from the light emitted by the illumination device, and the polarization direction of the light is the same as the polarization direction of the light emitted by the illumination device; The luminance distribution within the plane in which the light sources are arranged satisfies the pattern condition that there is no change in luminance in one direction within the plane, there is a change in luminance in another direction that is a direction perpendicular to the one direction, and further, the luminance in the other direction is expressed by a sine wave with the position in the other direction as an independent variable. Estimation device.

5. a control step of estimating the surface roughness of the inspection object based on the result of light reception by an imaging device having a plurality of light receiving elements arranged on a surface, the imaging device receiving at least a part of the result of light irradiated by an illumination device having a plurality of light sources arranged on a surface and reflected or scattered by the inspection object, and on orientation information indicating the orientation of each part of the inspection object; having the light source emits light in a predetermined polarization direction; The polarization direction of the light emitted from the light source is the same regardless of the light source, The light received by the light receiving element is light that is specularly reflected, diffusely reflected, or subsurface scattered by the inspection object from the light emitted by the illumination device, and the polarization direction of the light is the same as the polarization direction of the light emitted by the illumination device; The luminance distribution within the plane in which the light sources are arranged satisfies the pattern condition that there is no change in luminance in one direction within the plane, there is a change in luminance in another direction that is a direction perpendicular to the one direction, and further, the luminance in the other direction is expressed by a sine wave with the position in the other direction as an independent variable. Estimation method.

Images