Image processing device, image processing method, and program

Abstract

The objective is to improve the accuracy of inspecting object surfaces based on captured images. [Solution] The image processing device sets which light sources to be illuminated from among a plurality of light sources arranged at different positions based on the glossiness of the object to be inspected, and performs inspection processing on the object based on a plurality of images obtained by imaging the object illuminated by the set light sources.

Description

Technical Field

[0001] The present invention relates to an image processing technology for inspecting an object.

Background Art

[0002] As an appearance inspection technology for industrial products, a technology for detecting unevenness on an inspection surface is known. As a method for detecting unevenness, a photometric stereo technology for estimating the inclination and reflectance of each position on the inspection surface based on a plurality of images obtained by imaging an object irradiated with light from a plurality of directions may be used. Patent Document 1 discloses a technology for storing in advance the lighting patterns of a plurality of light sources in a lighting device for photometric stereo, and reading out and using the lighting patterns according to a control signal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When inspecting the surface of an object based on a captured image, depending on the gloss characteristics of the object surface, the accuracy of the inspection may decrease due to the influence of the light source reflected on the inspection surface.

[0005] Therefore, an object of the present invention is to improve the accuracy of inspecting the surface of an object based on a captured image.

Means for Solving the Problems

[0006] To solve the above problems, the image processing apparatus according to the present invention is characterized by comprising: setting means for setting which light source to be lit from among a plurality of light sources arranged at different positions based on the glossiness of the object to be inspected; and processing means for performing inspection processing on the object based on a plurality of images obtained by imaging the object irradiated with light from the set light sources. [Effects of the Invention]

[0007] According to the present invention, the accuracy of inspecting object surfaces based on captured images can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing how to image the surface of an object. [Figure 2] Diagram showing the appearance and hardware configuration example of a visual inspection system. [Figure 3] This figure shows an example of the functional configuration of a visual inspection system. [Figure 4] Diagram showing an example of light source placement. [Figure 5] Flowchart illustrating the processing in an image processing system [Figure 6] A flowchart showing the process of setting the lighting pattern of a light source. [Figure 7] Diagram showing an example of a user interface [Figure 8] This diagram shows an example of a list file that describes the correspondence between the object being inspected, its gloss information, and its light source ID. [Modes for carrying out the invention]

[0009] Each embodiment will be described below with reference to the drawings. Note that the following embodiments do not necessarily limit the invention. Furthermore, not all combinations of features described in each embodiment are essential to the solution of the present invention.

[0010] [First Embodiment] There is a technique for inspecting the surface of an object based on images obtained by imaging an object illuminated with light from multiple different directions. In this inspection, depending on the gloss characteristics of the object's surface, the accuracy of deriving the shape information and reflectivity of the object's surface may decrease due to the influence of the light source reflected on the inspection surface. Figure 1 shows how two object surfaces with different gloss mapping properties (the clarity of the reflected image), which is one element of gloss characteristics, are imaged. Geometric condition 101 is a geometric condition for imaging an object illuminated with light from a position with a relatively small zenith angle, and geometric condition 102 is a geometric condition for imaging an object illuminated with light from a position with a relatively large zenith angle.

[0011] Relationship 103 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with high gloss mapping at position A under geometric condition 101. Relationship 104 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with high gloss mapping at position B. Relationship 105 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with high gloss mapping at position C. Here, the reflection characteristics are represented by the generally known BRDF (Bidirectional Reflectance Distribution Function). At positions B and C, the strong reflection component (surface reflection component) near specular reflection is not included in the receiving angle range, whereas at position A, the surface reflection component is included in the receiving angle range. Image 115 shows an image obtained by imaging the surface of an object with high gloss mapping under geometric condition 101. In Image 115, it can be seen that the effect of gloss is strongly present in a part of the object surface. Image 119 shows an image obtained by imaging two objects with high gloss mapping together under geometric condition 101. In Image 119, it can be seen that when there are multiple objects, one of the objects is strongly affected by gloss.

[0012] Relationship 106 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with low gloss mapping at position A under geometric condition 101. Relationship 107 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with low gloss mapping at position B under geometric condition 107. Relationship 108 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with low gloss mapping at position C under geometric condition 108. On an object surface with low mapping, the surface reflection component is included in the receiving angle range at all positions A, B, and C. However, on an object surface with low mapping, the surface reflection component is dispersed over a wide angle, and strong reflection does not occur at a specific angle, so the difference in the amount of light received by the surface reflection component depending on the position is small. Image 116 shows an image obtained by imaging an object surface with low gloss mapping under geometric condition 101. In Image 116, it can be seen that a slight effect of gloss is present on a part of the object surface, but the effect is small. Image 120 shows an image obtained by imaging two objects with low gloss mapping together under geometric condition 101. In Image 120, it can be seen that when there are multiple objects, one of the objects is slightly affected by gloss, but the effect is small.

[0013] Relationship 109 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with high gloss mapping at position A under geometric condition 102. Relationship 110 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with high gloss mapping at position B. Relationship 111 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object with high gloss mapping at position C. At any of positions A, B, and C, the surface reflection component is not included in the receiving angle range. Image 117 shows an image obtained by imaging the surface of an object with high gloss mapping under geometric condition 102. It can be seen in Image 117 that the gloss effect is not included in the object surface. Image 121 shows an image obtained by imaging two objects with high gloss mapping together under geometric condition 102. It can be seen in Image 121 that even when there are multiple objects, the gloss effect is not included in the object surface.

[0014] Relationship 112 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object at position A with low gloss mapping under geometric condition 102. Relationship 113 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object at position B with low gloss mapping. Relationship 114 shows the relationship between the reflection characteristics and the receiving angle range when imaging an object at position C with low gloss mapping. At any of positions A, B, and C, the surface reflection component is not included in the receiving angle range. Image 118 shows an image obtained by imaging the surface of an object with low gloss mapping under geometric condition 102. It can be seen in Image 118 that the effect of gloss is not included in the object surface. Image 122 shows an image obtained by imaging two objects with low gloss mapping together under geometric condition 102. It can be seen in Image 122 that even when there are multiple objects, the effect of gloss is not included in the object surface. In other words, under geometric conditions with a relatively large zenith angle, the effect of gloss is less likely to be included in the captured image.

[0015] In photometric stereo imaging, it is known that using images obtained by imaging with multiple light sources positioned over the widest possible range in both zenith and azimuth angles improves the accuracy of deriving surface shape information and reflectivity of an object. However, for objects with high gloss, using light sources with small zenith angles results in a significant influence from surface reflection components, reducing the accuracy of deriving surface shape information and reflectivity. Therefore, in this embodiment, the lighting pattern of the light sources used for imaging is switched according to the gloss of the object. Specifically, for objects with high gloss, light sources with relatively large zenith angles are illuminated to minimize the influence of surface reflection components. On the other hand, for objects with low gloss, both light sources with small and large zenith angles are illuminated to improve the accuracy of deriving surface shape information and reflectivity of the object. This improves the accuracy of inspecting the object surface based on the captured images.

[0016] <Appearance and Hardware Configuration of the Visual Inspection System> Fig. 2(a) is a diagram showing an example of the hardware configuration of the appearance inspection system in this embodiment. Fig. 2(b) is a front view of the appearance of the entire appearance inspection system, and Fig. 2(c) is a top view of the appearance of the entire appearance inspection system. The appearance inspection system in this embodiment includes an image processing system 1, a start signal output interface 201, a conveyance control device 211, and a conveyance device 212.

[0017] The image processing system 1 includes an imaging control device 202, an imaging device 203, an image processing device 204, a display 205, a mouse 206, a keyboard 207, and a lighting device 208. The image processing system 1 is connected to a conveyance control device 211 that controls the conveyance device 212. The conveyance control device 211 conveys an object 213 to be inspected to the image processing system 1 using the conveyance device 212, and sends an inspection start signal to the image processing system 1 via the start signal output interface 201.

[0018] The imaging control device 202 has a control unit 214, controls the imaging device 203 and the lighting device 208, and images the object 213 in synchronization with the lighting of the light source. Specifically, when the imaging control device 2 receives an inspection start signal from the start signal input interface 215, it sends an imaging instruction to the imaging device 203 via the release signal output interface 216. In addition, the imaging control device 202 receives, via the sync signal input interface 217, a sync signal sent from the imaging device 203 to notify the lighting timing to an external strobe light source in synchronization with the imaging. Further, the imaging control device 202 lights the light sources of the lighting device 208 in a predetermined order and combination via the lighting signal output interface 219 according to the received sync signal. Through the above operations, the object 213 irradiated with light by the predetermined lighting can be imaged. The imaging control device 202 is connected to the image processing device 204 via a USB interface 218, receives commands from the image processing device 204, and provides information indicating the state of the imaging control device 202 to the image processing device 204.

[0019] The imaging device 203 includes a control unit 225 and an imaging optical system 221 including a lens, an imaging element, etc. The imaging device 203 generates an imaging image by quantizing an optical image obtained by imaging based on an imaging instruction received via a release signal input interface 220 by an image processing engine 223. The imaging device 203 transfers the generated imaging image to an image processing device 204 via a USB interface 224. In this embodiment, an example of acquiring and using a still image captured using a digital camera will be described, but a predetermined frame may be extracted from a moving image captured using a video camera and used as a still image. The imaging device 203 sends a sync signal to an imaging control device 202 via a sync signal output interface 222.

[0020] The image processing device 204 includes a RAM 226, a ROM 227, a CPU 228, a GPU 229, and a USB interface 230. Each component is connected by an internal bus. The processes shown in the flowchart described later are stored in the ROM 227 as program codes. These program codes are expanded in the RAM 226 and executed by the CPU 228 and the GPU 229.

[0021] The lighting device 208 has a plurality of light sources 209. The light source 209 in this embodiment is an LED, but it may be another light source such as a xenon lamp. An example of the arrangement of the plurality of light sources 209 is shown in FIG. 4. FIG. 4(a) is a front view of the lighting device 208, and FIG. 4(b) is a top view of the lighting device 208. The light sources 209 indicated by squares are installed so as to be dispersed hemispherically above the object 213, and at least one of the zenith angle and the azimuth angle is different from each other. In FIG. 4, reference numerals L01 to L31 indicate light source IDs (identification information) for identifying each light source. Note that the number and arrangement of the light sources are not limited to the above example, and it is sufficient that a plurality of light sources having different zenith angles and azimuth angles are arranged. The lighting device 208 lights a predetermined light source among the light sources 209 for a preset time in response to an instruction from the control unit 214.

[0022] In this embodiment of the image processing system 1, the imaging control device 202, imaging device 203, image processing device 204, and illumination device 208 are separate devices, but multiple devices may be integrated into one unit.

[0023] <Processing performed by the image processing system> Figure 3 shows an example of the functional configuration of the visual inspection system in this embodiment. The control unit 214 of the imaging control device 202 has an imaging control unit 302. The imaging control unit 302 has a release signal output unit 307, a sync signal input unit 308, a sync signal count unit 309, and a lighting signal output unit 310. The control unit 225 of the imaging device 203 has an imaging unit 303. The imaging unit 303 has a release signal input unit 311, a control unit 312, a sync signal output unit 313, and an image acquisition unit 314. The image processing device 204 has an image processing unit 304. The image processing unit 304 has a lighting pattern setting unit 315, an inspection image acquisition unit 316, an inspection unit 317, and an output unit 318. The transport control device 211 has a start signal output unit 301. The transport device 212 has a transport unit 306. The illumination device 208 has an illumination unit 305.

[0024] Figure 5 is a flowchart of the process performed by the image processing system 1 in this embodiment. When the object 213 is transported to a predetermined position by the transport device 212, the start signal output unit 301 of the transport control device 211 sends an inspection start signal to the imaging control unit 302. The process in Figure 5 starts when the imaging control unit 302 receives the inspection start signal. Hereafter, each step (process) is represented by adding an S before the reference numeral.

[0025] In S501, the lighting pattern setting unit 315 sets the lighting pattern of the light source. Figure 7 shows the user interface (UI) for inspection. In the UI of Figure 7, the selection area 701 allows the user to select an object to be inspected from a plurality of pre-registered types of objects. The selection area 702 allows the user to select the gloss characteristics of the object to be inspected from a plurality of pre-registered types of gloss characteristics. The user can specify either the selection area 701 or the selection area 702. The lighting pattern setting unit 315 sets the lighting pattern of the light source based on the object to be inspected specified by the user in the selection area 701, or the gloss characteristics specified by the user in the selection area 702. In this embodiment, the plurality of pre-registered types of objects are "Product A," "Product B," and "Product C." In this embodiment, the plurality of pre-registered types of gloss characteristics are "High Resolution," "Medium Resolution," and "Low Resolution." The UI of Figure 7 has a display area 706 that displays the inspection date and a display area 707 that displays the inspection time.

[0026] Figure 6 is a flowchart detailing the process of setting the lighting pattern of the light source in S501. In S601, the lighting pattern setting unit 315 receives instructions from the user regarding the object to be inspected. Here, the user's instructions regarding the object to be inspected are either the specification of the object to be inspected in the selection area 701, or the specification of gloss characteristics in the selection area 702. In S602, the lighting pattern setting unit 315 determines whether or not the object to be inspected in the selection area 701 has been specified. If the object to be inspected has been specified, in S603, the lighting pattern setting unit 315 acquires gloss information indicating the gloss characteristics corresponding to the selected object.

[0027] Figure 8 is a list file that describes the correspondence between the object to be inspected and its gloss information, and the correspondence between the gloss information and the light source ID. As shown in Figure 8, if the object selected by the user in the selection area 701 is "Product A", the selected object is identified as having "high resolution". If the object selected by the user in the selection area 701 is "Product B", the selected object is identified as having "medium resolution". If the object selected by the user in the selection area 701 is "Product C", the selected object is identified as having "low resolution". The lighting pattern setting unit 315 acquires gloss information based on the selected object and the above-described correspondence.

[0028] If the object to be inspected is not specified in S602, that is, if the gloss characteristics of the object to be inspected are specified, gloss information has already been obtained, and the process proceeds to S604. In S604, the lighting pattern setting unit 315 sets the lighting pattern of the light sources based on the gloss information obtained in S602 or S603. Specifically, the lighting pattern setting unit 315 sets the light source with the ID corresponding to the gloss information obtained in S602 or S603 as the lighting light source, based on the correspondence between the gloss information and the light source ID shown in the list in Figure 8. Note that the light source ID corresponds to the arrangement of light sources shown in Figure 4. In the list in Figure 8, it can be seen that the higher the gloss mapping quality, the more light sources with a relatively large zenith angle are used, and the lower the gloss mapping quality, the more light sources with high dispersion, including light sources with a relatively small zenith angle, are used.

[0029] In S502, the start signal output unit 301 determines whether the inspection start button 704 in the UI of Figure 7 has been pressed by the user. If it is determined that the inspection start button 704 has been pressed, the process proceeds to S503; if it is determined that the inspection start button 704 has not been pressed, the process returns to S502. If the inspection start button 704 has been pressed, the transport unit 306 transports the object to be inspected to a predetermined position in order to start the inspection. Once the object to be inspected has been transported to the predetermined position, the start signal output unit 301 sends an inspection start signal to the imaging control unit 302. In S503, the release signal output unit 307 receives the inspection start signal from the start signal output unit 301.

[0030] In S504, the imaging control unit 302 images the object to be inspected while sequentially illuminating the light sources based on the illumination pattern of the light sources set in S501. Specifically, first, when the release signal output unit 307 receives the inspection start signal, it sends a release ON signal to the release signal input unit 311, and the control unit 312 executes the imaging operation based on the release ON signal. In this embodiment, high-speed continuous imaging is performed using a known continuous imaging function. The continuous imaging function is a function that repeats imaging at a predetermined speed as long as the release signal continues to be sent, and in this embodiment, imaging is performed at 30 frames per second. In synchronization with the opening of the shutter curtain, the control unit 312 causes the sync signal output unit 313 to output a sync signal in order to notify the external strobe light source of the timing of light emission. When the sync signal input unit 308 receives the sync signal, the sync signal count unit 309 counts the number of times the sync signal has been received, i.e., the number of images captured. When the sync signal input unit 308 receives a sync signal, the lighting signal output unit 310 sends a lighting signal to the illumination unit 305 to switch the light sources to be lit sequentially, based on the lighting pattern of the light sources set by the lighting pattern setting unit 315. The illumination unit 305 lights up the light sources 209 based on the lighting signal. The lighting signal output unit 310 transmits the number of lit light sources, i.e., the total number of captured images, to the sync signal count unit 309 based on the lighting pattern of the light sources. When the number of sync signal receptions in the sync signal count unit 309 reaches the total number of captured images, it sends a stop signal to the release signal output unit 307. When the release signal output unit 307 receives the stop signal, it sends a release OFF signal to the release signal input unit 311, and the control unit 312 stops imaging. Through the above process, the light sources are lit for each object to be inspected for a preset number of images. The image acquisition unit 314 illuminates a predetermined light source and takes images of an object, then transfers multiple captured images obtained from this image to the image processing unit 304 as needed.

[0031] In S505, the image processing unit 304 performs inspection processing based on the captured image. The inspection image acquisition unit 316 receives the captured image from the image acquisition unit 314. The inspection unit 317 detects defects by spatial filtering using multiple captured images corresponding to light irradiation from multiple directions. The targets of spatial filtering are the captured image itself, or the normal image (shape image) and reflectance image (color image) obtained by combining multiple captured images using the photometric stereo method. The reaction value to the spatial filtering is integrated and quantified, and displayed as the abnormality level in the abnormality level display 709 in Figure 7. The inspection unit 317 determines whether the inspection is successful or not by comparing the judgment threshold set in the threshold setting area 703 in Figure 7 with the abnormality level. The threshold setting area 703 is a text box, and the user can set the judgment threshold. The output unit 318 displays OK (pass) / NG (fail) as the inspection result in the result display area 708 in Figure 7. If an abnormal event occurs, the details of the abnormal event are displayed in the display area 710. Furthermore, the output unit 318 notifies the transport unit 306 of inspection result information indicating OK / NG. Based on the inspection result information, the transport unit 306 performs predetermined processing, such as discharging objects determined to be NG from the system. In this embodiment, the inspection items are color and shape, but the inspection items are not limited to these. Anything that represents the appearance and can be identified by imaging is acceptable, such as material or pattern.

[0032] In S506, the start signal output unit 301 determines whether the inspection stop button 705 in the UI of Figure 7 has been pressed by the user. If the inspection stop button 705 has not been pressed, the process returns to S503. If the inspection stop button 705 has been pressed, the series of inspection operations is terminated.

[0033] As explained above, the image processing system 1 sets which light sources to illuminate from among multiple light sources arranged at different positions, based on the glossiness of the object to be inspected. Based on multiple images obtained by capturing the object illuminated by the set light sources, the system performs inspection processing on the object. This improves the accuracy of inspecting the object surface based on the captured images. In particular, for objects with high glossiness, a light source with a relatively small zenith angle is used to reduce the influence of surface reflection components. For objects with low glossiness, both light sources with small and large zenith angles are widely used to improve the accuracy of deriving shape information and reflectivity. By controlling the illumination of the light sources in this way, visual inspection can be performed using an illumination pattern suitable for the glossiness characteristics of the object.

[0034] However, when the object to be inspected is larger than a predetermined size, or when multiple objects to be inspected are inspected simultaneously, the gloss effect can reduce the inspection accuracy. Nevertheless, the above embodiment can be applied under any of these conditions.

[0035] Furthermore, when inspecting a single object smaller than a predetermined size, the problem of gloss affecting inspection accuracy is less likely to occur. Therefore, based on object information such as the size and number of objects to be inspected, it is possible to switch between an inspection mode that sets the lighting pattern based on the gloss mapping properties of the object, as in this embodiment, and an inspection mode that uses a fixed lighting pattern. Object information may be input by the user via the UI, or the size and number of objects may be identified using images obtained by the imaging device 203. It is possible to automatically switch the inspection mode depending on whether the size and number of objects to be inspected are greater than a predetermined threshold. Alternatively, the user may be able to select the inspection mode via the UI. The fixed lighting pattern can be a lighting pattern that widely uses both light sources with small and large zenith angles to improve inspection accuracy.

[0036] Furthermore, in this embodiment, the configuration allows for selection of either selection area 701 or selection area 702, but the UI in Figure 7 does not necessarily have to include either selection area 701 or selection area 702.

[0037] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0038] 204 Image Processing Device 304 Image Processing Unit 315 Lighting pattern setting section 317 Inspection Department

Claims

1. A setting means for selecting which light sources to illuminate from among multiple light sources arranged at different positions, based on the gloss mapping properties of the object to be inspected, Processing means for performing inspection processing on the object based on a plurality of images obtained by imaging the object illuminated by the aforementioned set light source, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that at least one of the zenith angle and azimuth angle of the plurality of light sources is different from each other.

3. The image processing apparatus according to claim 1, characterized in that the setting means is set to illuminate more light sources with a smaller zenith angle than when the object has a second glossy mapping property which is higher than the first glossy mapping property, when the object has a first glossy mapping property.

4. The image processing apparatus according to claim 1, characterized in that the setting means sets the light source to be illuminated based on the glossiness specified by the user via a user interface.

5. The image processing apparatus according to claim 1, characterized in that the setting means sets the light source to be illuminated based on the type of object specified by the user via a user interface.

6. The image processing apparatus according to claim 1, characterized in that the setting means sets the light source to be illuminated based on the correspondence between glossiness and light source identification information.

7. The image processing apparatus according to claim 1, further comprising imaging control means for performing control to synchronize the timing of the illumination of a light source with the timing of image capture.

8. The image processing apparatus according to claim 1, characterized in that the inspection process is a process for inspecting at least one of the shape and color of the surface of the object.

9. The image processing apparatus according to claim 1, characterized in that the processing means synthesizes the plurality of images by photometric stereo and performs the inspection process.

10. The image processing apparatus according to claim 1, further comprising a switching means that switches between a first mode in which a light source set by the setting means is turned on, and a second mode in which a predetermined light source is turned on, based on information about the object.

11. A program for causing a computer to function as an image processing device according to any one of claims 1 to 10.

12. Among multiple light sources positioned at different locations, the light sources to be illuminated are determined based on the gloss mapping properties of the object being inspected. Based on a plurality of images obtained by imaging the object illuminated by the aforementioned set light source, an inspection process is performed on the object. An image processing method characterized by the following:

Images