Photographing condition setting system, photographing condition setting method, and program

Abstract

To provide a photographing condition setting system that can sufficiently reduce influence of ambient light in design of an optimized illumination pattern in photographing a test object.SOLUTION: A photographing condition setting system comprises: a photographing device which photographs an image of a workpiece being a test object; an illuminating device which includes a light source irradiating the workpiece with light; and a photographing condition setting unit which sets a photographing condition used when photographing the workpiece. The photographing condition setting unit sets exposure time of the photographing device in such a manner that a pixel value of a designated area of the image of the workpiece is a value within a designated range, when performing photographing in such a way that intensity of light which is emitted from the illuminating device and with which the workpiece is irradiated, becomes equal to or more than a threshold value.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a photographing condition setting system, a photographing condition setting method, and a program. [Background technology]

[0002] Visual inspection of products at manufacturing sites is one of the areas where the replacement of humans with machines has progressed least, and is an important issue for automation that must be addressed in preparation for the future decline in the working population. In recent years, with the development of artificial intelligence and machine learning technologies, such as deep learning, inspection automation technology is improving dramatically.

[0003] However, in visual inspection and machine vision in general, the most time-consuming process in constructing an inspection system is the design of the imaging system, including the optimization design of the lighting pattern, and automation in this field has not progressed much. If this optimization design is done manually by a person, it requires a great deal of effort to design an imaging system that can reliably detect defects such as scratches on the workpiece while dealing with individual variations in the workpiece (inspection target). In other words, to obtain the desired detection performance, it is essential to alternate between optimizing the lighting by manually adjusting it and examining and adjusting the inspection algorithm while replacing various target workpieces, which requires a very large number of man-hours.

[0004] When actually photographing a workpiece for inspection, the light that hits the workpiece includes not only the illumination light but also the ambient light according to the shooting environment. However, since the effect of ambient light is not taken into account in the simulation for optimizing the design of the illumination pattern, an error occurs between the brightness of the image estimated by the simulation and the actual brightness of the image due to the luminance of the ambient light.

[0005] One method for dealing with this problem is to make the luminous intensity of the illumination light much greater than the luminous intensity of the ambient light, thereby relatively reducing the influence of the ambient light. However, when the luminous intensity is too high, there is a problem that pixels in the image tend to become saturated.

[0006] Regarding pixel saturation, Patent Document 1 describes that when there is a pixel in an image whose brightness value is saturated, multiple images are generated by changing the emission brightness of the illumination light source, and the sum of the brightness values ​​for each pixel is calculated to generate a composite image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6601264 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in conventional methods such as that described in Patent Document 1, in order to avoid pixel saturation, the luminous intensity of the illumination light irradiated onto the workpiece is less than the maximum luminous intensity that can be irradiated, so the relative influence of ambient light on the illumination light is not sufficiently small, and the error between the estimated image brightness and the actual image brightness cannot be sufficiently reduced.

[0009] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a shooting condition setting system that can sufficiently reduce the effects of ambient light in optimizing the design of an illumination pattern when photographing an object to be inspected. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the present invention employs the following configuration. A photographing condition setting system according to one aspect of the present invention comprises an imaging device which captures an image of a workpiece to be inspected, an illumination device including a light source which shines light on the workpiece, and an imaging condition setting unit which sets the conditions for photographing the workpiece, wherein the photographing condition setting unit sets the exposure time of the imaging device so that when the light intensity irradiated on the workpiece from the illumination device is set to a value above a threshold value, the pixel values ​​of a specified area of ​​the image of the workpiece are within a specified range. With the above configuration, when an image of an inspection object is captured, the illumination light is made sufficiently stronger than the ambient light, so that the influence of the ambient light can be relatively reduced.

[0011] The photographing condition setting unit may set an exposure time of the photographing device so that the pixel value of a designated area of ​​the image of the workpiece falls within a designated range when photographing the workpiece with the light intensity from the lighting device set to a maximum value. This makes it possible to sufficiently reduce the influence of the ambient light by making the illumination light as large as possible when photographing the inspection object.

[0012] The illumination device may include a plurality of light sources each having an adjustable luminous intensity, and the photographing condition setting unit may set the luminous intensity of at least one of the plurality of light sources to a value equal to or greater than a threshold value. This makes it possible to reduce the influence of ambient light while maintaining an optimal pattern of luminous intensity for each channel when multi-channel illumination is used.

[0013] Furthermore, when the characteristics of the pixel values ​​with respect to the exposure of the image capturing device are not linear, the image capturing condition setting unit may set the exposure time after performing calibration of the image capturing device, thereby making it possible to set the exposure time without depending on the characteristics of the image capturing device, even when the characteristics are unknown.

[0014] The photographing condition setting unit may set an exposure time of the photographing device based on a difference between an image photographed with the light intensity of the lighting device applied to the workpiece set to a threshold value or more and an image photographed with the light intensity of the lighting device applied to the workpiece set to a minimum value, so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range. This makes it possible to accurately set the exposure time based on an image that is irradiated only with light from the lighting device.

[0015] The photographing condition setting unit may set an exposure time of the photographing device based on a difference between an image photographed with the luminous intensity of the lighting device on the workpiece set to a threshold value or more and an image photographed with the luminous intensity of the lighting device on the workpiece set to a minimum value under a plurality of different ambient light conditions, so that a pixel value of a designated area of ​​the image of the workpiece is within a designated range. This makes it possible to set an exposure time that is robust to the ambient light conditions at the time of photographing. The photographing condition setting unit may set an exposure time of the photographing device so that the pixel value of a designated area of ​​the image of the workpiece falls within a designated range when photographing is performed with at least one of the set value of the luminous intensity irradiated from the lighting device to the workpiece, the measured value of the actually irradiated luminous intensity, and the luminous intensity displayed on the display set to a value equal to or greater than a predetermined threshold value. This allows the exposure time to be set based on any one of the set value, the measured value, and the displayed value on the display device according to the system configuration.

[0016] A photographing condition setting method according to one aspect of the present invention is a photographing condition setting method for determining photographing conditions for an image of a workpiece to be inspected by a computer, in which the computer sets the luminous intensity of an illumination device that shines light on the workpiece to a value equal to or greater than a threshold value, the computer photographs the workpiece using an imaging device under the luminous intensity conditions, and the computer sets the exposure time of the imaging device based on the photographed image of the workpiece so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range. With the above configuration, when photographing an inspection object, the influence of the environmental light can be relatively reduced by making the illumination light sufficiently larger than the environmental light.

[0017] A program according to one aspect of the present invention causes a computer that determines conditions for photographing an image of a workpiece to be inspected to execute the following: setting the luminous intensity of an illumination device that irradiates light on the workpiece to a value equal to or greater than a threshold value, photographing the workpiece using an imaging device under the conditions of the luminous intensity, and setting the exposure time of the imaging device based on the photographed image of the workpiece so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range. With the above configuration, when photographing an object to be inspected, the influence of the environmental light can be relatively reduced by making the illumination light sufficiently larger than the environmental light. Effect of the Invention

[0018] According to the present invention, it is possible to provide an imaging condition setting system capable of sufficiently reducing the influence of ambient light in the optimization design of an illumination pattern when imaging an object to be inspected. [Brief description of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating a configuration example of an inspection system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a plan view illustrating an example of an arrangement of light sources used in the inspection system according to the embodiment of the present invention. [Diagram 3] 2 is a diagram illustrating a hardware configuration of a control device included in the inspection system according to the embodiment of the present invention. FIG. [Figure 4] FIG. 1 is a diagram illustrating an example of a functional configuration of an inspection system according to an embodiment of the present invention. [Diagram 5] FIG. 1 is a schematic diagram showing an image synthesis model assuming that linearity is established in single-shot inspection using multi-channel illumination according to the present invention. [Figure 6] 4 is a graph showing CRF functions when the relationship between luminous intensity and pixel value is linear and when it is nonlinear according to the present invention. [Figure 7] 1 is a flowchart showing a procedure from determining illumination conditions of an inspection system according to an embodiment of the present invention to inspecting a workpiece under the determined illumination conditions. [Figure 8] 1 is a flowchart showing a procedure from determining illumination conditions of an inspection system according to an embodiment of the present invention to inspecting a workpiece under the determined illumination conditions. [Figure 9] 1 is a flowchart showing a procedure from determining illumination conditions of an inspection system according to an embodiment of the present invention to inspecting a workpiece under the determined illumination conditions. [Figure 10] 1 is a flowchart showing a procedure from determining illumination conditions of an inspection system according to an embodiment of the present invention to inspecting a workpiece under the determined illumination conditions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while the data appearing in this embodiment is described in natural language, more specifically, it is specified in pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0021] §1 Examples of application An example of a situation in which the present invention is applied will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of an inspection system (photographing condition setting system) 1 according to the present invention. Fig. 2 is a plan view showing an example of the arrangement of light sources used in the inspection system 1.

[0022] The inspection system 1 performs an appearance inspection of the workpiece 4, for example, by executing an image analysis process on an input image obtained by capturing an image of the workpiece 4, which is an inspection target, transported on the belt conveyor 2. In the following, as a typical example of the image analysis process, an application example such as an inspection for the presence or absence of defects on the surface of the workpiece 4 will be described, but the application is not limited to this and can also be applied to, for example, identifying the type of defect and measuring the appearance shape.

[0023] A camera 102 as a sensor integrated with a light source LS is disposed above the belt conveyor 2, and the imaging field of view 6 of the camera 102 is configured to include a predetermined area of ​​the belt conveyor 2. Here, the light source LS may be, for example, a multi-channel illuminator such as MDMC (Multi Direction Multi Color) illumination, and more specifically, the illuminator described in, for example, Japanese Patent Application No. 2018-031747 by the present applicant may be exemplified. The light source LS as a multi-channel illuminator has a plurality of channel illuminators LSi. More specifically, the light source LS has, for example, one channel illuminator LSi that is circular in plan view, and 12 channel illuminators LSi that are concentrically arranged around the one channel illuminator LSi and form a fan-shaped strip. In this case, the light source LS is composed of a total of 13 channel illuminators LSi, and when each channel illuminator LSi emits three colors, the default illumination emission pattern of the single color and single channel is 13×3=39 patterns. Furthermore, the data of the evaluation workpiece image generated by the imaging of the camera 102 is transmitted to the control device 100. The imaging by the camera 102 is performed periodically or on an event basis.

[0024] The control device 100 causes the light source LS to emit light in a plurality of evaluation light emission patterns based on the illumination command value, which is a discrete value, and captures a plurality of evaluation workpiece images of at least one workpiece illuminated with the plurality of evaluation light emission patterns. The control device 100 also simulates the light emission pattern by varying a parameter of a continuous value based on the plurality of evaluation workpiece images, and calculates a parameter representing the light emission pattern for identifying the state of the workpiece. Here, the parameter is a weight parameter for generating a composite image by superimposing the plurality of evaluation workpiece images. Furthermore, the control device 100 converts the parameter into an illumination command value based on the illuminance LUT.

[0025] The control device 100 captures an inspection image of the workpiece 4 to be inspected using the calculated illumination command value. The control device 100 may also include a learning device having a CNN (convolutional neural network) engine for visual inspection of the workpiece 4. A feature detection image for each class is generated from the inspection image using the CNN engine. Based on the generated single or multiple feature detection images, the state of the workpiece 4 to be inspected (presence or absence of a defect, the size of the defect, the position of the defect, etc.) is identified.

[0026] The control device 100 is connected to a PLC (programmable controller) 10, a database device 12, and the like via a higher-level network 8. The calculation results and detection results in the control device 100 may be transmitted to the PLC 10 and / or the database device 12. Any device may be connected to the higher-level network 8 in addition to the PLC 10 and the database device 12. The control device 100 may also be connected to a display 104 as an output unit for displaying the processing status, detection results, and the like, and, for example, a keyboard 106 and a mouse 108 as an input unit for accepting operations by a user.

[0027] In the present disclosure, the inspection system 1 illuminates a workpiece (inspection target) to be inspected with an inspection light emission pattern based on an illumination command value that is a discrete value, captures an image of the workpiece with an appropriate sensor, and acquires an inspection workpiece image. Then, the inspection workpiece image is subjected to image processing to identify the state of the workpiece (for example, the presence or absence of defects such as scratches).

[0028] In order to determine an illumination pattern optimized for inspecting a workpiece, the inspection system 1 captures a plurality of evaluation workpiece images of at least one workpiece in advance. The inspection system then generates a composite image by superimposing the plurality of evaluation workpiece images weighted by continuous-value parameters, and calculates parameters representing an illumination pattern suitable for identifying the state of the workpiece by evaluating the composite image by a predetermined evaluation function. The inspection system 1 converts the parameters into illumination command values ​​based on a table that associates illumination command values ​​with parameters.

[0029] Furthermore, the inspection system 1 eliminates the influence of ambient light in order to improve the accuracy of matching between the image synthesis result and the image of the workpiece captured under actual inspection conditions. Specifically, the luminous intensity of each channel illumination LSi of the light source LS is controlled to match the optimized light emission pattern, while controlling the luminous intensity of at least one channel illumination LSi to be maximum. Furthermore, when illuminating with maximum luminous intensity, the influence of ambient light can be relatively suppressed, but pixel saturation may occur in part of the captured image. For this reason, the inspection system 1 adjusts the exposure time or gain of the camera 102 with the luminous intensity of the illumination set to maximum, and searches for the optimal exposure time that does not cause pixel saturation.

[0030] §2 Configuration Example (1. Hardware Configuration) Next, an example of a hardware configuration of the control device 100 included in the inspection system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a diagram illustrating a schematic diagram of the hardware configuration of the control device 100.

[0031] The control device 100 may be realized, for example, by using a general-purpose computer configured according to a general-purpose computer architecture. The control device 100 includes a processor 110, a main memory 112, a camera interface 114, an input interface 116, a display interface 118, a communication interface 120, and a storage 130. These components are typically connected to each other via an internal bus 122 so as to be able to communicate with each other.

[0032] The processor 110 realizes functions and processes described below by expanding various programs stored in the storage 130 into the main memory 112 and executing them. The main memory 112 is composed of a volatile memory, and functions as a work memory required for the processor 110 to execute the programs.

[0033] The camera interface 114 is connected to the camera 102 and acquires an evaluation workpiece image 138 and an inspection workpiece image 140 captured by the camera 102. The camera interface 114 may instruct the camera 102 on the timing of capturing images and the like.

[0034] The input interface 116 is connected to input units such as the keyboard 106 and the mouse 108, and acquires commands indicating operations performed by the user on the input units.

[0035] The display interface 118 outputs various processing results generated by the processor 110 executing the programs to the display 104 .

[0036] The communication interface 120 is responsible for processing for communicating with the PLC 10, the database device 12, and the like via the higher level network 8.

[0037] The storage 130 stores programs such as an OS (Operating System) and an inspection program 132 for causing the computer to function as the control device 100. The storage 130 may further store lighting parameters 134, an illuminance LUT 136, a plurality of evaluation workpiece images 138, and a plurality of inspection workpiece images 140. The lighting parameters 134 are continuous-value parameters having a dimension equal to the number of the evaluation workpiece images, and are weighting parameters for generating a composite image by superimposing the plurality of evaluation workpiece images 138. The illuminance LUT 136 is a table that associates lighting command values ​​with illuminance.

[0038] The inspection program 132 stored in the storage 130 may be installed in the control device 100 via an optical recording medium such as a DVD (Digital Versatile Disc) or a semiconductor recording medium such as a USB (Universal Serial Bus) memory. Alternatively, the inspection program 132 may be downloaded from a server device or the like on a network.

[0039] When realizing the program using a general-purpose computer in this manner, a part of the functions according to the present embodiment may be realized by calling and processing necessary software modules among the software modules provided by the OS in a predetermined order and / or timing. That is, the inspection program 132 according to the present embodiment does not include all the software modules for realizing the functions according to the present embodiment, and the necessary functions may be provided by cooperating with the OS.

[0040] Furthermore, the inspection program 132 may be provided by being incorporated into a part of another program. In that case, the inspection program 132 itself does not include the modules included in the other programs to be combined as described above, and executes processing in cooperation with the other programs. In this way, the inspection program 132 according to this embodiment may be in a form incorporated into another program.

[0041] 3 shows an example in which the control device 100 is realized using a general-purpose computer, but the present invention is not limited to this, and all or part of the functions may be realized using a dedicated circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)). Furthermore, part of the processing may be assigned to an external device connected to a network.

[0042] As described above, from the viewpoint of hardware configuration, the inspection system 1 corresponds to an example of the "inspection system 1" in the present disclosure. The workpiece 4 corresponds to an example of the "workpiece" in the present disclosure. Furthermore, the light source LS corresponds to an example of the "light source" in the present disclosure, and the camera 102 corresponds to an example of the "camera" in the present disclosure.

[0043] (2. Functional Configuration) Next, an example of a functional configuration of the inspection system 1 according to the embodiment of the present disclosure will be described with reference to Fig. 4. Fig. 4 is a diagram illustrating an example of a functional configuration of the inspection system 1 according to the embodiment of the present disclosure. The control device 100 of the inspection system 1 can include an imaging unit 141, a calculation unit 142, a conversion unit 143, and a storage unit 144.

[0044] The imaging unit 141, the calculation unit 142, and the conversion unit 143 in the control device 100 can be realized by a general-purpose processor, and are not limited in this disclosure, and all or part of the functions of these components may be realized by a dedicated circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)). Furthermore, part of the processing may be assigned to an external device connected to a network.

[0045] The imaging unit 141 causes the light source LS to emit light in a plurality of evaluation light emission patterns based on the illumination command value, which is a discrete value, and captures a plurality of evaluation workpiece images for at least one workpiece illuminated with the plurality of evaluation light emission patterns by the camera 102. The imaging unit 141 may capture N evaluation workpiece images.

[0046] The calculation unit 142 simulates a light emission pattern by varying a parameter of a continuous value based on the multiple evaluation workpiece images, and calculates a parameter representing a light emission pattern for identifying the state of the workpiece. The calculation unit 142 generates a composite image by superimposing the multiple evaluation workpiece images weighted by a parameter having a dimension equal to the number of the multiple evaluation workpiece images, and calculates a parameter representing a light emission pattern for identifying the state of the workpiece by evaluating the composite image by a predetermined evaluation function.

[0047] The conversion unit 143 converts the calculated parameters into illumination command values. The conversion unit 143 converts the parameters into illumination command values ​​based on an illuminance LUT that associates illumination command values ​​with parameters.

[0048] The memory unit 144 stores a plurality of evaluation workpiece images obtained by the imaging unit 141 in the illumination optimization stage, and stores inspection workpiece images obtained by the imaging unit 141 in the inspection stage. The memory unit 144 also stores the illumination parameters obtained by the calculation unit 142. Furthermore, the memory unit 144 stores an illuminance LUT used by the conversion unit 143. The memory unit 144 is realized by the above-mentioned storage 130 that stores programs or data necessary for the operation of the inspection system 1. Note that the control device 100 does not need to include the memory unit 144, and an external (device) storage may be used instead of the memory unit 144.

[0049] As described above, from the viewpoint of functional modules, the control device 100, together with the light source LS and the camera 102, corresponds to an example of an "imaging unit" in the present disclosure. In addition, the control device 100 functions as an example of a "calculation unit" and a "conversion unit" in the present disclosure.

[0050] §3 Example of operation In the following explanation of the operation example, the meaning of each symbol used in the formula is as shown in Table 1. Bold lowercase letters indicate vectors, and bold uppercase letters indicate matrices (in the text, they are shown in non-bold). All other symbols are scalars. In addition, ∥∥ indicates the L2 norm for a vector, [X] i,j Let denote the element in the i-th row and j-th column of the matrix X.

[0051] [Table 1]

[0052] (Equivalence of multi-channel lighting photography and dot product calculation) In the inspection system 1, an imaging unit 141 using a multi-channel illumination as a light source LS is used, and here, it is assumed that linearity (linearity between luminance and QL value) is established on the sensor side in the inspection system 1. This means that linearity is established in a system including all of the imaging corrections such as signal processing such as color filter demosaicing, gamma correction, and dark current offset correction. In order to avoid nonlinearity due to pixel saturation, HDR (High Dynamic Range) synthesis, etc. may be used for capturing images. In this case, considering a situation in which an image of a workpiece 4 illuminated by K channel illuminations LSi is captured, the emission intensity of each channel illumination is expressed as a vector of the following formula (1).

[0053]

number

[0054] At this time, u i =1,u j = 0 (j ≠ i), only the i-th light is turned on at a specified intensity, and the others are turned off. The images are arranged as a column vector and f i Then, an image g captured under any lighting condition u can be modeled by image synthesis as shown in the following equation (2).

[0055]

number

[0056] Here, f′ is a multi-channel image (one large column vector) consisting of K images arranged vertically, as defined by the following equation (3).

[0057]

number

[0058] The operators in the above formulas (2) and (3) are defined as follows:

number

[0059] In this way, the optimization of multi-channel lighting is achieved by generating a feature vector g by performing an inner product operation on the original image f′, that is, the projection matrix u T This is equivalent to optimally designing the linear projection direction u in ×I (where “×” is the Kronecker product).

[0060] (Acquisition of evaluation images using the default lighting pattern) In what follows, we will generally consider a situation in which K lighting fixtures are installed, in which the lighting emission pattern of one work is changed to capture N patterns of images, and the optimal lighting is determined (teaching).

[0061] The light emission pattern in capturing the nth image (1≦n≦N) is defined as in the following formula (5).

[0062]

number

[0063] By photographing the workpiece with these illumination patterns, it is possible to reconstruct the photographed image of the workpiece when illuminated with any illumination pattern. This is equivalent to estimating the LT (Light Transport) matrix. In general, to obtain all the degrees of freedom of the LT matrix, n It is desirable to determine the light emission pattern so that the degrees of freedom of the K lights are linearly independent. In order to fully utilize all of the K lights, at least N=K, and in that case, the rank of H, rank(H)=min(N,K), must be full rank.

[0064] On the one hand, when N < K, the degree of freedom in the number of illuminations cannot be fully utilized, but it is an effective method for shortening the shooting time of the workpiece. Various methods can be considered as means to achieve this, and a typical method is to perform LT matrix estimation using Compressed Sensing.

[0065] Also, the case of N > K can be considered. This is excessive in terms of fully utilizing the degree of freedom in illumination and results in a wasteful number of shooting images, but it can be selected for other purposes such as improving the signal-to-noise ratio or dynamic range.

[0066] Let f be the arrangement of the images taken with these light emission patterns as column vectors. i Furthermore, define the multi-channel image vector obtained by arranging these vertically for N more as follows in Equation (6) below. Here, e i represents the standard basis of R N .

[0067]

Equation

[0068] The luminance u for each illumination channel and the captured image g at that time are expressed as in Equation (7) below.

[0069]

Equation

[0070] Here, w i is the image synthesis weight (illumination parameter). Note that the command value for the light emission intensity is calculated based on the image synthesis weight, but it is not this value itself. Although the image synthesis weight is generally allowed to take negative values, since the light emission intensity of the illumination channel must be non-negative, the condition in Equation (8) below is imposed.

[0071]

Equation

[0072] Here, inequality signs for vectors and matrices are assumed to apply to all elements. When H=I, that is, when K(=N) images are taken with each light turned on, u and w are equivalent. Below, we address the problem of finding the optimal lighting u by finding the optimal image synthesis weight w.

[0073] (Lighting optimization criteria) Here, we will outline the significance of lighting optimization and its evaluation criteria, and then explain the specific processing procedures with reference to drawings, etc. First, the purpose of lighting design in the visual inspection of workpieces, etc. is to correctly distinguish between good (normal) and bad (defective) products. This problem has traditionally been solved manually by skilled workers, but if one tries to solve it systematically, it is necessary to determine and limit the "judgment criteria" and "degree of control freedom in lighting design" in advance.

[0074] Here, when a predefined discrimination algorithm is given as the judgment criterion and multi-channel lighting such as MDMC lighting is given as the control degree of freedom, the lighting optimization problem is formulated as a "cross-entropy minimization problem" that matches the categorization of good / bad products with the correct answer. If the judge for the work label judgment is a machine learning machine, it is possible to simultaneously perform the learning of the lighting optimization and the judge, and automatically tune them to maximize their performance.

[0075] However, a major problem when trying to minimize cross-entropy is that a large number of samples with pass / fail labels are required. In particular, when trying to optimize lighting with a large degree of freedom, it is difficult to uniquely determine the optimal lighting with a standard that only distinguishes a small number of pass / fail labels. This problem is particularly worrisome and a major challenge when starting up an inspection system where many evaluation workpiece images (sample images) cannot be obtained.

[0076] In this disclosure, in order to solve this problem, we propose a method to optimize lighting based on the evaluation criteria of the image itself (contrast, brightness, closeness). In this case, the requirements for optimized lighting design in the appearance inspection of workpieces, etc. can be roughly classified into the following two points.

[0077] [Requirement 1] Make the features that make it easier to distinguish between good and bad products (labels) visible (i.e. make defects visible) [Requirement 2] Making it difficult to see quality variation (individual variation)

[0078] However, since these two requirements are generally contradictory, it is difficult to design lighting that balances them. In the following, in order to find a solution under constraints that make the light emission intensity of the lighting a positive value using evaluation criteria that satisfy these conditions, the evaluation criteria are considered to be discriminant analysis, and the image evaluation criteria are expressed in a quadratic form. In that case, this problem can be solved using SDP (Semidefinite Programming).

[0079] Regardless of which evaluation criterion is used, the evaluation function for lighting optimization can be expressed as the minimization of a function F with M workpiece images as input, as shown in the following formula (9). i This is the image synthesis weight (lighting parameter) expressed as: This means the luminance of the i-th base light-emitting pattern, and is originally a discrete quantity due to the fact that the lighting command value is discrete, but here it is treated as a continuous quantity by relaxing the continuity.

[0080]

number

[0081] FIG. 5 is a schematic diagram showing an image synthesis model assuming that linearity is established in single-shot inspection using multi-channel illumination according to the present disclosure.

[0082] This optimization problem can be efficiently performed because it can be searched for in a continuous quantity with fewer dimensions, w, instead of u. In what follows, we consider the optimal image synthesis weight w opt is written as w.

[0083] (How to adjust exposure time) Here, we will explain AEC (Auto Exposure Control) photography and HDR (High Dynamic Range) photography as examples of methods for adjusting the exposure time. In both photography methods, the exposure time t is set so that pixel saturation within the Region of Interest (ROI) of the image can be avoided when the base light emission pattern is irradiated onto the workpiece. n The exposure time is t t The image taken in is used as the base image.

[0084] In AEC photography, the first step is to determine the target brightness (target pixel value y t ) and take an image at any exposure time, and then take an image of the ROI S n Extract this image S n The maximum pixel value y m (pixel value of interest) to the target pixel value y t The exposure time t required to convert t Calculate the target pixel value y t is the threshold value y at which pixels become black min and the threshold y max The value is set to a value between .

[0085] Target pixel value y t The exposure time t required to convert t is calculated based on the camera's CRF (Camera Response Function). Note that the CRF is assumed to be a known value. The CRF is a function that represents the relationship between luminance and pixel value. When the CRF function f(i,t) is used, the pixel value y is expressed by the following general formula: y=f(i,t)+d Here, i represents the illuminance, t represents the exposure time, and d represents the dark current offset.

[0086] The CRF function f(i,t) can be expressed as f(i)*t or a*i*t (where a is a coefficient representing weighting) when the luminance is linear with respect to the exposure time, as a characteristic of the camera, or as f(i*t), i*f(t), or f1(i)*f2(t) when the luminance is non-linear with respect to the exposure time.

[0087] The graph in Figure 6 shows the CRF function f when the relationship between luminous intensity I and pixel value y is linear and when it is non-linear. CRF (i,t), where y m is the pixel value, y t is the target pixel value, t m is the exposure time of the pixel of interest, t t represents the exposure time to obtain the target pixel value.

[0088] Next, the exposure time t t The procedure for obtaining the pixel value y m Then, the luminosity I is calculated by the following formula: m Calculate. I m =f -1 CRF (y m -d)

[0089] Next, the target pixel value y t Then, the luminosity I is calculated by the following formula: t Calculate. I t =f -1 CRF (y t -d)

[0090] Next, t t =I t / I m Calculate. t t can be expressed by the following formula: (In the case of linear motion) (y t -d) / (y m -d)·tm

[0091] (In the case of non-linear) f -1 CRF (y t -d)·t m / f -1 CRF (y m -d)

[0092] Here, the calculation is performed assuming that d (the pixel value when the luminance is zero) is a constant value. If the above assumption is no longer valid, a correction process must be introduced.

[0093] In the AEC method, images are repeatedly taken at any exposure time, and the maximum pixel value (pixel of interest) y m is the target pixel value y t The exposure time t t Get the.

[0094] In HDR photography, the image is first captured with the shortest exposure time, and the pixel value y of the pixel with the maximum pixel value in the image is calculated. m is the target pixel value y t The exposure time t t Next, the same process is repeated for the remaining pixels, and the target pixel value y t The exposure time t t Calculate.

[0095] (General formula for image synthesis when exposure time is variable) Next, the exposure time τ of the image to be synthesized and the exposure time τ of the base image taken are n The method of image synthesis when f is different will be described. The calculation formula for image synthesis in this case is shown in the following formula (10). The following formula is a synthesis formula for n (1≦n≦N) captured images in the base emission pattern, and f n represents the image captured in the nth base emission pattern. n denotes the composition weight of each image. τ is an arbitrary exposure time, and τ nrepresents the exposure time when photographing the workpiece in the nth base emission pattern.

[0096]

number

[0097] The difference from the case where the exposure time is constant is that the base image is converted to luminance for a unit exposure time, and an image at an arbitrary exposure time τ is reproduced. Note that here, we assume that the camera's CRF is linear. If the CRF is not linear, a correction process must be introduced.

[0098] (Luminance correction based on ambient light image) One of the causes of errors in the brightness of an image actually captured and the brightness of an image synthesized from a base image is the influence of ambient light. In order to reduce the influence of ambient light, in this embodiment, the luminous intensity of the lighting is increased so that the influence of ambient light becomes relatively small. Furthermore, the accuracy is further improved by correcting the luminous intensity of the ambient light mixed in the base image based on an image of the workpiece captured with the lights turned off (ambient light image). The calculation formula for image synthesis that takes into account the luminous intensity correction based on the ambient light image is shown in the following formula (11). n (1≦n≦N) represents the number of captured images in the base emission pattern, and f n represents the image captured in the nth base emission pattern. n is the composition weight of each image, τ is an arbitrary exposure time, and τ n represents the exposure time when photographing the workpiece in the nth base emission pattern.

[0099]

number

[0100] The idea behind the luminance correction in equation (11) is to convert the basis image and the ambient light image into a common index called luminance, and then reconstruct the image. nis the ambient light image converted to the exposure time when each base image was taken, and iτ is the ambient light image converted to the exposure time of the image to be synthesized. These are expressed as the following formula (12) based on the reference ambient light image z. In the following formula, f z is the image captured when all the lighting channels are set to the minimum luminance, and τ z indicates the exposure time when shooting in ambient light.

[0101]

number

[0102] Next, a procedure from determining the illumination conditions (teaching phase) by the inspection system 1 of this embodiment to inspecting a workpiece under the determined illumination conditions (inspection phase) will be described with reference to the flowchart of FIG.

[0103] As shown in FIG. 7, first, in the teaching phase, the inspection system 1 calculates the maximum luminous intensity vector T max Illuminate M workpieces with a luminance according to the following, and take N × M evaluation images f n,m At this time, the inspection system 1 captures each image f n,m The exposure time τ is set so that pixel saturation does not occur within the ROI. n,m Here, f n,m The exposure times are τ n,m This is expressed as:

[0104] Next, the inspection system 1 executes determination of the illumination condition (S102). The inspection system 1 receives an input of an evaluation formula F for optimizing illumination from a user, and outputs N×M captured images f for evaluation. n,m and each image f n,m Exposure time τ n,m is evaluated using the evaluation formula F to determine the optimal lighting condition vector T and exposure time τ.

[0105] Next, in the inspection phase, the inspection system 1 captures an inspection image of the workpiece to be inspected (S103). The inspection system 1 illuminates the workpiece using the illumination condition vector T and exposure time τ calculated in the teaching phase, and captures an inspection image g.

[0106] Furthermore, the inspection system 1 inputs the captured inspection image to an inspection decision device (for example, a decision device using CNN or the like) and outputs a decision result based on the inspection parameter P (S104).

[0107] If the CRF of the camera is not linear, then as shown in the flowchart of FIG. 8, camera calibration is performed at the beginning of the teaching phase (S100), and then images for evaluation are captured.

[0108] In addition, in the flowchart shown in FIG. 9, in the teaching phase, the influence of ambient light is eliminated with greater precision by taking the difference between an image captured under maximum lighting conditions and an image captured under minimum lighting conditions (ambient light image).

[0109] Specifically, in step S101, N×M evaluation images f n,m After capturing the image, the camera captures the image using only ambient light (S201). min (For example, in a light-off state) z The exposure time is set to τ so that pixel saturation does not occur. z It is held.

[0110] Next, the image f taken at maximum light intensity in step S101 is n,m and the ambient light image f z The difference image f' n,m is generated (S202).

[0111] Next, the inspection system 1 determines the illumination conditions (S203). The inspection system 1 accepts an input of an evaluation formula F for optimizing illumination from a user, and calculates N×M difference images f′ acquired in step S202. n,m is evaluated using the evaluation formula F to determine the optimal lighting condition vector T and exposure time τ.

[0112] (Tuning lighting conditions and inspection parameters using multiple ambient light images) In the flowchart shown in FIG. 9, an ambient light image is captured under constant ambient light conditions, whereas in the flowchart shown in FIG. 10, an ambient light image is captured assuming that ambient light conditions change (for example, changes in sunlight conditions over time).

[0113] Specifically, first, in step S101, a photographed image f for evaluation at maximum light intensity is taken for M workpieces. n,m Then, for each workpiece, the minimum light intensity vector T min Under the conditions, i ambient light images f i,d The exposure time of the ambient light image is set to an exposure time τ i,d It is held.

[0114] Next, the image f taken at maximum light intensity in step S101 is n,m and the ambient light image f i,d The difference image f' i,n,m (S302). I differential images are generated for each workpiece.

[0115] Next, the inspection system 1 determines at least one of the illumination conditions and the inspection parameters (S303). The inspection system 1 receives an evaluation formula F for optimizing illumination and an inspection algorithm A, and calculates the i×N×M difference images f′ obtained in step S302. i,n,m is evaluated using evaluation formula F and inspection algorithm A, and the optimal lighting condition vector T, exposure time τ, and inspection parameter P are determined.

[0116] In this way, by determining the illumination conditions for the same workpiece using multiple ambient light images with different ambient light conditions, it is possible to calculate illumination conditions that are robust to changes in ambient light. In addition, by using multiple ambient light images, it is also possible to search for inspection parameters that are robust to changes in ambient light.

[0117] In the above example, the lighting conditions and inspection parameters are obtained by directly using the multiple ambient light images, but it is also possible to generate multiple ambient light images by numerical calculation from an ambient light model (for example, a model in which the image vector x follows a multidimensional normal distribution N(μ,Σ)) and obtain lighting conditions and inspection parameters that are robust to ambient light. Here, μ=E[x] is a vector that represents the average value of each pixel value of the multiple ambient light images, and Σ=E[(x-μ)(x-μ) T ] is the variance-covariance matrix between each pixel of multiple ambient light images.

[0118] As described above, according to this embodiment, when photographing a workpiece, the luminance of the light from the lighting is set to the maximum and the exposure of the camera is adjusted to avoid pixel saturation. This makes it possible to relatively sufficiently reduce the influence of the ambient light by maximizing the illumination light.

[0119] In addition, if the camera's CRF is not linear, the camera is calibrated and the exposure time when the lighting is at maximum brightness is determined. This makes it possible to adjust the exposure time without relying on the camera's characteristics, even if the CRF is unknown.

[0120] In addition, when optimizing the exposure time, the difference between the image taken with the lighting at maximum brightness and the image taken with the lighting at minimum brightness is used, allowing the exposure to be adjusted precisely based on the image when only the light from the lighting is applied.

[0121] In addition, when optimizing the exposure time, the system uses the difference between an image taken with the lighting at maximum brightness and an image taken with the lighting at minimum brightness under multiple different ambient light conditions, allowing for robust exposure adjustments to suit the ambient light conditions at the time of shooting.

[0122] In the above embodiment, the light from the lighting is set to maximum intensity and the exposure or gain of the camera is adjusted, but as long as the intensity is sufficiently stronger than the ambient light, it is not necessarily limited to maximum intensity and light of a certain intensity or higher may be applied.

[0123] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. Note that a part or all of the above embodiment can be described as in the following supplementary notes, but is not limited to the following. (Appendix 1) an imaging device (102) for capturing an image of a workpiece (4) to be inspected; A lighting device (LS) including a light source that irradiates light onto the work (4); and a photographing condition setting unit (100) for setting photographing conditions when photographing the work (4), The imaging condition setting unit (100) The photographing condition setting system (1) sets the exposure time of the photographing device (102) so that, when the light intensity irradiated from the lighting device (LS) on the work (4) is set to a threshold value or higher and an image is taken, the pixel values ​​of a specified area of ​​the image of the work (4) are within a specified range.

[0124] (Appendix 2) The imaging condition setting unit (100) The photographing condition setting system (1) described in Appendix 1 sets an exposure time of the photographing device (102) so that, when photographing is performed with the light intensity irradiated from the lighting device (LS) on the workpiece (4) at maximum value, the pixel values ​​of a designated area of ​​the image of the workpiece (4) are within a designated range.

[0125] (Appendix 3) The lighting device (LS) A plurality of light sources, each of which can adjust its luminosity, The imaging condition setting unit (100) The photographing condition setting system (1) according to appendix 1 or 2, wherein the luminous intensity of at least one of the plurality of light sources is set to a value equal to or greater than a threshold value.

[0126] (Appendix 4) The imaging condition setting unit includes: The photographing condition setting system (1) according to any one of appendices 1 to 3, wherein, when a characteristic of a pixel value relative to an exposure of the photographing device is not in a linear relationship, the exposure time is set after performing calibration of the photographing device.

[0127] (Appendix 5) The imaging condition setting unit (100) The photographing condition setting system (1) described in any one of Appendices 1 to 4, wherein an exposure time of the photographing device (102) is set so that pixel values ​​of a designated area of ​​the image of the workpiece (4) are within a designated range based on a difference between an image photographed with the light intensity irradiated from the lighting device (LS) on the workpiece (4) set to a threshold value or higher and an image photographed with the light intensity irradiated from the lighting device (LS) on the workpiece (4) set to a minimum value.

[0128] (Appendix 6) The imaging condition setting unit (100) The photographing condition setting system (1) described in any one of Appendices 1 to 4, which sets an exposure time of the photographing device (102) so that pixel values ​​of a specified area of ​​the image of the workpiece (4) are within a specified range based on a difference between an image photographed with the light intensity irradiated from the lighting device (LS) on the workpiece (4) set to a threshold value or higher and an image photographed with the light intensity irradiated from the lighting device (LS) on the workpiece (4) set to a minimum value under a plurality of different ambient light conditions. (Appendix 7) The imaging condition setting unit (100) The photographing condition setting system (1) described in any one of Appendices 1 to 6, wherein an exposure time of the photographing device (102) is set so that when at least one of the set value of the luminous intensity irradiated from the lighting device (LS) on the workpiece (4), the measured value of the actual luminous intensity, and the luminous intensity displayed on a display is set to a value equal to or higher than a predetermined threshold value, the pixel value of a specified area of ​​the image of the workpiece is within a specified range.

[0129] (Appendix 8) A method for setting photographing conditions for an image of a workpiece (4) to be inspected by a computer (1), comprising: A computer (1) sets the luminous intensity of a lighting device (LS) that irradiates light onto the work (4) to a value equal to or greater than a threshold value; The computer (1) photographs the work (4) using the photographing device (102) under the above-mentioned light intensity conditions; A photographing condition setting method in which a computer (1) sets an exposure time of the photographing device (102) based on the photographed image of the work (4) so ​​that pixel values ​​of a specified area of ​​the image of the work (4) are within a specified range.

[0130] (Appendix 9) A computer (1) determines conditions for taking an image of a workpiece (4) to be inspected, Setting the luminous intensity of a lighting device (LS) that irradiates light onto the work (4) to a value equal to or greater than a threshold value; Photographing the work (4) with an imaging device (102) under the light intensity conditions; A program that executes setting an exposure time of the photographing device (102) based on the photographed image of the work (4) so ​​that the pixel values ​​of a specified area of ​​the image of the work (4) are within a specified range. [Explanation of symbols]

[0131] 1...inspection system, 2...belt conveyor, 4...workpiece, 6...imaging field of view, 8...upper network, 10...PLC, 12...database device, 100...control device, 102...camera, 104...display, 106...keyboard, 108...mouse, 110...processor, 112...main memory, 114...camera interface, 116...input interface, 118...display interface, 120...communication interface, 122...internal bus, 130...storage, 132...inspection program, 134...lighting parameters, 136...illuminance LUT, 138...evaluation workpiece image, 140...inspection workpiece image, 141...imaging unit, 142...calculation unit, 143...conversion unit, 144...storage unit, LS...light source, LSi...channel lighting

Claims

1. An imaging device that captures an image of a workpiece to be inspected; An illumination device including a light source that irradiates light onto the workpiece; A photographing condition setting unit that sets photographing conditions when photographing the workpiece, The imaging condition setting unit includes: When the light intensity applied from the lighting device to the workpiece is set to a value equal to or greater than a threshold value and the workpiece is photographed, an exposure time of the photographing device is set so that the pixel value of a designated area of ​​the image of the workpiece falls within a designated range. The imaging condition setting unit includes: Based on a difference between an image captured by setting the light intensity applied from the lighting device to the workpiece at a threshold value or more and an image captured by setting the light intensity applied from the lighting device to the workpiece at a minimum value, an exposure time of the photographing device is set so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range. Shooting condition setting system.

2. An imaging device that captures an image of a workpiece to be inspected; An illumination device including a light source that irradiates light onto the workpiece; A photographing condition setting unit that sets photographing conditions when photographing the workpiece, The imaging condition setting unit includes: When the light intensity applied from the lighting device to the workpiece is set to a value equal to or greater than a threshold value and the workpiece is photographed, an exposure time of the photographing device is set so that the pixel value of a designated area of ​​the image of the workpiece falls within a designated range. The imaging condition setting unit includes: Based on a difference between an image captured by setting the luminous intensity of the lighting device on the workpiece to a threshold value or more and an image captured by setting the luminous intensity of the lighting device on the workpiece to a minimum value under a plurality of different environmental light conditions, an exposure time of the photographing device is set so that pixel values ​​of a designated region of the image of the workpiece are within a designated range. Shooting condition setting system.

3. The imaging condition setting unit includes: The photographing condition setting system according to claim 1 or 2, wherein the exposure time of the photographing device is set so that when the light intensity irradiated from the lighting device onto the workpiece is set to the maximum value and the image is taken, the pixel values ​​of a specified area of ​​the image of the workpiece are within a specified range.

4. The lighting device includes a plurality of light sources, each of which has an adjustable luminance; The imaging condition setting unit includes:

3. The photographing condition setting system according to claim 1, wherein the luminous intensity of at least one of the plurality of light sources is set to a value equal to or greater than a threshold value.

5. The imaging condition setting unit includes:

3. The photographing condition setting system according to claim 1, wherein when the characteristic of the pixel value with respect to the exposure of the photographing device is not in a linear relationship, the exposure time is set after calibration of the photographing device is performed.

6. The imaging condition setting unit includes:

3. The photographing condition setting system according to claim 1 or 2, wherein the exposure time of the photographing device is set so that when an image is taken with at least one of the set value of the luminous intensity irradiated from the lighting device onto the workpiece, the measured value of the actual luminous intensity, and the luminous intensity displayed on the display set to a value equal to or above a predetermined threshold, the pixel values ​​of a specified area of ​​the image of the workpiece are within a specified range.

7. A method for setting photographing conditions for an image of a workpiece to be inspected by a computer, comprising: The computer sets the luminous intensity of the lighting device that irradiates the workpiece with light to a value equal to or greater than a threshold value; The computer photographs the workpiece using an imaging device under the light intensity conditions, The computer sets the exposure time of the photographing device based on the photographed image of the workpiece so that the pixel value of a designated area of ​​the image of the workpiece falls within a designated range. In setting the exposure time of the imaging device, Based on a difference between an image captured by setting the light intensity applied from the lighting device to the workpiece at a threshold value or more and an image captured by setting the light intensity applied from the lighting device to the workpiece at a minimum value, an exposure time of the photographing device is set so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range. How to set shooting conditions.

8. A method for setting photographing conditions for an image of a workpiece to be inspected by a computer, comprising: The computer sets the luminous intensity of the lighting device that irradiates the workpiece with light to a value equal to or greater than a threshold value; The computer photographs the workpiece using an imaging device under the light intensity conditions, The computer sets the exposure time of the photographing device based on the photographed image of the workpiece so that the pixel value of a designated area of ​​the image of the workpiece falls within a designated range. In setting the exposure time of the imaging device, Based on a difference between an image captured by setting the luminous intensity of the lighting device on the workpiece to a threshold value or more and an image captured by setting the luminous intensity of the lighting device on the workpiece to a minimum value under a plurality of different environmental light conditions, an exposure time of the photographing device is set so that pixel values ​​of a designated region of the image of the workpiece are within a designated range. How to set shooting conditions.

9. A computer determines the conditions for taking an image of the workpiece to be inspected. Setting the luminous intensity of an illumination device that irradiates light onto the workpiece to a value equal to or greater than a threshold value; Photographing the workpiece with an imaging device under the light intensity conditions; and setting an exposure time of the photographing device so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range based on the photographed image of the workpiece; In setting the exposure time of the imaging device, Based on a difference between an image captured by setting the light intensity applied from the lighting device to the workpiece at a threshold value or more and an image captured by setting the light intensity applied from the lighting device to the workpiece at a minimum value, an exposure time of the photographing device is set so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range. program.

10. A computer determines the conditions for taking an image of the workpiece to be inspected. Setting the luminous intensity of an illumination device that irradiates light onto the workpiece to a value equal to or greater than a threshold value; Photographing the workpiece with an imaging device under the light intensity conditions; and setting an exposure time of the photographing device so that pixel values ​​of a designated area of ​​the image of the workpiece are within a designated range based on the photographed image of the workpiece; In setting the exposure time of the imaging device, Based on a difference between an image captured by setting the luminous intensity of the lighting device on the workpiece to a threshold value or more and an image captured by setting the luminous intensity of the lighting device on the workpiece to a minimum value under a plurality of different environmental light conditions, an exposure time of the photographing device is set so that pixel values ​​of a designated region of the image of the workpiece are within a designated range. program.

Images