Inspection condition determination system
Patent Information
- Application Number
- JP2022163558
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2022-10-11
- Publication Date
- 2025-05-13
AI Technical Summary
Existing inspection systems primarily focus on optimizing illumination and inspection algorithm parameters for accuracy, neglecting other critical factors such as test time, cost, robustness, and versatility, which are crucial for effective inspection processes.
An inspection condition determination system that considers multiple evaluation indicators including test accuracy, false alarm rate, test time, equipment cost, and environmental robustness to determine optimal inspection conditions, using a processor to evaluate and adjust illumination, imaging, and handling mechanisms based on user-defined criteria.
Enables the appropriate determination of inspection conditions that balance accuracy, cost, and efficiency, improving the overall performance and versatility of the inspection system.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for determining inspection conditions in an inspection system that inspects an object to be inspected. [Background technology]
[0002] For example, an inspection system is known that performs visual inspection of products manufactured in a manufacturing process.
[0003] Patent Document 1 discloses a method for setting an illumination condition when inspecting an object, the method including: illuminating the object with a light source capable of changing illumination parameters that define the illumination condition when the object is imaged; imaging the object with a plurality of illumination parameters by an image sensor; acquiring an image corresponding to the plurality of illumination parameters; and generating an estimated image of the object when the object is illuminated with variable illumination parameters based on an image data set obtained by associating the captured image with the illumination parameters corresponding to the captured image. The estimated image and the label data corresponding to the variable illumination parameters of the object are applied to learning of a machine learning model, and both the illumination parameters and the inspection algorithm parameters are simultaneously optimized based on a comparison result between the estimation result of the machine learning model and the label data of the object, thereby setting both the illumination condition and the inspection algorithm parameters of the machine learning model. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] International Publication No. 2019 / 171121 Summary of the Invention [Problem to be solved by the invention]
[0005] The technology described in Patent Document 1 makes it possible to simultaneously optimize both the illumination parameters and the parameters of the inspection algorithm by taking into consideration only the inspection accuracy rate.
[0006] However, in an inspection system, it is not always preferable to determine the conditions only by considering the inspection accuracy rate.
[0007] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technique capable of appropriately determining inspection conditions in an inspection system. [Means for solving the problem]
[0008] In order to achieve the above-mentioned object, an inspection condition determination system according to one aspect is an inspection condition determination system that includes a processor and determines inspection conditions for an inspection by an inspection system of a specified inspection object, wherein the processor accepts input of a user request regarding at least two or more element evaluation indexes for the inspection by the inspection system, the element evaluation indexes being the inspection accuracy rate, the false alarm rate, the inspection time, the inspection reproducibility, the learning time, the equipment cost, the operating cost, the equipment robustness, the inspection versatility, the life span, the ease of assembly, the required assembly precision, the robustness against the environment, the ease of maintenance, the clarity of the judgment basis for the recognition processing, and the ease of recognizing the detection point, and the goals of the element evaluation indexes, and creates an evaluation index based on the multiple element evaluation indexes for the inspection by the inspection system based on the user request, evaluates the inspection results under the multiple inspection conditions using the evaluation indexes, and determines appropriate inspection conditions from among the multiple inspection conditions. Effect of the Invention
[0009] According to the present invention, the inspection conditions in the inspection system can be appropriately determined. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an inspection system according to an embodiment. [Diagram 2] FIG. 2 is a configuration diagram of a computer that determines inspection conditions according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating the inspection conditions according to an embodiment. [Figure 4] FIG. 4 is a flowchart of a first inspection condition optimization process according to an embodiment. [Diagram 5] FIG. 5 is a flowchart of a second inspection condition optimization process according to an embodiment. [Figure 6] FIG. 6 is a flowchart of an optical model calibration process according to an embodiment. [Figure 7] FIG. 7 is a diagram showing a GUI screen according to an embodiment. [Figure 8] FIG. 8 is a diagram for explaining inspection conditions when dividing an inspection area according to an embodiment. [Figure 9] FIG. 9 is a diagram illustrating a change in an inspection image according to an embodiment. [Figure 10] FIG. 10 is a flowchart of an inspection condition redetermining process according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating a manufacturing process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.
[0012] FIG. 1 is a diagram showing the overall configuration of an inspection system according to the first embodiment.
[0013] The inspection system 100 includes a lighting device 110, an imaging sensor 120, a handling mechanism 140, and a computer 150. The lighting device 110 and the computer 150 are connected via a communication line 111, the imaging sensor 120 and the computer 150 are connected via a communication line 121, and the handling mechanism 140 and the computer 150 are connected via a communication line 141.
[0014] The illumination 110 emits light for inspection. When the illumination 110 emits light onto the inspection target 130, defects present in the inspection target 130 can be made apparent.
[0015] The imaging sensor 120 is a two-dimensional camera or the like that captures an image (digital image) of the inspection object 130. The imaging sensor 120 can capture an image of defects present in the inspection object 130.
[0016] The handling mechanism 140 changes the relative positions of the illumination 110 and the imaging sensor 120, and the inspection target 130. The handling mechanism 140 is capable of moving at least one of the illumination 110 and the imaging sensor 120 and the inspection target 130.
[0017] The computer 150 includes a memory 151 , a processor 152 , an input device 153 , and a display device 154 .
[0018] The memory 151 is a storage device that stores various information. The memory 151 may be a non-volatile or volatile memory medium such as a random access memory (RAM) or a read only memory (ROM). The memory 151 may also be a rewritable storage medium such as a flash memory, a hard disk, or a solid state drive (SSD), or may be a universal serial bus (USB) memory, a memory card, or the like.
[0019] The memory 151 stores a recognition program 161 and a control program 162. The recognition program 161 is executed by the processor 152 to perform processing (recognition processing) for recognizing defects from an image (digital data) captured by the imaging sensor 120. In this embodiment, the recognition program 161 is capable of changing at least a portion of a recognition engine that executes the recognition processing and parameters used in the recognition processing. The control program 162 is executed by the processor 152 to control the lighting 110, the imaging sensor 120, and the handling mechanism 140 in accordance with inspection conditions.
[0020] The processor 152 executes various processes according to the programs stored in the memory 151. The processor 152 may be, for example, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a quantum processor, or other semiconductor devices capable of performing calculations.
[0021] The input device 153 is, for example, a mouse, a keyboard, etc., and accepts information input by a user. The display device 154 is, for example, a display, and displays and outputs a user interface including various types of information.
[0022] In the inspection system 100, at least some of the following inspection conditions can be changed: conditions related to illumination by the illumination 110 (illumination conditions), conditions related to imaging by the imaging sensor 120 (imaging conditions), and conditions related to operations by the handling mechanism 140 (handling conditions). The illumination conditions include, for example, at least one of the following: illumination type, illumination arrangement, illumination intensity, illumination time, and lighting cycle. The imaging conditions include at least one of the following: imaging sensor type, imaging sensor arrangement, aperture value, imaging time, and imaging cycle. The handling conditions include at least one of the following: handling mechanism type, position and orientation, and trajectory to reach the target position and orientation.
[0023] Next, the computer 200 that performs processing to determine optimal inspection conditions in the inspection system 100 will be described.
[0024] FIG. 2 is a configuration diagram of a computer that determines inspection conditions according to the first embodiment.
[0025] The computer 200 is an example of an inspection condition determination system, and includes a memory 201, a processor 202, an input device 203, and a display device 204. The computer 200 is connected to a knowledge database 220 via a communication path. The computer 200 of this embodiment can acquire an inspection image from the computer 150, and can also transmit inspection conditions to the computer 150.
[0026] The memory 201 is a storage device that stores various information. The memory 201 may be a non-volatile or volatile memory medium such as a RAM or a ROM. The memory 201 may also be a rewritable storage medium such as a flash memory, a hard disk, or an SSD, or may be a USB memory, a memory card, or the like.
[0027] The memory 201 stores a recognition program 211 , an inspection condition optimization program 212 , an optical simulation program 213 , an optical model calibration program 214 , an inspection image change detection program 215 , and a multiple-machine inspection condition optimization program 216 .
[0028] The recognition program 211 is executed by the processor 202 to perform processing for recognizing defects from an inspection image. The inspection condition optimization program 212 is executed by the processor 202 to receive a request for inspection (customer request: user request) from a user (customer) and perform processing (inspection condition optimization processing) for detecting optimal inspection conditions that satisfy the customer request. The optical simulation program 213 is executed by the processor 202 to perform a simulation of the inspection using an optical model that can reproduce an inspection state for an inspection target in the inspection system 100. The optical model calibration program 214 is executed by the processor 202 to perform processing for calibrating the optical model (optical model calibration processing). The inspection image change detection program 215 is executed by the processor 202 to perform processing for detecting changes in the inspection image. The multiple-unit inspection condition optimization program 216 is executed by the processor 202 to perform processing for determining optimal inspection conditions in a manufacturing process using multiple inspection systems.
[0029] The processor 202 executes various processes according to the programs stored in the memory 201. The processor 202 may be, for example, a microprocessor, a CPU, a GPU, an FPGA, a quantum processor, or other semiconductor devices capable of performing calculations.
[0030] The input device 203 is, for example, a mouse, a keyboard, etc., and accepts information input by a user. The display device 204 is, for example, a display, and displays and outputs a user interface including various types of information.
[0031] The knowledge database 220 manages information on cases inspected by the inspection system 100, such as information on the image of the inspection target (inspection image) and the inspection conditions at that time. Although the knowledge database 220 is provided outside the computer 200, the present invention is not limited to this, and the knowledge database 220 may be provided inside the computer 200.
[0032] Next, customer requests accepted by the inspection condition optimization program 212 will be described.
[0033] The customer requirements include multiple element evaluation indexes, namely, the inspection accuracy rate f1, the false alarm rate f2, the inspection time f3, the inspection reproducibility f4, the learning time f5, the equipment cost f6, the operation cost f7, the equipment robustness f8, the inspection versatility f9, the life span f10, the ease of assembly f11, the required assembly precision f12, the robustness against the environment f13, the ease of maintenance f14, the clarity of the judgment grounds for the recognition processing f15, and the ease of recognizing the detection point f16.
[0034] The inspection accuracy rate f1 is the rate at which good and bad products are correctly recognized relative to the number of inspection objects. If the number of good products recognized as good products is TN, the number of bad products recognized as bad products is TP, the number of good products misidentified as bad products is FP, and the number of bad products misidentified as good products is FN, then the inspection accuracy rate f1 can be expressed as the following formula (1).
[0035]
number
[0036] The false alarm rate f2 is the rate at which good products are mistakenly recognized as defective products relative to the total number of good products. The false alarm rate f2 can be expressed as the following formula (2).
[0037]
number
[0038] Here, the test accuracy rate f1 and the false alarm rate f2 are given as examples of indices relating to the accuracy of the test, but the present invention is not limited to these, and other indices using more than one of TN, TP, FP, and FN may be used.
[0039] The inspection time f3 is the time related to the inspection, and may be, for example, the time required to inspect one inspection object, which is the sum of the movement time of the handling mechanism 140, the imaging time of the imaging sensor 120, the data transfer time, and the time it takes for the recognition processing program to determine defects.
[0040] Test reproducibility f4 is the reproducibility of a test when the same test object is repeatedly tested.
[0041] Learning time f5 is the time required for learning when a learning-type inspection algorithm is used as recognition program 161, and is 0 when recognition program 161 uses a non-learning type inspection algorithm, for example, a method of detecting defects by comparing image features with a threshold value.
[0042] The equipment cost f6 is the cost of constructing the inspection system 100.
[0043] The operating cost f7 is a cost associated with operating the inspection system 100. The operating cost f7 includes the electricity cost and the maintenance cost of the inspection system 100.
[0044] The device robustness f8 is an index that represents how unlikely the inspection system 100 is to malfunction.
[0045] Inspection versatility f9 is an index that indicates whether inspection system 100 can be applied to various inspection targets.
[0046] Lifetime f10 is the lifespan of the inspection system 100.
[0047] The ease of assembly f11 is an index representing the ease of assembly of the inspection system 100.
[0048] The required assembly precision f12 is an index showing the assembly precision required to achieve an inspection accuracy rate, etc. The higher the required assembly precision, the more the assembly error adversely affects the inspection accuracy rate, etc., and the higher the device assembly cost.
[0049] The robustness f13 against the environment is an index showing whether the inspection result will not be distorted by the environment such as foreign matter, temperature and humidity changes, and vibrations, and whether the inspection system 100 will not break down.
[0050] The ease of maintenance f14 is an index representing the ease of cleaning and adjusting the inspection system 100.
[0051] The clarity of the judgment basis of the recognition process f15 is an index showing whether the recognition program 161 of the inspection system 100 is a recognition program that can explain the judgment basis of the recognition process. In general, recognition programs that use deep learning often have poor judgment basis and low clarity.
[0052] The recognition ease f16 of the detected portion is an index that indicates the portion of the image where the defect is recognized when the recognition program 161 detects a defect. Depending on the type of recognition engine of the recognition program 161, it may not be possible to recognize the portion of the image where the defect is, and in such a case, the recognition ease is low.
[0053] The target values of the customer indices (fi(i=1 to 16)) are input by the user via the GUI screen 900 (see FIG. 7), for example, but the target values of some of the element evaluation indices of the customer indices include qualitative indices. Therefore, in this embodiment, the received customer indices (fi(i=1 to 16)) are quantified and converted into comparable customer indices (gi(i=1 to 16)).
[0054] Then, based on the comparative customer index (gi (i=1 to 16)), an index Ftarget including a plurality of element evaluation indexes is generated as shown in formula (3), and used in the optimization process.
[0055]
number
[0056] Here, wi is a weight corresponding to each element evaluation index g i. Input of wi may be received via the GUI screen 900, and resetting of wi may also be received via the GUI screen 900.
[0057] FIG. 3 is a diagram for explaining the inspection conditions according to the first embodiment.
[0058] The inspection conditions of the inspection system 100 include conditions related to illumination (illumination conditions), conditions related to imaging (imaging conditions), conditions related to handling (handling conditions), and conditions related to recognition (recognition conditions).
[0059] The lighting conditions include lighting type, lighting arrangement, lighting intensity, lighting time, lighting cycle, etc. The imaging conditions include imaging sensor type, imaging sensor arrangement, aperture value, imaging time, imaging cycle, etc. The handling conditions include handling mechanism type, target position and orientation, path to the target position and orientation, etc. The recognition conditions include recognition engine type, recognition processing parameters, etc.
[0060] In this embodiment, each condition is represented by a symbol shown in FIG. 3. Here, T indicates the type of object subject to the condition. P indicates the three-dimensional position and orientation of the object. P corresponds to, for example, a vector (x, y, z, i, j, k) that combines (x, y, z) indicating the three-dimensional position and a unit vector (i, j, k) indicating the orientation. S indicates a period related to the object. H indicates a time related to the object. I indicates illumination intensity (for example, watts), A indicates the aperture amount (aperture strength, for example, a value from 1 to 0) of the image sensor, K indicates the trajectory of the handling mechanism 140 (for example, a set of vectors (x, y, z, i, j, k) of the three-dimensional coordinates and orientation of each position on the trajectory), and M indicates a parameter (a set of parameter values) of the recognition process.
[0061] The suffixes (1) to (4) at the top right of the alphabet indicate values related to lighting, imaging, handling, and recognition, respectively. The suffix i at the bottom right of the alphabet indicates the number of arrangements of the same type of item. K has suffixes i and j at the bottom right, where i indicates the start point of the trajectory and j indicates the end point of the trajectory. T and M, which are related to recognition, have suffixes i and l at the bottom right, where i indicates the location of the captured image and l indicates the number of the recognition process performed on that image.
[0062] In the inspection condition optimization process by the inspection condition optimization program 212, a combination of appropriate inspection conditions (T, P, S, H, I, A, K, M) is calculated and output based on the index Ftarget. Here, the appropriate inspection conditions are determined depending on the index to be adopted, and are, for example, inspection conditions in which the value of the index Ftarget satisfies any of the following: minimum, maximum, minimal, maximal, a predetermined value, a gradual (stable) maximal value, or a gradual minimal value.
[0063] Next, the processing operation of the inspection condition optimization process performed by the computer 200 will be described.
[0064] First, an inspection condition optimization process (first inspection condition optimization process) for optimizing the inspection conditions based on the actual inspection results by the inspection system 100 will be described.
[0065] FIG. 4 is a flowchart of a first inspection condition optimization process according to an embodiment.
[0066] The inspection condition optimization program 212 (more precisely, the processor 202 executing the inspection condition optimization program 212) receives customer indexes (f1 to f16) corresponding to customer requests, target values for each element evaluation index, and weight settings for each element evaluation index via the GUI screen 900 (S501).
[0067] Next, the inspection condition optimization program 212 creates an evaluation index (equation (3)) to be used for processing based on the comparison evaluation indexes (g1 to g16) corresponding to the customer indexes (f1 to f16) and the weighting settings, and converts the target values of each element evaluation index into target values of the comparison evaluation indexes (comparison target values) (S502).
[0068] Next, the inspection condition optimization program 212 receives, for example, via the GUI screen 900, the consideration conditions to be used as constraint conditions for the inspection conditions in the optimization process (S503). Here, the consideration conditions include, for example, the type of illumination to be considered, the type of imaging sensor, the type of handling mechanism, the type of recognition engine, and the like.
[0069] Next, the inspection condition optimization program 212 determines the inspection conditions to be considered according to the consideration conditions, and sets the inspection system 100 according to the determined inspection conditions (target inspection conditions) (S504). Note that if the target inspection conditions include a condition that the inspection system 100 itself cannot set, the user of the inspection system 100 may set it.
[0070] Next, the inspection condition optimization program 212 causes the inspection system 100 to drive the handling mechanism 140 according to a predetermined trajectory (if the target inspection conditions include a trajectory condition, a trajectory corresponding to that condition) (S505), and causes the inspection system 100 to acquire an image of the inspection object 130 according to the target inspection conditions (S506).
[0071] Next, the inspection condition optimization program 212 judges whether or not all the movement of the trajectory has been completed and imaging by all the imaging sensors 120 has been completed (S507), and if all the movement of the trajectory has been completed and imaging by all the imaging sensors has not been completed (S507: NO), the process proceeds to step S505. This allows the movement of the trajectory and imaging by the imaging sensors 120 to continue.
[0072] On the other hand, when all the movements of the orbit have been completed and imaging by all the imaging sensors 120 has been completed (S507: YES), the inspection condition optimization program 212 causes the recognition program 211 to evaluate the inspection performance using the captured images (S508), and calculates the evaluation index Ftarget based on the results of the performance evaluation, etc. (S509).
[0073] Next, the inspection condition optimization program 212 determines whether or not the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have been calculated (S510), and if the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have not been calculated (S510: NO), the inspection condition optimization program 212 proceeds to step S504 to execute the inspection conditions including the considered conditions for which the evaluation indexes have not been calculated.
[0074] On the other hand, if the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have been calculated (S510: YES), the inspection condition optimization program 212 determines optimal inspection conditions based on the values of the evaluation indexes for the multiple inspection conditions, and outputs the determined results (for example, displays them on the GUI screen 900) (S511). Note that in step S511, optimal inspection conditions are determined and output, but this is not limiting, and appropriate inspection conditions that satisfy a predetermined standard (for example, exceeding a target value of the evaluation index) may be determined and output.
[0075] Next, an inspection condition optimization process (second inspection condition optimization process) for optimizing the inspection conditions based on the inspection results obtained by simulating the inspection in the inspection system 100 using an optical model will be described.
[0076] FIG. 5 is a flowchart of a second inspection condition optimization process according to an embodiment.
[0077] The inspection condition optimization program 212 (more precisely, the processor 202 executing the inspection condition optimization program 212) receives customer indexes (f1 to f16) corresponding to customer requests, target values for each element evaluation index, and weight settings for each element evaluation index via the GUI screen 900 (S601).
[0078] Next, the inspection condition optimization program 212 creates an evaluation index (equation (3)) to be used for processing based on the comparison evaluation indexes (g1 to g16) corresponding to the customer indexes (f1 to f16) and the weighting settings, and converts the target values of each element evaluation index into target values of the comparison evaluation indexes (comparison target values) (S602).
[0079] Next, the inspection condition optimization program 212 receives, for example, via the GUI screen 900, the consideration conditions to be used as constraint conditions for the inspection conditions in the optimization process (S603). Here, the consideration conditions include, for example, the type of illumination to be considered, the type of imaging sensor, the type of handling mechanism, the type of recognition engine, and the like.
[0080] Next, the inspection condition optimization program 212 determines the inspection conditions to be considered in accordance with the conditions to be considered, and the optical simulation program 213 creates an optical model representing the inspection state of the inspection object 130 in the inspection system 100 based on the shape model 601 (three-dimensional shape model) of the inspection object 130, the surface texture 602 of the inspection object 130, and the inspection conditions (S604).
[0081] Here, the shape model 601 is a 3D-CAD model which is design information of a product, or mesh or CAD data created from the results of measurement by a three-dimensional shape measuring device. The shape model 601 may be acquired from the memory 201 of the computer 200, for example, or may be received from an external device. The surface texture 602 is information (surface texture information) on the reflection characteristics of light determined by minute irregularities and materials not included in the shape model, which is expressed by a BRDF (Bidirectional Reflectance Distribution Function), for example, and may be input by a user, or may be data measured by a surface texture measuring device. The surface texture 602 may be acquired from the memory 201 of the computer 200, for example, or may be received from an external device.
[0082] The optical simulation program 213 moves the inspection object in the optical model according to a predetermined trajectory (if the object inspection conditions include a trajectory condition, the trajectory corresponding to that condition) (S605), and generates an image (estimated image) that is estimated to be captured of the inspection object 130 using the optical model (S606).
[0083] Next, the inspection condition optimization program 212 judges whether or not all the movement of the trajectory has been completed and the generation of estimated images estimated to be captured by all the imaging sensors has been completed (S607), and if all the movement of the trajectory has been completed and the generation of estimated images by all the imaging sensors has not been completed (S607: NO), the process proceeds to step S605. This results in the generation of estimated images in the case where the trajectory is subsequently moved.
[0084] On the other hand, when all the movements of the trajectory have been completed and all the estimated images have been generated (S607: YES), the inspection condition optimization program 212 causes the recognition program 211 to evaluate the inspection performance using the estimated images (S608), and calculates the evaluation index Ftarget based on the results of the performance evaluation, etc. (S609).
[0085] Next, the inspection condition optimization program 212 determines whether or not the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have been calculated (S610), and if the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have not been calculated (S610: NO), the inspection condition optimization program 212 proceeds to step S604 to execute the inspection conditions including the considered conditions for which the evaluation indexes have not been calculated.
[0086] On the other hand, if the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have been calculated (S610: YES), the inspection condition optimization program 212 determines optimal inspection conditions based on the values of the evaluation indexes for the multiple inspection conditions, and outputs the determined results (for example, displays them on the GUI screen 900) (S611). Note that in step S611, optimal inspection conditions are determined and output, but this is not limiting, and appropriate inspection conditions that satisfy a predetermined standard (for example, exceeding a target value of the evaluation index) may be determined and output.
[0087] In the inspection condition optimization process shown in FIG. 4 and FIG. 5, the inspection conditions are comprehensively changed to perform optimization, but the optimization method is not limited to this. For example, an efficient search such as an experimental design method or Bayesian optimization may be performed. For example, a Benders decomposition method may be used to search for optimal conditions by dividing the conditions into linear part conditions (conditions represented by the symbol T in FIG. 3) related to the device configuration and nonlinear part conditions (parameters). In addition, the type of illumination, the arrangement, and the type of recognition engine may be limited based on similar cases in the knowledge database 220. At this time, the similar cases may be those found by the user or those found using a machine learning matching function.
[0088] Next, an optical model calibration process for calibrating the optical model will be described. The optical model calibration process is executed, for example, before the second inspection condition optimization process shown in FIG.
[0089] FIG. 6 is a flowchart of an optical model calibration process according to an embodiment.
[0090] The optical model calibration program 214 (more precisely, the processor 202 executing the optical model calibration program 214) receives calibration condition settings from the user via the GUI screen, including the inspection conditions of the inspection system 100 to be used for calibration (e.g., the current inspection conditions of the inspection system 100) and the change conditions of the optical model (e.g., the change range of the reflectance of the inspection object (0.2 to 0.8, etc.)) (S701).
[0091] Next, the optical model calibration program 214 creates an evaluation index in the calibration (calibration evaluation index) (S702). Here, the calibration evaluation index is a function that combines feature quantities to be compared in the calibration, and may be, for example, the average value or the sum of variances of the differences for each pixel between an estimated image described below and an acquired image. The calibration evaluation index may be defined by the user, or may be selected or generated based on examples.
[0092] Next, the optical model calibration program 214 executes a process of acquiring an estimated image using the optical model (S711 to S714) and a process of actually acquiring an image by the inspection system 100 (S721 to S724).
[0093] In the process of obtaining an estimated image using an optical model (S711 to S714), first, the optical model calibration program 214 determines the conditions of the optical model to be considered based on the calibration conditions (e.g., a predetermined value within the change range of the reflectance of the object to be inspected), and the optical simulation program 213 creates an optical model representing the inspection state of the object to be inspected 130 in the inspection system 100 based on the setting conditions in the calibration conditions and the determined conditions of the optical model (S711).
[0094] The optical simulation program 213 moves the inspection object in the optical model according to a predetermined trajectory (if the object inspection conditions include a trajectory condition, the trajectory corresponding to that condition) (S712), and generates an image (estimated image) that is estimated to be captured of the inspection object 130 using the optical model (S713).
[0095] Next, the optical model calibration program 214 determines whether or not all the movement of the orbit has been completed and generation of estimated images estimated to be captured by all the imaging sensors has been completed (S714), and if all the movement of the orbit has been completed and generation of estimated images by all the imaging sensors has not been completed (S714: NO), the process proceeds to step S712. This results in generation of estimated images in the case where the orbit has been subsequently moved.
[0096] On the other hand, if all the movements of the trajectory have been completed and all the estimated images have been generated (S714: YES), the optical model calibration program 214 advances the process to step S731.
[0097] In the process (S721 to S724) of actually acquiring an image by the inspection system 100, first, the optical model calibration program 214 sets the inspection system 100 according to the inspection conditions (target inspection conditions) included in the calibration conditions (S721). Note that if the target inspection conditions include a condition that cannot be set by the inspection system 100 itself, the user of the inspection system 100 may set it.
[0098] Next, the optical model calibration program 214 causes the inspection system 100 to drive the handling mechanism according to a predetermined trajectory (if the target inspection conditions include a trajectory condition, then a trajectory corresponding to that condition) (S722), and causes the inspection system 100 to acquire an image (actual image) of the inspection target 130 according to the target inspection conditions (S723).
[0099] Next, the optical model calibration program 214 determines whether or not all the movement of the orbit has been completed and imaging by all the imaging sensors has been completed (S724), and if all the movement of the orbit has been completed and imaging by all the imaging sensors has not been completed (S724: NO), the process proceeds to step S722, whereby movement of the orbit and imaging by the imaging sensors are continued.
[0100] On the other hand, if all movement along the orbit has been completed and imaging by all image sensors has been completed (S724: YES), the optical model calibration program 214 advances the process to step S731.
[0101] In step S731, the optical model calibration program 214 compares the feature amounts of the estimated image and the actual image, and calculates a calibration evaluation index for the optical model used.
[0102] Next, the optical model calibration program 214 determines whether or not changes have been made to all optical models corresponding to the change conditions included in the calibration conditions (S732).
[0103] As a result, if changes have not been made to all optical models (S732: NO), the optical model calibration program 214 changes the conditions for the optical model based on the change conditions (S733), proceeds to step S711, and performs a process of obtaining estimated images for other optical models.
[0104] On the other hand, if all the optical models have been changed (S732: YES), the optical model calibration program 214 identifies the parameters of the optical model that appropriately represent the inspection of the actual inspection target based on the evaluation index (for example, the reflectance of the inspection target), and outputs the parameters of the optical model as calibration data (S734). Here, the parameters of the optical model that appropriately represent the inspection are determined by the calibration evaluation index to be adopted, and are, for example, parameters that satisfy any of the following values of the calibration evaluation index: minimum, maximum, minimum, maximum, a predetermined value, a gradual (stable) maximal value, and a gradual maximal value.
[0105] After that, when the second inspection condition optimization process shown in FIG. 5 is executed, the optical simulation program 213 generates an optical model based on the calibration data, and generates an optical model that conforms to the inspection conditions in the actual inspection system 100, thereby generating a highly accurate estimated image.
[0106] Next, the GUI screen 900 will be described.
[0107] FIG. 7 is a diagram showing a GUI screen according to an embodiment.
[0108] The GUI screen 900 includes an input area 910 and an output area 920. The input area 910 includes a customer requirement input area 911, a consideration condition input area 912, and a weight input area 913.
[0109] The customer requirement input area 911 is an area for inputting customer requirements, and includes a symbol indicating the element evaluation index of the customer requirement, a customer requirement indicating the content of the element evaluation index, and a target value field for inputting the target value of the element evaluation index. Here, by setting the target value to unnecessary, it is possible to set the corresponding element evaluation index to an evaluation index that does not take into account. In other words, it is possible to prevent the value of the corresponding element evaluation index from being taken into account in formula (3).
[0110] The consideration condition input area 912 is an area for inputting conditions to be considered in the inspection conditions of the inspection system 100 (consideration conditions).
[0111] The weight input area 913 is an area for inputting a weight value for each element evaluation index value. In the weight input area 913, a weight value and a setting of whether the weight value is a fixed value or a variable value can be input.
[0112] A processing result list 921 is displayed in the output area 920. The processing result list 921 includes entries of appropriate inspection conditions detected in the inspection condition optimization process. For example, when the weights wi of all element evaluation indexes are input as fixed values, one entry is displayed, and when the weights wi of the element evaluation indexes are input as variable values, entries of appropriate inspection conditions corresponding to each set of weights of each element evaluation index are displayed.
[0113] An entry in the processing result list 921 includes fields for element evaluation index / weighting 921a and inspection conditions 921b. The element evaluation index / weighting 921a displays a value for each element evaluation index and a weighting value for the element evaluation index. The inspection conditions 921b store values (inspection condition information) of each inspection condition of the inspection system 100.
[0114] In the above explanation, an example was shown in which the inspection target 130 was inspected as a whole, but the inspection target 130 may have different appropriate lighting, imaging sensors, and recognition programs for each part of the inspection target depending on its shape and surface properties. For example, if the inspection target is a cut-out part of a casting, the surface properties will be different between the part where the casting surface remains and the cut-out part, and the image characteristics will be significantly different. Therefore, changing the type of lighting, lighting angle, lighting intensity, or changing the parameters of the recognition program will lead to an improvement in the inspection success rate.
[0115] Therefore, the inspection area of the inspection object 130 may be divided into a plurality of parts for inspection. The case of dividing into a plurality of inspection areas for inspection will be described below.
[0116] Fig. 8 is a diagram for explaining inspection conditions when dividing an inspection area according to one embodiment. Fig. 8 shows an example in which an inspection object 130 is divided into a flat surface 131 on the upper surface and a cylindrical surface 132 for inspection, and the flat surface 131 is imaged by the illumination 110a and the imaging sensor 120a, and the cylindrical surface 132 is imaged by the illumination 110b and the imaging sensor 120b for inspection.
[0117] The inspection conditions of the inspection system 100 include conditions related to illumination (illumination conditions), conditions related to imaging (imaging conditions), conditions related to handling (handling conditions), and conditions related to recognition (recognition conditions).
[0118] The lighting conditions include lighting type, lighting arrangement, lighting intensity, lighting time, lighting cycle, etc. The imaging conditions include imaging sensor type, imaging sensor arrangement, aperture value, imaging time, imaging cycle, etc. The handling conditions include handling mechanism type, target position and orientation, path to the target position and orientation, etc. The recognition conditions include recognition engine type, recognition processing parameters, etc.
[0119] Each condition is represented by a symbol shown in Fig. 8. Here, the differences between the symbols shown in Fig. 3 will be explained.
[0120] The subscript m is added to the bottom right of the alphabet. m is a number representing the surface to be inspected (area to be inspected). In this example, when the surface to be inspected is a flat surface 131, m is set to 1, and when the surface to be inspected is a cylindrical surface 132, m is set to 2.
[0121] Next, a process for optimizing inspection conditions when the inspection object 130 is divided into a plurality of parts and inspected will be described.
[0122] First, the inspection condition optimization program 212 determines a division plan for dividing the inspection object 130 into a plurality of inspection regions (partial inspection regions) based on the processing conditions of the inspection object 130 (division plan determination step).
[0123] Next, the inspection condition optimization program 212 executes the inspection condition optimization process of FIG. 4 or FIG. 5 for each inspection area in the division plan to determine optimal inspection conditions, and also sums the evaluation index values of each inspection area to calculate an overall evaluation index value (division plan evaluation index value) at the time of this division (division plan evaluation index calculation step).
[0124] Next, the inspection condition optimization program 212 performs a division plan determination step and a division plan evaluation index calculation step to determine inspection conditions for a plurality of division plans for the inspection object 130 and calculate the respective division plan evaluation index values.
[0125] Next, the inspection condition optimization program 212 determines appropriate (eg, optimal) division plans and inspection conditions that satisfy predetermined conditions based on the division plan evaluation indexes of the multiple division plans, and outputs the determination results.
[0126] According to this process, it is possible to determine an appropriate division plan for the inspection object 130 and appropriate inspection conditions for each inspection region in the division plan.
[0127] Next, a description will be given of changes over time in the inspection image of the object to be inspected by the inspection system 100.
[0128] Fig. 9 is a diagram illustrating changes in an inspection image according to an embodiment. Fig. 9(A) shows a state in which an inspection object is imaged, Fig. 9(B) shows an inspection image immediately after the inspection system 100 starts operating, and Fig. 9(C) shows an inspection image after a certain amount of time has passed since the inspection system 100 started operating.
[0129] In the inspection system 100, as shown in FIG. 9A, an illumination 110 is applied from above an inspection object 130, and an image is captured by an image sensor 120.
[0130] Immediately after the inspection system 100 starts operating, the inspection image of the inspection object 130 is in a state in which the inspection object 130 is clearly captured, as shown in FIG. 9(B).
[0131] Thereafter, if the operation of the inspection system 100 continues and, for example, the lens of the image sensor 120 is scratched, the inspection image of the inspection object 130 may include a scratch 1031 as shown in FIG. 9C. Also, if the lens becomes dirty, the image may become dark overall. Even if the inspection object 130 is the same product, the color and brightness of the surface 1032 of the inspection object 130 may change over time due to changes in the material used to manufacture the inspection object 130, etc.
[0132] In such a state, the inspection conditions in the inspection system 100 may no longer be optimal.
[0133] Therefore, in the inspection system 100 according to this embodiment, a process for redetermining the inspection conditions in such a case (inspection condition redetermining process) is performed.
[0134] FIG. 10 is a flowchart of an inspection condition redetermining process according to an embodiment.
[0135] The inspection image change detection program 215 receives a tolerance for variation in a feature in the inspection image (S1101). Here, the feature may be a statistic such as a brightness average or brightness variance in the inspection image or a predetermined region of the inspection image. The tolerance may be received from a user when the inspection system 100 starts operating or after a predetermined number of inspections are performed. The tolerance may be a default value.
[0136] Next, the inspection image change detection program 215 calculates image features based on the inspection image of the inspection target captured and stored by the inspection system 100 (S1102).
[0137] Next, the inspection image change detection program 215 determines whether the calculated feature amount exceeds a tolerance value (S1103).
[0138] As a result, if the calculated feature value exceeds the allowable value (S1103: YES), the inspection image change detection program 215 issues an alert and executes the inspection condition optimization program 212 to execute the inspection condition optimization process of FIG. 4 or 5 (S1104), and the inspection image change detection program 215 outputs the result (S1105). When the inspection condition optimization process is performed in the inspection condition re-determination process, some of the inspection conditions such as the illumination type, image sensor type, and handling mechanism may be fixed to their current state. In this way, the inspection conditions can be set to enable the use of the configuration already used in the inspection system 100 as is, and additional investment in the inspection system 100 can be prevented.
[0139] On the other hand, if the calculated feature value does not exceed the allowable value (S1103: NO), the inspection image change detection program 215 outputs a result indicating that it is not necessary to execute the inspection condition optimization process (S1105).
[0140] Next, an example will be described in which the computer 200 optimizes various conditions in a manufacturing process including a plurality of processing steps and a plurality of inspection steps.
[0141] FIG. 11 is a diagram illustrating a manufacturing process according to an embodiment.
[0142] This manufacturing process includes inspection process 1, processing process 1, inspection process 2, processing process 2, inspection process 3, processing process 3, and inspection process 4. In inspection process 1, inspection of the received material is performed by inspection system 100-1, in inspection process 2, inspection of the processed product processed in processing process 1 is performed by inspection system 100-2, in inspection process 3, inspection of the processed product processed in processing process 2 is performed by inspection system 100-3, and in inspection process 4, inspection of the processed product processed in processing process 3 is performed by inspection system 100-4. Here, the process capacity, yield, production speed, and production cost as information related to manufacturing in the processing processes (manufacturing information) are denoted as Ak, Tk, Sk, and Ck, respectively. Note that k represents the number of the processing process.
[0143] The inspection condition optimization process by the multiple machine inspection condition optimization program 216 in this example will be described. For convenience, the difference between the processing of the multiple machine inspection condition optimization program 216 and the inspection condition optimization program 212 will be described with reference to Fig. 3 (Fig. 4). Also, it is assumed that the process capacity Ak, yield Tk, manufacturing cost Sk, and manufacturing cost Ck for each processing step are known or assumed and have been set or input.
[0144] In step S501 (S601), the multiple-machine inspection condition optimization program 216 receives, via the GUI screen, customer indexes fij (i=1 to 16, j is the process number of the inspection process, and in the example of FIG. 11, j=1 to 4) corresponding to the customer requirements in each inspection system 100, target values, and weight settings (Wij) for each element evaluation index of the customer requirements. The multiple-machine inspection condition optimization program 216 also receives overall customer indexes Fi (i=1 to 16) corresponding to the customer requirements for the entire inspection process, target values, and weight settings (Wi) for each element evaluation index of the overall customer requirements.
[0145] In step S502 (S602), the multiple machine inspection condition optimization program 216 obtains a comparison evaluation index gij corresponding to the customer index fij (i = 1 to 16, j = 1 to 4), obtains a comparison evaluation index Gi corresponding to the overall customer index Fi (i = 1 to 16), creates an evaluation index (equation (4)) to be used for processing taking into account the inspection process and the processing process, and converts the target values of the customer index and the overall customer index into target values of the comparison evaluation index (comparison target value).
[0146]
number
[0147] Here, wak, wtk, wsk, and wck are the weights of process capability Ak, yield Tk, manufacturing cost Sk, and manufacturing cost Ck, respectively. Note that in formula (4), the evaluation index is a linear sum of the evaluation index of each processing step and the information on the manufacturing status of the processing step, but the present invention is not limited to this. For example, it may be a product or a sum of the reciprocals of each index.
[0148] In step S511 (S611), the multiple-unit inspection condition optimization program 216 determines optimal inspection conditions based on the values of the evaluation index (equation (4)) under multiple inspection conditions, and outputs the determined results.
[0149] This process of optimizing conditions in the manufacturing process combines customer requirements for the inspection process and information on the processing process into a single evaluation index, making it possible to determine appropriate inspection conditions for the entire manufacturing process.
[0150] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.
[0151] For example, in the above embodiment, the optimal inspection conditions are determined, but the present invention is not limited to this. It is also possible to determine inspection conditions that are not limited to the optimal inspection conditions but that satisfy certain conditions.
[0152] In the above embodiment, the computer 200 is capable of executing the inspection condition optimization processes in FIG. 3 and FIG. 4, but it may be possible to execute only one of the inspection condition optimization processes.
[0153] Furthermore, in the above embodiment, the computers 150 and 200 are separate computers, but for example, the functions of the computer 200 may be incorporated into the computer 150.
[0154] In the above-described embodiments, a part or all of the processing performed by the processor may be performed by a hardware circuit. The programs in the above-described embodiments may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]
[0155] 100... inspection system, 150... computer, 200... computer, 201... memory, 202... processor, 203... input device, 204... display device, 211... recognition program, 212... inspection condition optimization program, 213... optical simulation program, 214... optical model calibration program, 215... inspection image change detection program, 216... multiple machine inspection condition optimization program
Claims
1. An inspection condition determination system including a processor, the inspection condition determination system determining an inspection condition for an inspection of a predetermined inspection object by an inspection system, The processor, accepts an input of a user request regarding at least two or more element evaluation indexes and targets of the element evaluation indexes for the inspection in the inspection system, the element evaluation indexes being at least two or more of the following: an inspection accuracy rate, a false alarm rate, an inspection time, an inspection reproducibility, a learning time, an equipment cost, an operation cost, an equipment robustness, an inspection versatility, a lifespan, an ease of assembly, a required assembly precision, robustness against the environment, an ease of maintenance, a clarity of a judgment basis for a recognition process, and an ease of recognition of a detection point; creating an evaluation index based on a plurality of element evaluation indexes for the inspection in the inspection system based on the user request; The inspection results under a plurality of inspection conditions are evaluated using the evaluation index, and an appropriate inspection condition is determined from among the plurality of inspection conditions. Inspection condition determination system.
2. The processor, Displaying and outputting inspection condition information regarding the determined inspection conditions The inspection condition determining system according to claim 1 .
3. The evaluation index is an index obtained by adjusting and combining the values of the multiple element evaluation indexes by weighting. The inspection condition determining system according to claim 2 .
4. The processor, Adjusting the weights; Evaluating the inspection results under a plurality of inspection conditions using the evaluation index with the weights adjusted, and determining appropriate inspection conditions from among the plurality of inspection conditions; Displaying inspection conditions and values of element evaluation indices corresponding to the inspection conditions The inspection condition determining system according to claim 3 .
5. and receiving adjustments to the weights from a user. The inspection condition determining system according to claim 4 .
6. the inspection system includes an illumination device that irradiates the inspection object with light, an image sensor that images the inspection object, and a handling mechanism that adjusts a relative position between the inspection object and the image sensor; The inspection conditions are: At least a part of the lighting type, lighting arrangement, lighting intensity, lighting time, lighting cycle, imaging sensor type, imaging sensor arrangement, aperture amount, imaging time, imaging cycle, handling mechanism type, target position and orientation, trajectory, recognition engine, and recognition processing parameters. The inspection condition determining system according to claim 1 .
7. The inspection conditions include at least one of an image sensor type, an image sensor arrangement, an aperture amount, an image capture time, and an image capture cycle, and at least one of a handling mechanism type, a target position and orientation, and a trajectory. The inspection condition determining system according to claim 6 .
8. The inspection system includes an illumination device that irradiates the inspection object with light, and an image sensor that images the inspection object; The processor, creating an optical model relating to an inspection state of the inspection object based on a three-dimensional shape model of the inspection object, surface texture information of the inspection object, inspection conditions relating to the illumination, and inspection conditions relating to the image sensor; creating an estimated image that is estimated to be captured by the imaging sensor based on the optical model; An evaluation is performed on the inspection result using the estimated image using the evaluation index, and appropriate inspection conditions are determined from among a plurality of inspection conditions. The inspection condition determining system according to claim 1 .
9. The processor, The three-dimensional shape model and the surface texture information are input. The inspection condition determining system according to claim 8 .
10. The processor, acquiring an actual image captured by the inspection system under the same inspection conditions as those used to create the estimated image; Identifying features of the estimated image and the actual image, and calibrating the optical model based on the features. The inspection condition determining system according to claim 8 .
11. The processor, creating a plurality of partition plans for partitioning an inspection area of an inspection object into a plurality of partial inspection areas; For each division plan, evaluation is performed for a plurality of partial inspection areas under a plurality of inspection conditions using the evaluation index, and appropriate inspection conditions are determined from among the plurality of inspection conditions; Decide on the most appropriate division plan from among multiple division plans The inspection condition determining system according to claim 1 .
12. The processor, An actual image captured by the inspection system under the determined inspection conditions is acquired, and it is determined whether or not the inspection result based on the actual image satisfies the target of the element evaluation index requested by the user. If the target is not satisfied, an alert is issued, new inspection results based on a plurality of inspection conditions are evaluated based on the evaluation index, and appropriate inspection conditions are determined from among the plurality of inspection conditions. The inspection condition determining system according to claim 1 .
13. The processor, A plurality of actual images are captured by the inspection system under the determined inspection conditions, and it is determined whether or not the feature values of the plurality of actual images exceed a predetermined allowable value. If the feature values exceed the allowable value, an alert is issued, and new inspection results under the plurality of inspection conditions are evaluated using the evaluation index, and appropriate inspection conditions are determined from among the plurality of inspection conditions. The inspection condition determining system according to claim 1 .
14. The processor, accepting input of the user request for each of the plurality of inspection systems; Based on the user request, an evaluation index based on a plurality of element evaluation indexes for the inspection in the inspection system is created, the inspection results under a plurality of inspection conditions are evaluated using the evaluation index, and appropriate inspection conditions are determined from among the plurality of inspection conditions. The inspection condition determining system according to claim 1 .
15. Multiple inspection systems are used in multiple inspection processes in the manufacturing process, The processor, further accepting an input of a user request regarding at least two or more element evaluation indexes and targets of the element evaluation indexes for the inspection of the entirety of the multiple inspection systems, the element evaluation indexes being at least two or more of the following: inspection accuracy rate, false alarm rate, inspection time, inspection reproducibility, learning time, equipment cost, operation cost, equipment robustness, inspection versatility, lifespan, ease of assembly, required assembly precision, robustness against the environment, ease of maintenance, clarity of judgment grounds for recognition processing, and ease of recognition of detected parts; Based on at least one of manufacturing information of the process capacity, yield, manufacturing speed, and manufacturing cost of the manufacturing process, an element evaluation index for the entire plurality of inspection systems, and an element evaluation index for each inspection system, an evaluation index for manufacturing and inspection in the manufacturing process is created, inspection results and manufacturing status under a plurality of inspection conditions are evaluated using the evaluation index, and appropriate inspection conditions are determined from among the plurality of inspection conditions. The inspection condition determining system according to claim 14.