A training image collection system, a training image collection method, and a training image collection program.

Abstract

To enable images for learning for an inspection engine that inspects the object of inspection to be easily and appropriately collected.SOLUTION: Provided is a processor system 100 for collecting an image group for learning for an inspection engine that inspects the object of inspection. The system has a processor 102, and the processor 102 is constituted so as to: generate a first simulation image group corresponding to a plurality of variations regarding the object of inspection; identify a second simulation image group in which images are cut back from the first simulation image group, and which can realize prescribed inspection performance for an image group for evaluation by training the inspection engine; receive a captured image group from the object of inspection which is to be corresponded to each image in the second simulation image group; and register the captured image group as an image group for learning for the inspection engine.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for collecting a group of learning images for an inspection engine that inspects an inspection target.

Background Art

[0002] In recent years, there have been many reports that learning-based image processing, mainly deep learning, shows performance superior to rule-based image processing in a wide range of fields such as recognition, classification, and inspection. Since the performance of learning-based image processing greatly depends on the learning data used, the quality of the learning data is important. Requirements for high-quality learning data include that variations of the learning target are covered, that unnecessary data that does not contribute to learning is not included, and that the data is not deteriorated by unintended effects such as misalignment, focus shift, and structural differences.

[0003] For example, in order to cover variations of the learning target in learning data, Patent Document 1 describes that "when applying machine learning to image inspection, it is necessary to comprehensively learn variations of the subject to be inspected. In this example, an example of efficiently selecting learning data so as to cover variations in the pattern shape on a semiconductor device using the design data of the semiconductor device and the shooting conditions of SEM is explained. In the case of a semiconductor, due to variations in the manufacturing process, the amount of deformation of the pattern shape varies depending on the formation position of the pattern on the semiconductor device. Therefore, in addition to variations in the pattern shape, selection considering the pattern formation position is effective."

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The method described in Patent Document 1 allows for the acquisition of training data that covers all variations of the target based on design data and shooting conditions. However, from the perspective of training data quality, since only variations are guaranteed, there is a problem that unnecessary or degraded data may be included in the training data.

[0006] This invention has been made in view of the above circumstances, and its purpose is to provide a technology that can easily and appropriately collect training images for an inspection engine that inspects objects to be inspected. [Means for solving the problem]

[0007] To achieve the above objective, a learning image collection system relating to one viewpoint is a learning image collection system that collects a set of learning images for an inspection engine that inspects an object to be inspected, and has a processor, the processor generates a first set of simulation images corresponding to multiple variations of the object to be inspected, identifies a second set of simulation images which is a set of images obtained by reducing the number of images from the first set of simulation images, and which can achieve a predetermined inspection performance for an evaluation set by training the inspection engine, receives a set of captured images of the object to be inspected which corresponds to each image in the second set of simulation images, and registers the captured image set as a set of learning images for the inspection engine. [Effects of the Invention]

[0008] According to the present invention, training images for an inspection engine that inspects an object to be inspected can be easily and appropriately collected. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is an overall configuration diagram of the imaging system according to the first embodiment. [Figure 2] Figure 2 illustrates the captured image and simulation image according to the first embodiment. [Figure 3]Figure 3 is a flowchart of the learning data collection process according to the first embodiment. [Figure 4] Figure 4 is a flowchart of the inspection target and environmental condition setting process according to the first embodiment. [Figure 5] Figure 5 is a flowchart of the provisional simulation image setting process according to the first embodiment. [Figure 6] Figure 6 is a flowchart showing an example of the provisional simulation image setting process related to the modified example. [Figure 7] Figure 7 is a flowchart of the variation reduction process related to the modified form. [Figure 8] Figure 8 is a flowchart of the deviation evaluation process related to the modified example. [Figure 9] Figure 9 shows an example of a GUI screen according to the first embodiment. [Figure 10] Figure 10 is a flowchart of the learning data collection process according to the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments will be described with reference to the drawings. Note that the embodiments described below are not intended to limit the invention as defined in the claims, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0011] [First Embodiment] <Configuration of imaging system 10> Figure 1 is an overall configuration diagram of the imaging system according to the first embodiment.

[0012] The imaging system 10 includes a processor system 100 as an example of a learning image acquisition system and an imaging device 105. The imaging device 105 is a device capable of capturing the surface or interior of an object to be inspected as digital data. Specifically, the imaging device 105 is a CCD (Charge Coupled Device) camera, an optical microscope, a charged particle microscope, an ultrasound inspection device, an X-ray inspection device, etc. The imaging device 105 transmits the captured image of the object to the processor system 100. The object is not limited as long as it can be imaged by the imaging device 105, but examples include industrial parts, daily necessities, food products, etc.

[0013] <<Details of Processor System 100>> The processor system 100 includes a processor 102, a communication device 103, an input / output device 104, and memory resources 110. Specifically, the processor system 100 is a computer such as a personal computer, tablet, smartphone, server, or cloud. The processor 102 reads various programs and data stored in the memory resources 110 and executes various processes related to the collection of learning data. Specifically, the processor 102 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or other arithmetic semiconductor device. The communication device 103 communicates information with an external device (such as an imaging device 105). The communication device 103 communicates information such as captured images with the imaging device 105, for example, via a wired or wireless LAN (Local Area Network). The input / output device 104 includes an input device that receives instructions from the user to the processor system 100 and an output device that outputs screens, etc., generated by the processor system 100. Input devices include, for example, keyboards, touch panels, mice, and microphones. Output devices include, for example, displays and speakers. Output devices output a screen (GUI screen) using a GUI (Graphical User Interface), which will be described later.

[0014] <<<Details of Memory Resource 110>>> Memory resource 110 holds various programs and various data. Specifically, memory resource 110 is, for example, RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), etc. Memory resource 110 holds simulation image generation program 111, inspection engine learning program 112, inspection engine evaluation program 113, deviation degree evaluation program 114, GUI execution program 115, simulation image database 116, captured image database 117, information database 118, inspection engine 119, and deviation degree evaluation engine 120.

[0015] <<<<Simulation Image Generation Program 111>>>> When simulation image generation program 111 is executed by processor 102, it executes various processes related to simulation images. Simulation image generation program 111 generates a set of simulation images (first simulation image group) with a predetermined number of variations based on the conditions of the inspection target and the environment. Also, simulation image generation program 111 generates a set of simulation images (second simulation image group) with the number of variations reduced according to the performance information of inspection engine 119 read from information database 118. Note that the set of simulation images includes a set of simulation images for learning to train inspection engine 119 and a set of simulation images for evaluating the performance of inspection engine 119.

[0016] <<<<Inspection Engine Learning Program 112>>>> Inspection engine learning program 112 reads a set of simulation images for learning from simulation image database 116 and performs machine learning on inspection engine 119 that performs a predetermined inspection task (inspection process). Examples of the predetermined inspection task include abnormal detection and defect detection of an object.

[0017] <<<<Inspection Engine Evaluation Program 113>>>> The inspection engine evaluation program 113 reads the inspection engine 119 and the set of simulation images for evaluating the simulation image database 116, and evaluates the inspection performance of the inspection engine 119. The inspection engine evaluation program 113 stores the information on the inspection performance of the inspection engine 119 in the information database 118.

[0018] <<<<Deviation Degree Evaluation Program 114>>>> The deviation degree evaluation program 114 reads the deviation degree evaluation engine 120 and evaluates the deviation degree between the simulation image received from the GUI execution program 115 and the captured image. The deviation degree evaluation program 114 stores the deviation degree information evaluated by the deviation degree evaluation engine 120 in the information database 118. Also, the deviation degree evaluation program 114 outputs the deviation degree information to the GUI execution program 115.

[0019] <<<<GUI Execution Program 115>>>> The GUI execution program 115 displays the GUI screen (see Fig. 9), and via the GUI screen, accepts user instructions such as changing the parameter value of the deviation degree threshold and changing the simulation image, and uses them as various inputs for the learning data collection process. Also, the GUI execution program 115 displays the simulation image read from the simulation image database 116 and the captured image read from the captured image database 117 on the GUI screen. Also, the GUI execution program 115 displays the deviation degree information between the captured image and the simulation image read from the information database 118 on the GUI screen.

[0020] <<<<Simulation Image Database 116>>>> The simulation image database 116 stores the set of simulation images generated by the simulation image generation program 111.

[0021] <<<<<Image Database 117>>>>> The image database 117 stores images of objects (captured images) captured by the imaging device 105. The stored captured images include provisional captured images (candidate images) displayed by the GUI execution program 115, and a group of captured images (training image group) registered as training data by the GUI execution program 115.

[0022] <<<<Information Database 118>>>> The information database 118 stores various types of information used in each process of the learning data collection process. For example, the information database 118 stores inspection performance information output by the inspection engine 119, deviation degree information output by the deviation degree evaluation engine 120, and so on.

[0023] <<<<Inspection Engine 119>>>> The inspection engine 119 is an engine that is machine-learned by the inspection engine learning program 112 and performs a predetermined inspection task (inspection process). The inspection engine 119 is, for example, a neural network model that takes an image as input and outputs an inspection result for the image.

[0024] <<<<Degree of deviation evaluation engine 120>>>> The deviation degree evaluation engine 120 is called by the deviation degree evaluation program 114 and is an engine that evaluates the degree of deviation between the captured image and the simulated image for one or more evaluation items. If there are multiple evaluation items for the degree of deviation, the deviation degree evaluation engine 120 may include multiple engines that evaluate each evaluation item. The deviation degree evaluation engine 120 is not limited to a machine learning-type model, but may also be a rule-based engine.

[0025] <Details of the learning data acquisition process in the imaging system 10> Before describing the training data collection process that implements the training image acquisition method according to this embodiment, we will explain the captured images and simulation images of the inspection target related to the training data collection process.

[0026] Figure 2 illustrates the captured image and simulation image according to the first embodiment.

[0027] The inspection target 300 includes, for example, structures 301 to 304. The inspection target 300 has different variations depending on, for example, the shape variations of structures 301 to 304. The inspection target 300 is placed, for example, on a base 315 and is imaged by an imaging device 105 installed above the base 315. The captured images of the inspection target 300 are as shown in, for example, captured images 306, 308, and 310.

[0028] Image 306 is an image of a certain inspection target 300. Image 308 is an image of an inspection target 300 that has a different variation from the inspection target in image 306. In image 308, structures 301 and 303 are the same as in image 306, but structure 302 is larger and structure 304 is smaller. In addition to variations in the inspection target, there are also variations in the imaging conditions in the images of the inspection targets.

[0029] The captured image 310 is the same as the inspection target 300 in the captured image 306, but it is an example of an image in which the position and orientation of the inspection target 300 have shifted, and unwanted objects 314 have been captured. The position and orientation shifts of the inspection target 300 and the inclusion of unwanted objects are not intended by the user, but they degrade the quality of the training data and reduce the inspection performance of the inspection engine 119.

[0030] 3DCG (Dimensional Computer Graphics) data 305 is data that represents the inspection target 300 in 3DCG space. Simulation image 307 is an image generated by projecting the 3DCG data 305 upwards in 2D. Simulation image 307 is a simulation image corresponding to the captured image 306. It is difficult to make simulation image 307 the same as captured image 306. Note that in Figure 2, a dot pattern is used to show that the images are not in the same state. Simulation image 309 is a simulation image corresponding to captured image 308, and the inspection target 300 in simulation image 307 has a different variation.

[0031] Next, we will explain the training data collection process.

[0032] Figure 3 is a flowchart of the learning data collection process according to the first embodiment.

[0033] Here, the conditions of the object to be inspected and the environment, as well as the user input information used in the processing, may be entered by the user at that time, or they may be entered and set in advance. Also, for simplicity, the variations in the image of the object to be inspected will be explained as only variations in the shape of the object to be inspected, but variations in the physical properties of the object to be inspected may also be included, as well as various other variations such as variations in imaging conditions. Furthermore, good product learning, which learns anomaly detection using only normal objects to be inspected, will be explained as an example, but good / bad product learning, which learns defect detection using both normal objects and objects with defects, may also be used. When performing good / bad product learning, for example, variations in defects can be included in the variations of the image of the object to be inspected.

[0034] In step S201, the simulation image generation program 111 (more precisely, the processor 102 that executes the simulation image generation program 111) reads the conditions for the inspection target and environment entered by the user and performs the process of setting up the simulation space. This process is equivalent to setting up a 3DCG space in which the imaging device 105, the inspection target 300, and the base 315 are configured.

[0035] In step S202, the simulation image generation program 111 reads variation information (tolerance range, stages, etc.) for the image of the object to be inspected input by the user, generates a 3DCG dataset corresponding to each variation, and performs a provisional simulation image setting process to generate a provisional simulation image set (first simulation image group) by projecting it onto two dimensions. The tolerance range is, for example, the tolerance range of the size of the structure of the object to be inspected 300 (e.g., ±10 mm), and the stages are, for example, the stages of size within the tolerance range.

[0036] Specifically, the simulation image generation program 111 sets a provisional set of simulation images generated within the acceptable range of variations as the training simulation images for the inspection engine 119. This process is equivalent to projecting the 3DCG data 305 into two dimensions to generate simulation images 307, 309, etc., with different variations, and using them as training simulation images. The initial number of variations in the training simulation images will be, for example, the number of combinations that cover the shape variation settings entered by the user. For example, if the shape variations of each of the structures 301 to 304 are set to 10 levels within the acceptable range, the initial number of variations in the training simulation images will be 10,000.

[0037] Furthermore, the simulation image generation program 111 generates a provisional set of simulation images, including those outside the acceptable range of variation, and sets this as the simulation image set (evaluation image group) for evaluating the inspection engine 119. Here, the simulation images outside the acceptable range correspond to images of defective products being inspected.

[0038] In step S203, the inspection engine learning program 112 reads a training simulation image set and trains the inspection engine 119 to perform a predetermined inspection task. Here, for example, the predetermined inspection task is anomaly detection. One method of performing anomaly detection is to use an autoencoder network that has been trained to match input and output images using images of good products. When a defective product image is input, the network cannot reconstruct the defective area, and the difference between the input and output images becomes large, which is used to perform anomaly detection.

[0039] In step S204, the inspection engine evaluation program 113 reads the inspection engine 119 and the evaluation simulation image set, and evaluates whether the predetermined inspection performance is met based on the inspection results obtained by inputting the evaluation simulation images to the inspection engine 119. For example, the predetermined inspection performance may be an anomaly detection rate of ≥ 95%.

[0040] In step S205, the inspection engine evaluation program 113 determines whether to terminate the trial of reducing the number of simulation images. In this embodiment, it is determined to terminate the trial of reducing the number of simulation images if it is determined in step S204 that the inspection performance is no longer met. Alternatively, it may be determined to terminate the trial of reducing the number of simulation images if the inspection performance is met and the number of simulation images falls below a predetermined number, or if other conditions set by the user are met.

[0041] If, as a result, it is determined that the trial of reducing the simulation images should not be terminated (S205: NO), the inspection engine evaluation program 113 proceeds to step S206.

[0042] In step S206, the simulation image generation program 111 performs a variation reduction process to reduce the number of variations in the training simulation images. Here, one method for reducing the variations in the simulation images is to thin out simulation images that are close in distance or have a high image correlation in the feature space. After reducing the variations, the simulation image generation program 111 proceeds to step S202. In step S202, the simulation image generation program 111 sets the simulation images with reduced variations as provisional simulation images for training the inspection engine 119.

[0043] On the other hand, if it is determined that the trial of reducing the simulation images is finished (S205:YES), it means that the limit of variation reduction has been reached, so the inspection engine evaluation program 113 proceeds to step S207.

[0044] In step S207, the simulation image generation program 111 saves a set of training simulation images that meets the inspection performance requirements (for example, if it is determined that the inspection performance requirements are not met and the trial of reducing the number of simulation images is terminated, the set of training simulation images that met the inspection performance requirements immediately before) to the simulation image database 116 as a simulation image for collecting training images (reference simulation image: second simulation image).

[0045] After step S207, the deviation degree evaluation program 114 repeatedly executes the loop 1 process (S208-S212) until the loop count i changes from 1 to N (the number of variations of the reference simulation image). The loop 1 process allows for the collection of captured images corresponding to the simulation images of each variation. It is assumed that, when the loop 1 process is performed, the inspection targets corresponding to each simulation image have been prepared, using the simulation images as a reference.

[0046] In step S208, the deviation degree evaluation program 114 acquires a provisional image of the subject being examined, which is captured by the imaging device 105. Here, the imaging device 105 may acquire the image in real time.

[0047] In step S209, the deviation degree evaluation program 114 reads the deviation degree evaluation engine 120, evaluates the degree of deviation between the simulated image and the captured image acquired in step S208, and performs a deviation evaluation process to store the deviation degree information from the deviation degree evaluation engine 120 in the information database 118. Methods for evaluating the degree of deviation include methods based on image correlation values ​​and distances in the feature space.

[0048] In step S210, the deviation degree evaluation program 114 determines whether the deviation degree recorded in the deviation degree information is below a predetermined threshold. If the deviation degree is not below the predetermined threshold (S210: NO), the deviation degree evaluation program 114 proceeds to step S211. If the deviation degree is below the predetermined threshold (S210: YES), the program proceeds to step S212.

[0049] In step S211, the GUI execution program 115 displays the simulation image and the captured image on the GUI screen 900. Furthermore, the GUI execution program 115 reads the degree of deviation information and displays instructions (correction instructions) on the GUI screen 900 to reduce the degree of deviation between the captured image and the simulation image, and proceeds to step S208. This allows the user to correct the captured image by changing the position of the object to be inspected, etc., in accordance with the instructions to reduce the degree of deviation.

[0050] In step S212, the deviation degree evaluation program 114 acquires an image of the object to be inspected using the imaging device 105. Here, the image acquired in step S212 may have a higher resolution than the image acquired in step S208. If the image acquired in step S208 has a high resolution, the image acquired in step S208 may be used without acquiring a new image.

[0051] The processing of Loop 1 is performed on the simulation images of each variation, and if the captured image corresponding to each simulation image is obtained, the deviation degree evaluation program 114 proceeds to step S213.

[0052] In step S213, the GUI execution program 115 registers the captured images corresponding to each simulation image as training captured images in the captured image database 117.

[0053] According to the process described above, a minimum set of simulation images that meets the required inspection performance can be created based on the inspection performance of the inspection engine learned from the simulation images. Furthermore, when capturing images corresponding to the simulation images, the degree of deviation between the simulation image and the captured image is evaluated, and instructions to reduce the degree of deviation are displayed. This allows for the capture of captured images with less deviation from the simulation image, enabling the efficient collection of high-quality training data. In addition, since captured images with a deviation below a threshold are registered as training images, high-quality training data can be collected.

[0054] Next, an example of the inspection target and environmental condition setting process in step S201 will be described.

[0055] Figure 4 is a flowchart of the inspection target and environmental condition setting process according to the first embodiment.

[0056] In step S501, the simulation image generation program 111 reads prior information about the inspection target and the environment. Here, the prior information is, for example, multiple images of the inspection target that have been captured in advance.

[0057] In step S502, the simulation image generation program 111 estimates the variations of the inspection target based on prior information. For example, it performs image analysis on the captured images of the prior information to analyze the shapes of structures 301 to 304 of the inspection target 300 and estimates the range of the actual size of the inspection target structure.

[0058] In step S503, the simulation image generation program 111 sets the inspection target and environmental conditions based on the estimated variations. For example, the conditions include information on the actual size range of each structure.

[0059] This process allows for the determination of variations by taking into account not only user information entered via the GUI, but also the actual images of the object being inspected. For example, if the tolerance range for the size of a structure is ±10 mm, and the actual size range of the structure is ±5 mm, the variation can be determined by narrowing the range to this actual size range. This allows for setting variations within a narrower range that is closer to reality, rather than comprehensive variations that follow the tolerance range.

[0060] Next, we will describe an example of the provisional simulation image setting process in step S202.

[0061] Figure 5 is a flowchart of the provisional simulation image setting process according to the first embodiment.

[0062] In step S401, the simulation image generation program 111 generates multiple training simulation images for each variation. For example, by introducing positional shifts to structures 302-304 of the inspection target 300 at a level that does not affect inspection performance, multiple simulation images with the same shape variation from a training perspective are generated. Here, a level that does not affect inspection performance means that the performance of the inspection engine does not change even if the images are replaced.

[0063] Next, the simulation image generation program 111 repeatedly executes the loop 2 process (S402, S403) until the loop count i changes from 1 to N (the number of simulation image variations). The loop 2 process allows the user to select a more appropriate simulation image for each variation.

[0064] In step S402, the GUI execution program 115 displays multiple simulation images corresponding to the variations on the GUI. In step S403, the GUI execution program 115 receives a selection of one simulation image from the user and sends it to the simulation image generation program 111. In this step, the user selects a simulation image that closely resembles one of several pre-prepared inspection targets.

[0065] If the processing in Loop 2 selects a simulation image corresponding to each variation, the simulation image generation program 111 proceeds to step S404.

[0066] In step S404, the simulation image generation program 111 sets the selected simulation image as a provisional simulation image.

[0067] This process allows for the selection of a simulation image that closely resembles the object being inspected from multiple simulation images, thereby reducing the effort required to prepare an object similar to the simulation image.

[0068] Next, we will explain the provisional simulation image setting process related to the modified example.

[0069] Figure 6 is a flowchart showing an example of the provisional simulation image setting process related to the modified example.

[0070] This provisional simulation image setup process includes a process to transform the simulation image to resemble the captured image.

[0071] Step S601 performs the same processing as step S501. In step S602, the simulation image generation program 111 extracts feature information from the simulation image and the captured image based on prior information. For example, the simulation image generation program 111 generates an engine (image quality conversion engine) that converts the simulation image to have the same image quality as the captured image of the object to be inspected. Here, the image quality conversion engine is a network that has been trained to output an image in which the input simulation image has been converted to an image quality close to that of the captured image.

[0072] In step S603, the simulation image generation program 111 generates one training simulation image for each variation. In step S604, the simulation image generation program 111 transforms the simulation image to resemble the captured image based on the extracted feature information. For example, the image quality transformation engine described above is used to transform the simulation image to an image quality close to that of the captured image.

[0073] In step S605, the simulation image generation program 111 sets the converted simulation image as a provisional simulation image.

[0074] This process allows for the creation of provisional simulation images that closely resemble the image quality of the captured images. Therefore, by training the inspection engine with these simulation images, the engine can be brought closer to the state it would have achieved with captured images, enabling more appropriate variation reduction.

[0075] Next, an example of the variation reduction process in step S206 will be described.

[0076] Figure 7 is a flowchart of the variation reduction process related to the modified form.

[0077] Variation reduction processing reduces the variations in the simulation image based on clustering information in the feature space of the simulation image.

[0078] In step S701, the simulation image generation program 111 projects the simulation images for each variation into the feature space. One method for projecting into the feature space is to use the intermediate output of a pre-trained network capable of accurately classifying hundreds of classes.

[0079] In step S702, the simulation image generation program 111 clusters the simulation images in the feature space. One clustering method is the k-means method.

[0080] In step S703, the simulation image generation program 111 determines which simulation images to reduce and deletes them based on the size of each cluster and the number of images belonging to each cluster. For example, if a cluster is small but has many images, many simulation images belonging to this cluster are deleted. Alternatively, simulation images located at the cluster boundaries may be avoided as much as possible, regardless of the cluster size.

[0081] This process is expected to maintain an appropriate distribution in the feature space of the simulation images and prevent a decrease in the inspection engine's performance due to a reduction in the variation of the simulation images.

[0082] Next, we will explain an example of the deviation evaluation process in S209.

[0083] Figure 8 is a flowchart of the deviation evaluation process related to the modified example.

[0084] The discrepancy evaluation process is a process that uses semantic segmentation to evaluate the degree of discrepancy between the captured image and the simulated image.

[0085] In step S801, the deviation degree evaluation program 114 performs semantic segmentation on the provisional captured image to separate the background and the structure to be inspected. Here, the semantic segmentation process can use a network that has been trained to separate and output segmented images for each region to be evaluated for deviation degree from the input image. For example, the deviation degree evaluation program 114 separates the inspection target 300 into five parts: the background and each of the inspection target structures 301 to 304.

[0086] Next, the deviation degree evaluation program 114 repeatedly executes the processing of loop 3 (S802~S806) for each deviation evaluation item until the loop count j is from 1 to M (number of deviation evaluation items). Through the processing of loop 3, the degree of deviation can be evaluated for each deviation evaluation item.

[0087] In step S802, the deviation degree evaluation program 114 sets evaluation regions corresponding to deviation evaluation items in the captured image. Here, the evaluation regions are set using segmentation images.

[0088] In step S803, the deviation degree evaluation program 114 uses the deviation degree evaluation engine 120 to calculate the degree of deviation between the simulated image and the captured image using an evaluation method corresponding to the deviation evaluation item. For example, if the deviation evaluation item is structures 301 to 304, the evaluation method could be the calculation of positional and orientation deviations based on feature point matching, or the calculation of structural differences based on alignment and area comparison. If the deviation evaluation item is the background, it could be the detection of unwanted reflections based on labeling.

[0089] In step S804, the deviation degree evaluation program 114 records the deviation evaluation results for the deviation evaluation items in the deviation degree information.

[0090] In step S805, the deviation degree evaluation program 114 determines whether the deviation degree of the deviation evaluation item is below a threshold. If the deviation degree is not below the threshold (S805: NO), the process proceeds to step S806. If the deviation degree is below the threshold (S805: YES), the loop 3 process for this deviation evaluation item is terminated.

[0091] In step S806, the deviation degree evaluation program 114 records a deviation correction flag in the deviation degree information to indicate that a correction is needed for the deviation evaluation item, and terminates the loop 3 processing for this deviation evaluation item.

[0092] If processing has been performed for all deviation evaluation items by the processing in Loop 3 (i.e., when j is the number of deviation evaluation items M), the deviation degree evaluation program 114 proceeds to step S807.

[0093] In step S807, the deviation degree evaluation program 114 stores the deviation degree information in the information database 118.

[0094] This process allows for the evaluation of the degree of discrepancy for each structure under inspection, enabling a more detailed assessment of the discrepancy between captured images and simulated images. Therefore, it is useful for adjusting the process to further reduce the degree of discrepancy between captured and simulated images.

[0095] Next, I will explain the GUI screen.

[0096] Figure 9 shows an example of a GUI screen according to the first embodiment.

[0097] Figure 9 shows an example of a GUI screen that displays the degree of discrepancy between the captured image and the simulated image.

[0098] The GUI screen 900 includes a simulation image ID field 901, a simulation image replacement button 902, a registration button 903, a parameter setting field 905, a simulation image display area 920, an captured image display area 921, a simulation image replacement candidate area 922, and a deviation adjustment announcement field 912.

[0099] The simulation image ID field 901 is an area for specifying the ID (number) of the simulation image to be displayed. The simulation image replacement button 902 is a button to press when replacing a simulation image. The registration button 903 is a button to press when registering an image corresponding to the image displayed in the image capture display area 921 as a training image.

[0100] The parameter setting field 905 is for setting thresholds for the degree of deviation. The parameter setting field 905 includes areas for setting thresholds for positional deviation 906, orientation deviation 907, structural 1 difference 908, structural 2 difference 909, structural 3 difference 910, structural 4 difference 911, etc. In the example in Figure 9, positional deviation, orientation deviation, and structural 1 difference are areas for setting thresholds for the degree of deviation for deviation evaluation items related to structure 301, while structural 2-4 differences are areas for setting thresholds for the degree of deviation for deviation evaluation items related to structures 302-304, respectively.

[0101] The simulation image display area 920 displays the simulation image corresponding to the number specified in the simulation image ID field 901. The captured image display area 921 displays a provisional captured image of the inspection target captured by the imaging device 105. In this embodiment, the inspection target being captured by the imaging device 105 is displayed in real time (as a moving image). The simulation image replacement candidate area 922 displays simulation images similar to the captured image displayed in the captured image display area 921.

[0102] The deviation adjustment announcement section 912 displays deviation adjustment information 913, which is information that provides instructions (correction instructions) to adjust (reduce) the degree of deviation (reduce) of evaluation items between the simulation image displayed in the simulation image display area 920 and the captured image displayed in the captured image display area 921.

[0103] The user can modify the imaging conditions of the object to be inspected by referring to the deviation adjustment information 913. When the imaging conditions are modified, the image displayed in the image display area 921 changes from the image 310 to the image 306, which was captured after the imaging conditions were changed. The information displayed in the deviation adjustment announcement field 912 changes from the deviation adjustment information 913 to the deviation adjustment information 914 between the simulation image 309 and the image 306. In the example in Figure 9, by changing to the image 306, the deviation adjustment information 914 is displayed, and it can be confirmed that the quality of the image has improved. The deviation adjustment information 914 contains information corresponding to the deviation adjustment flags included in the deviation degree information. For example, if the structural difference is such that it is not a problem to readjust the threshold value, the threshold value in the parameter setting field 905 may be readjusted to be larger. In this case, the corresponding deviation adjustment information can be removed from the deviation adjustment information 914.

[0104] Furthermore, when the simulation image replacement button 902 is pressed, the image in the simulation image display area 920 changes to the simulation image 307. Details of the simulation image replacement will be explained in the second embodiment. When the degree of deviation between the simulation image and the captured image changes due to the replacement of the simulation image, and the deviation adjustment flag disappears from the deviation degree information, the deviation adjustment information will no longer be displayed in the deviation adjustment announcement field 912.

[0105] This GUI screen 900 allows for visual comparison of the simulation image and the captured image. Furthermore, it enables adjustment of the captured image while checking discrepancy adjustment information for evaluation items that require adjustment, preventing rework such as re-acquisition and allowing for the efficient collection of high-quality captured images.

[0106] [Second Embodiment] Next, an imaging system according to the second embodiment will be described.

[0107] The processor system of the imaging system according to the second embodiment differs from the processor system according to the first embodiment in some aspects of the learning data collection process. The difference between the learning data collection process according to the second embodiment and the learning data collection process according to the first embodiment is that the second embodiment searches for a simulation image similar to the provisional image of the object to be inspected from the simulation image set, and if there is no simulation image similar to the provisional image, it prompts the user to change the object to be inspected.

[0108] Figure 10 is a flowchart of the training data collection process according to the second embodiment. In Figure 10, steps similar to those in the training data collection process shown in Figure 3 are denoted by the same reference numerals.

[0109] First, processor 102 executes the processes in steps S201 to S206.

[0110] Next, the processor 102 repeatedly executes the processes of loop 4 (S208, S209, S1001, S210, S1002, S212) until the loop count i changes from 1 to N (number of variations). Through the processes of loop 4, it is possible to collect captured images corresponding to the simulation images of each variation.

[0111] In step S1001, the deviation degree evaluation program 114 searches for a simulation image that is more similar to the provisional image from among the simulation images other than the simulation image evaluated in step S209. Specifically, the deviation degree evaluation program 114 evaluates the degree of deviation between the provisional image and each simulation image and extracts the simulation image with the smallest degree of deviation. If the degree of deviation with the extracted simulation image is smaller than the degree of deviation with the simulation image evaluated in step S209, the more similar simulation image is recorded in the deviation degree information. On the other hand, if there is no simulation image with a smaller degree of deviation than the simulation image evaluated in step S209, and step S209 has been processed multiple times with the same simulation image, a flag indicating a change in the inspection target is recorded in the deviation degree information.

[0112] In step S1002, in addition to the processing in step S211, the GUI execution program 115 displays a more similar simulation image in the simulation image replacement candidate area 922 of the GUI screen 900. When the simulation image replacement button 902 is pressed on the GUI screen 900, the deviation degree evaluation program 114 changes the simulation image to be evaluated in step S209 to a more similar simulation image. The GUI execution program 115 also displays information on the GUI screen 900 indicating that the inspection target has been changed. In this case, the user should select a new inspection target from several inspection targets prepared in advance and take an image. If there are not enough inspection targets for the simulation image, the user should prepare a new inspection target using the simulation image as a reference.

[0113] According to the learning data collection process of the second embodiment, a simulation image similar to the provisional captured image can be selected from the simulation image set, thus enabling more efficient collection of learning data. Furthermore, since there is an instruction to change the inspection target, the validity of multiple pre-prepared inspection targets can be efficiently confirmed based on the simulation images.

[0114] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0115] For example, in the above embodiment, when the degree of deviation is not below a threshold, an instruction to reduce the degree of deviation is displayed on the GUI screen 900, and the user changes the captured image according to the display. However, the present invention is not limited to this, and for example, the captured image may be changed automatically without user intervention according to the instruction to reduce the degree of deviation, for example, by automatically changing the position of the object to be inspected, etc., when taking images.

[0116] Furthermore, in the above embodiment, when collecting training data, the degree of discrepancy between the simulated image and the captured image was evaluated, and an instruction to reduce the degree of discrepancy was output. However, for example, the degree of discrepancy between the captured image when inspecting the object to be inspected and the image of a good product (for example, a simulated image of a good product) may be evaluated, and an instruction to reduce the degree of discrepancy may be output. In this way, the usability for inspectors during inspection can be improved, and the captured image can be automatically adjusted by the instruction to reduce the degree of discrepancy.

[0117] Furthermore, in the above embodiment, some or all of the processing performed by the processor may be performed by hardware circuits. Also, the program in the above embodiment may be installed from a program source. The program source may be a program distribution server or a recording medium (for example, a portable recording medium). [Explanation of Symbols]

[0118] 10…Imaging system, 100…Processor system, 102…Processor, 103…Communication device, 104…Input / output device, 105…Imaging device, 110…Memory resources, 111…Simulation image generation program, 112…Inspection engine learning program, 113…Inspection engine evaluation program, 114…Degree of deviation evaluation program, 115…GUI execution program, 116…Simulation image database, 117…Imaging image database, 118…Information database, 119…Inspection engine, 120…Degree of deviation evaluation engine

Claims

1. A training image acquisition system for collecting training images for an inspection engine that inspects an object to be inspected, It has a processor, The aforementioned processor, A first set of simulation images corresponding to multiple variations of the subject being examined is generated. A second set of simulation images is identified, which is a set of images obtained by reducing the number of images from the first set of simulation images, and which can achieve a predetermined inspection performance for the evaluation set by training the inspection engine. The system receives the image acquisition data of the subject to be inspected, which corresponds to each image in the second set of simulation images. The aforementioned group of captured images is registered as a group of learning images for the inspection engine. A system for collecting images for learning purposes.

2. In the learning image acquisition system according to claim 1, The aforementioned processor, Each image from the second set of simulation images is displayed side by side with the candidate image received as a candidate for the image of the subject to be inspected, which corresponds to the simulation image. A system for collecting images for learning purposes.

3. In the learning image acquisition system according to claim 1, The aforementioned processor, The degree of discrepancy between the images in the second set of simulation images and the candidate images received as candidates for imaging images of the inspection target to be associated with the images is calculated for a predetermined evaluation item. The candidate images whose degree of deviation is below a predetermined threshold are registered as images in the learning image group. A system for collecting images for learning purposes.

4. In the learning image acquisition system according to claim 3, The aforementioned processor, If the calculated deviation is not below the threshold, the system displays a correction instruction for the captured image to reduce the deviation. A system for collecting images for learning purposes.

5. In the learning image acquisition system according to claim 4, The aforementioned processor, The degree of discrepancy between the images in the second set of simulation images and the candidate images for each of the multiple evaluation items is calculated. If there are multiple evaluation items where the calculated deviation is not below the threshold, the system will display correction instructions for the captured image to reduce the deviation for those multiple evaluation items. A system for collecting images for learning purposes.

6. In the learning image acquisition system according to claim 5, The aforementioned processor, The candidate image is separated into multiple segmented images by semantic segmentation processing, and the degree of discrepancy between the images in the second group of simulation images and the candidate image is calculated for evaluation items that correspond to the regions of the segmented images. A system for collecting images for learning purposes.

7. In the learning image acquisition system according to claim 1, The aforementioned processor, The second set of simulation images is identified by repeatedly reducing the number of images from the first set of simulation images, training the inspection engine with the reduced set of images, and evaluating the inspection performance of the inspection engine. A system for collecting images for learning purposes.

8. In the learning image acquisition system according to claim 1, The aforementioned processor, Multiple simulation images are generated for at least one variation of the subject to inspection, and one simulation image selected by the user from among the multiple simulation images for one variation is designated as the first simulation image group for that variation. A system for collecting images for learning purposes.

9. In the learning image acquisition system according to claim 1, The aforementioned processor, Based on the captured images of the subject to be inspected, the variation of the subject to be inspected is estimated. A system for collecting images for learning purposes.

10. In the learning image acquisition system according to claim 1, The aforementioned processor, A group of simulation images corresponding to multiple variations of the subject to inspection is generated, and based on the feature quantities of the simulation image group and the feature quantities of the captured image of the subject to inspection, the group of simulation images is converted to an image quality close to that of the captured image, and this group of images is defined as the first simulation image group. A system for collecting images for learning purposes.

11. In the learning image acquisition system according to claim 1, The aforementioned processor, The first set of simulation images is clustered based on the features of the first set of simulation images, and images to be deleted from the first set of simulation images are determined based on the size of the clusters and the number of images belonging to each cluster. A system for collecting images for learning purposes.

12. In the learning image acquisition system according to claim 3, The aforementioned processor, If the calculated degree of deviation is not less than or equal to the threshold, A simulation image similar to the candidate image is selected from the second group of simulation images, the degree of deviation between the candidate image and the selected simulation image for predetermined evaluation items is calculated, and the candidate image whose degree of deviation is below a predetermined threshold is registered as an image in the group of training images, or The system issues an instruction to change the subject used to capture the candidate images to a different subject. A system for collecting images for learning purposes.

13. A method for collecting training images using a training image collection system for collecting training images for an inspection engine that inspects an object to be inspected, The aforementioned learning image acquisition system is A first set of simulation images corresponding to multiple variations of the subject being examined is generated. A second set of simulation images is identified, which is a set of images obtained by reducing the number of images from the first set of simulation images, and which can achieve a predetermined inspection performance for the evaluation set by training the inspection engine. The system receives the image acquisition data of the subject to be inspected, which corresponds to each image in the second set of simulation images. The aforementioned group of captured images is registered as a group of learning images for the inspection engine. Method for collecting training images.

14. In the learning image acquisition method described in claim 13, The aforementioned learning image acquisition system is Each image from the second set of simulation images is displayed side by side with the candidate image received as a candidate for the image of the subject to be inspected, which corresponds to the simulation image. Method for collecting images for training.

15. A learning image acquisition program that causes a computer to execute the learning image acquisition method described in claim 13 or 14.

Images