Inspection device, unit selection device, inspection method, and inspection procedure
Through the combination method of two inferred models, the learning data of defect-free products extracts the area of attention and combines the low-dimensional spatial projection feature map, the problems of high data collection cost and difficult to guarantee accuracy are solved, and efficient product appearance inspection is achieved.
Patent Information
- Application Number
- CN202180035168.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-06-23
- Filing Date
- 2021-05-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-05-28
AI Technical Summary
In the prior art, generating a speculative model for product appearance inspection requires a large amount of sample data, which leads to high data collection costs and is difficult to ensure inspection accuracy when there are many types of samples.
Using the combination method of two speculative models, the first model extracts the areas of concern that may have defects through learning data of defect-free products, and the second model generates feature maps through low-dimensional spatial projection, and determines the results of the two models. To improve inspection accuracy.
It effectively reduces the cost of data collection, improves the accuracy of product appearance inspection, and is suitable for products of various appearance types.
Smart Images

Figure CN115605746B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an inspection device, a unit selection device, an inspection method and an inspection program. Background Art
[0002] At present, in the scenario of manufacturing products such as production lines, the following technology is used: the manufactured products are observed by image sensors, and the products are checked for defects based on the obtained captured images. For example, Patent Document 1 proposes an inspection device that determines whether the inspection object captured in the image is normal or abnormal based on a learned first neural network, and when the inspection object is determined to be abnormal, the type of the abnormality is classified based on a learned second neural network.
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-026982 Summary of the invention
[0006] Technical problem to be solved by the invention
[0007] In the case of a machine learning model using a neural network or the like, machine learning can be implemented using learning data to generate an inference model that has acquired the ability to perform the desired inference task. Therefore, by implementing machine learning using images of products that have been determined to have defects as learning data, an inference model that can be used for product appearance inspection can be generated (e.g., the above-mentioned patent document 1). In addition, such an inference model is not limited to methods using machine learning, and can also be generated by other methods such as manual methods. However, the inventors of the present invention have found that the inspection method using the inference model generated based on the learning data has the following problems.
[0008] As one of the factors that affect the performance of the inference model (i.e., the accuracy of the appearance inspection), the quality of the learning data used in the generation of the inference model can be cited. Basically, the more complex the collected learning data is (in other words, the more diverse the conditions of the defects represented in the learning data), the more the performance of the generated inference model can be improved. However, the more types of backgrounds there are, including the types of defects and the appearance of the product, the more combinations of defect types and background types there are, and the more the number and types of samples collected as learning data are expected to increase. Therefore, in the object environment where the inspection is to be carried out, it is costly to collect learning data in advance based on all conditions, which is unrealistic.
[0009] One aspect of the present invention has been made in view of such circumstances, and an object of the present invention is to provide a technology for improving the appearance inspection accuracy of an estimation model while suppressing the cost of collecting learning data.
[0010] Technical solutions to solve problems
[0011] The present invention adopts the following configuration in order to solve the above-mentioned technical problems.
[0012] That is, an inspection device according to one aspect of the present invention comprises: a data acquisition unit for acquiring an object image of an object product to be inspected; a region extraction unit for extracting a region of interest from the acquired object image using a first inference model, the first inference model being generated in a manner to obtain an ability to extract a region with a probability of capturing a defect as a region of interest based on first learning data, the first learning data including a plurality of first learning images of defect-free products obtained in an environment to be inspected; a model operation unit for performing operation processing of a second inference model using an extraction result of the region of interest for the object image based on the first inference model, generating the second inference model in a manner to obtain an ability to determine the likelihood of a defect based on second learning data including a plurality of second learning images in which defects are captured, the operation processing of the second inference model including generating a plurality of feature maps of different dimensions by performing multiple projection processing on the object image, each projecting the image onto a space of lower dimensions, and the extraction result of the region of interest is merged into at least one of the plurality of feature maps during the operation processing of the second inference model; a determination unit for determining whether the object product has a defect based on the operation result of the second inference model; and an output unit for outputting information indicating a result of determining whether the object product has a defect.
[0013] In this configuration, in addition to the second inference model that has the ability to determine the likelihood of defects, the appearance inspection also uses a first inference model generated based on the first learning data, and the first learning data includes multiple first learning images of defect-free products obtained in the object environment where the inspection is performed. By using the first inference model specialized for the object environment, it is expected to improve the accuracy of the appearance inspection of the object product. In addition, in this configuration, the calculation processing of the second inference model includes performing multiple projection processing on the object image to a low-dimensional space respectively, thereby generating multiple feature maps of different dimensions. The extraction result of the first inference model for the area of interest is merged into at least any one of the multiple feature maps during the calculation processing of the second inference model. Which feature map the extraction result of the first inference model for the area of interest is merged into can be determined in a manner that makes the determination accuracy of whether the object product has a defect as high as possible. Therefore, according to this configuration, for products that may have multiple appearances, it can be expected to improve the appearance inspection accuracy of the inference model.
[0014] In the inspection device involved in the above-mentioned aspect, the second inference model may include an encoder and a decoder, the encoder converts an image into a feature quantity, the decoder decodes the image from the feature quantity, and the decoded image represents the likelihood of a defect for each pixel. The encoder may include a plurality of encoding units connected in series from the input side. Each of the encoding units may be configured to convert the input data into a feature map having a lower dimension than the input data. The multiple projection processing performed on the object image may be configured by performing the operation processing of each of the encoding units in sequence from the input side. Moreover, the extraction result of the area of interest may be merged into the feature map generated by at least one of the plurality of encoding units during the operation processing of the second inference model. According to this configuration, the dimension in which the extraction result of the area of interest of the first inference model is merged can be easily changed, thereby making it easy to operate the improvement of the accuracy of the appearance inspection of the inference model.
[0015] The inspection device according to the first aspect may also include a selection unit, which selects at least one object encoding unit from the plurality of encoding units. Furthermore, the extraction result of the focus area may be incorporated into the feature map generated by the selected at least one object encoding unit during the operation process of the second inference model. According to this configuration, the appearance inspection accuracy of the inference model can be optimized by selecting the object encoding unit.
[0016] In the inspection device according to the first aspect, the selection unit may acquire an evaluation image of an evaluation product including a predetermined defect, extract a focus area from the acquired evaluation image using the first inference model, and repeatedly perform arithmetic processing of the second inference model using the extraction result of the focus area for the evaluation image while changing the coding unit assumed to be selected as the target coding unit, thereby trial-facilitating, for each coding unit, a determination of whether the evaluation product has a defect for the evaluation image when each coding unit is selected as the target coding unit, and selecting an optimal coding unit as the target coding unit based on the trial result. According to this configuration, the appearance inspection accuracy of the inference model can be automatically optimized.
[0017] In the inspection device according to the first aspect, the selection unit may acquire an evaluation image of an evaluation product including a predetermined defect, extract a focus area from the acquired evaluation image using the first inference model, and repeatedly perform arithmetic processing of the second inference model using the extraction result of the focus area for the evaluation image while changing the coding unit assumed to be selected as the target coding unit, thereby trial-assuming for each coding unit whether the evaluation product of the evaluation image has a defect when each coding unit is selected as the target coding unit, output the trial result to an operator, and accept the selection of the at least one target coding unit from the plurality of coding units by the designation of the operator. According to this configuration, the appearance inspection accuracy of the inference model can be optimized by operation.
[0018] In the inspection device according to the first aspect, the encoder may be composed of a convolutional neural network, and each of the encoding units may be composed of one or more convolutional layers and pooling layers. According to this structure, the second inference model can be easily installed.
[0019] In the inspection device according to the first aspect, the inspection device may further include a setting unit, the setting unit setting a parameter value, the parameter defining the degree of extracting the region where the defect is captured as the region of interest. Furthermore, the region extraction unit may also extract the region of interest based on the set parameter value. According to this configuration, the degree of reflecting the inference result of the first inference model can be adjusted, thereby optimizing the accuracy of the appearance inspection of the inference model in the target environment.
[0020] In the inspection device involved in the above-mentioned aspect, the first inference model may include an encoder and a decoder, the encoder converting an image into a feature quantity, and the decoder decoding the image from the feature quantity. The region extraction unit may also use the encoder and the decoder of the first inference model to generate an object decoded image for the object image, and calculate the differential region between the generated object decoded image and the object image. The parameters may also include an expansion parameter, which specifies the degree to which the differential region is expanded. Moreover, the region extraction unit may also expand the calculated differential region according to the set value of the expansion parameter, thereby obtaining the extraction result of the region of interest. According to this configuration, the degree to which the inference result of the first inference model is reflected can be easily adjusted.
[0021] In the inspection device according to the first aspect, the extraction result of the focus area may be constituted by an attention map. According to this configuration, since both the feature map and the attention map are image data, it is easy to implement a merging process of the extraction results.
[0022] In the inspection device according to the first aspect, the first inference model can be generated by training using machine learning of the first learning data to obtain the ability to extract a region with a probability of capturing the defect as a region of interest. According to this configuration, the first inference model can be easily installed.
[0023] In the inspection device according to the above aspect, the second estimation model may be generated by training to obtain the ability to determine the likelihood of a defect through machine learning using the second learning data. According to this configuration, the second estimation model can be easily implemented.
[0024] As other aspects of the inspection device involved in the above-mentioned aspects, one aspect of the present invention may be an information processing method for realizing the above-mentioned aspects, a program, or a computer-readable storage medium storing such a program. Here, a computer-readable storage medium refers to a medium that stores information such as a program through electrical, magnetic, optical, mechanical or chemical action.
[0025] For example, one aspect of the present invention involves an inspection method in which a computer executes the following steps: acquiring an object image of an object product to be inspected; extracting a region of interest from the acquired object image using a first inference model, wherein the first inference model is generated in a manner that obtains the ability to extract a region with a probability of capturing a defect as a region of interest based on first learning data, wherein the first learning data includes multiple first learning images of defect-free products obtained in an object environment to be inspected; performing computational processing of a second inference model on an extraction result of the region of interest for the object image based on the first inference model, generating the second inference model in a manner that obtains the ability to determine the likelihood of a defect based on second learning data including multiple second learning images in which defects are captured, wherein the computational processing of the second inference model includes generating multiple feature maps of different dimensions by performing multiple projection processing on the object image, each of which is projected onto a low-dimensional space, wherein the extraction result of the region of interest is merged into at least any one of the multiple feature maps during the computational processing of the second inference model; determining whether the object product has a defect based on the computational result of the second inference model; and outputting information indicating the result of determining whether the object product has a defect.
[0026] In addition, for example, one aspect of the present invention involves an inspection program for causing a computer to execute the following steps: acquiring an object image of an object product to be inspected; extracting a region of interest from the acquired object image using a first inference model, wherein the first inference model is generated in a manner that obtains the ability to extract a region with a probability of capturing a defect as a region of interest based on first learning data, wherein the first learning data includes multiple first learning images of defect-free products obtained in an object environment where the inspection is to be performed; performing computational processing of a second inference model on the object image using an extraction result of the region of interest based on the first inference model, generating the second inference model in a manner that obtains the ability to determine the likelihood of a defect based on second learning data including multiple second learning images in which defects are captured, wherein the computational processing of the second inference model includes generating multiple feature maps of different dimensions by performing multiple projection processing on the object image, each of which is projected onto a lower-dimensional space, wherein the extraction result of the region of interest is merged into at least one of the multiple feature maps during the computational processing of the second inference model; determining whether the object product has a defect based on the computational result of the second inference model; and outputting information indicating the result of determining whether the object product has a defect.
[0027] (Effects of the Invention)
[0028] According to the present invention, it is possible to improve the accuracy of appearance inspection of an estimated model. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 An example of a scenario to which the present invention is applicable is schematically illustrated.
[0030] Figure 2 An example of the hardware configuration of the model generation device according to the embodiment is schematically illustrated.
[0031] Figure 3 An example of the hardware configuration of the inspection device according to the embodiment is schematically illustrated.
[0032] Figure 4 An example of the software configuration of the model generation device according to the embodiment is schematically illustrated.
[0033] Figure 5 An example of the software configuration of the inspection device according to the embodiment is schematically illustrated.
[0034] Figure 6 This is a flowchart showing an example of a processing procedure related to generation of a first estimation model by the model generation device according to the embodiment.
[0035] Figure 7 This is a flowchart showing an example of a processing procedure related to generation of a second estimation model by the model generation device according to the embodiment.
[0036] Figure 8 This is a flowchart showing an example of a processing procedure related to selection of a target coding unit performed by the inspection device according to the embodiment.
[0037] Fig. 9 This is a flowchart showing an example of a processing procedure related to the appearance inspection by the inspection device according to the embodiment.
[0038] Fig.10 An example of a second estimation model according to another embodiment is schematically illustrated.
[0039] Fig.11 An example of the hardware configuration of a cell selection device according to another embodiment is schematically illustrated.
[0040] Fig.12 An example of a software configuration of a cell selection device according to another embodiment is schematically illustrated. DETAILED DESCRIPTION
[0041] In one aspect of the present invention, two inference models, a first inference model and a second inference model, are used in the appearance inspection. The first inference model is configured to extract a region with a high probability of a defect based on the view of the appearance of a non-defective product obtained in the target environment, and the second inference model is configured to determine the presence or absence of a defect regardless of the target environment. Based on the second inference model, it is possible to determine whether the target product has a defect. However, due to the influence of the product condition shown in the target environment, errors may occur in the determination using only the second inference model. For example, because the appearance of the product is similar to the defect pattern, the appearance of the product may be mistaken for a defect.
[0042] Therefore, the first inference model is also used in the appearance inspection. According to the first inference model, it is possible to preferentially extract image areas that will not appear in defect-free products in the target environment, that is, areas with a high probability of defects in the target environment. Therefore, by fusing the inference results of the two inference models (that is, narrowing the range of defects detected by the first inference model, and determining whether there are defects in the range by the second inference model), the accuracy of the appearance inspection can be optimized. In other words, by suppressing the cost of collecting learning data, even if the inference accuracy of at least one of the inference models is slightly lower, it is expected to improve the accuracy of the appearance inspection in the target environment.
[0043] However, the appearance of products varies. For example, in the case of a product with a patterned appearance (wood grain, etc.), it is sometimes easier to determine whether the product has a normal appearance or a defect by referring to a larger range than by narrowing the range of determination. On the other hand, for example, in the case of a product with a simple appearance in which a design is partially implemented, it is sometimes easier to determine whether the product has a normal appearance or a defect by narrowing the range of determination. Therefore, when the inference results of the two inference models are simply merged without considering the appearance of the product, it is conceivable that the normal appearance of the product, such as the outline and shape, may be misjudged as a defect, and the accuracy of the appearance inspection is reduced. Therefore, in one aspect of the present invention, during the computational processing of the second inference model, the extraction result of the focus area of the first inference model is merged into at least one of the multiple feature maps generated by the second inference model.
[0044] Hereinafter, an embodiment (hereinafter, also referred to as "the present embodiment") according to one aspect of the present invention will be described with reference to the accompanying drawings. However, the present embodiment described below is merely an illustration of the present invention in all aspects. Of course, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, a specific configuration that conforms to the embodiment may also be appropriately adopted. It should be noted that, in the present embodiment, natural language is used to describe the data that appears, but, more specifically, it is specified using a simulation language, instructions, parameters, machine language, etc. that can be recognized by a computer.
[0045] §1 Application examples
[0046] Figure 1 An example of a scenario in which the present invention is applicable is schematically illustrated. Figure 1 As shown, the appearance inspection system 100 according to the present embodiment includes a model generation device 1 and an inspection device 2 .
[0047] The model generation device 1 involved in the present embodiment is a computer configured to generate a first inference model 5 and a second inference model 6 that can be used in appearance inspection. The first inference model 5 is generated in a manner to obtain the ability to extract an area with a probability of capturing a defect as a focus area based on the first learning data 30, and the first learning data 30 includes a plurality of first learning images of non-defective products obtained in the object environment where the inspection is performed. The second inference model 6 is generated in a manner to obtain the ability to determine the likelihood of a defect based on the second learning data 35 including a plurality of second learning images of defects. As long as each inference model (5, 6) is generated in a manner that can obtain each ability based on each learning data (30, 35), the generation method of each inference model (5, 6) is not particularly limited and can be appropriately selected according to the embodiment. For example, each inference model (5, 6) can also be generated by manual operation based on each learning data (30, 35). In the present embodiment, machine learning is used in the generation method of each inference model (5, 6). As a result, each inference model (5, 6) can be easily installed.
[0048] Specifically, the model generation device 1 obtains the first learning data 30, which includes a plurality of good product images 31 of defect-free products obtained in the object environment where the inspection is to be carried out. The good product image 31 is an example of the first learning image. The model generation device 1 trains the first inference model 5 by machine learning using the first learning data 30 to obtain the ability to extract an area with a probability of capturing a defect (i.e., an area containing features originating from the defect) as a focus area from the provided image. Thus, based on the first learning data 30, a first inference model 5 that has been learned and has acquired this ability is generated.
[0049] In addition, the model generation device 1 obtains the second learning data 35 including a plurality of defect images 371 in which defects are captured. The defect image 371 is an example of a second learning image. The model generation device 1 trains the second inference model 6 by machine learning using the second learning data 35 to obtain the ability to determine the likelihood of defects for a provided image (e.g., an image of a product). Thus, based on the second learning data 35, a second inference model 6 that has been learned and has acquired this ability is generated.
[0050] On the other hand, the inspection device 2 is a computer configured to perform an appearance inspection of a product using the two generated inference models (5, 6). Specifically, the inspection device 2 acquires an object image 221 of the object product R to be inspected. In the present embodiment, the camera S is appropriately arranged at a position where the object product R can be photographed, and the inspection device 2 is connected to the camera S. The inspection device 2 acquires the object image 221 from the camera S.
[0051] The inspection device 2 extracts a region of interest from the acquired object image 221 using the first inference model 5. Next, the inspection device 2 performs a computational process of the second inference model 6 using the extraction result of the region of interest for the object image 221 based on the first inference model 5. The computational process of the second inference model 6 is configured to include generating a plurality of feature maps 63 of different dimensions by performing a plurality of projection processes on the object image 221, each of which is projected onto a space of lower dimensions. Each projection process is configured to project input data onto a space of lower dimension than the input data.
[0052] The composition of generating multiple feature maps 63 of different dimensions by multiple projection processes can be appropriately determined according to the implementation method. For example, the operation processing of the second inference model 6 may also include N (N is an integer greater than 2) projection processes performed in series. That is, the first projection process can be configured to project the object image 221 into a low-dimensional space. Moreover, the second and subsequent projection processes (kth projection process) can be configured to project the results of the previous projection process (k-1th projection process) into a lower-dimensional space.
[0053] The extraction result of the first inference model 5 on the target area is incorporated into at least one of the plurality of feature maps 63 during the calculation process of the second inference model 6. The inspection device 2 determines whether the target product R has a defect based on the calculation result of the second inference model 6. Furthermore, the inspection device 2 outputs information indicating the determination result of whether the target product R has a defect. Thus, the inspection device 2 according to the present embodiment uses each inference model (5, 6) to perform the appearance inspection of the product.
[0054] As described above, in the present embodiment, in the appearance inspection of the target product R, in addition to the second estimation model 6 having the ability to determine the likelihood of a defect, the first estimation model 5 is also used, which is generated based on a plurality of images of good products 31 obtained in the target environment where the inspection is performed. By using the first estimation model 5 specialized for the target environment, it is expected that the accuracy of the appearance inspection of the target product R will be improved. In addition, in the present embodiment, the extraction result of the region of interest by the first estimation model 5 is merged into at least one of the plurality of feature maps 63 of different dimensions during the operation process of the second estimation model 6. The feature map to which the extraction result of the region of interest by the first estimation model 5 is merged is determined in such a way that the accuracy of the determination of whether the target product has a defect is as high as possible. As for which feature map to merge, it can be determined by the manufacturer of the inspection device 2 through repeated experiments in advance so that it can correspond to a certain degree of diverse appearances of the target product, or it can be determined through experiments in the target environment where the inspection device 2 is used so that a high-accuracy defect determination result can be obtained corresponding to a target product with a specific appearance. The operation process of the second estimation model 6 can also be configured to change the dimension to which the extraction result of the region of interest by the first estimation model 5 is merged. In this case, the range of likelihood of determining a defect can be adjusted at the dimensional level, and as a result, it can be expected that appropriate merging processing can be performed for the appearance of the target product R. Therefore, according to this embodiment, for the target product R that may have a variety of appearances, it can be expected that the appearance inspection accuracy can be improved by using two inference models (5, 6).
[0055] In addition, Figure 1 In the example, the model generation device 1 and the inspection device 2 are connected to each other via a network. The type of network can be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between the model generation device 1 and the inspection device 2 is not limited to such an example, and can be appropriately selected according to the embodiment. For example, data can be exchanged between the model generation device 1 and the inspection device 2 using a storage medium.
[0056] In addition, Figure 1 In the example of , the model generation device 1 and the inspection device 2 are respectively composed of independent computers. However, the configuration of the appearance inspection system 100 involved in this embodiment is not limited to such an example, and can be appropriately determined according to the embodiment. For example, the model generation device 1 and the inspection device 2 can also be an integrated computer. In addition, for example, at least one of the model generation device 1 and the inspection device 2 can also be composed of multiple computers.
[0057] The target product R may be, for example, a product conveyed on a production line of electronic equipment, electronic components, automobile components, medicines, foods, etc. Electronic components may be, for example, substrates, chip capacitors, liquid crystals, windings of relays, etc. Automobile components may be, for example, connecting rods, rotating shafts, engine blocks, electric window switches, panels, etc. Medicines may be, for example, packaged tablets, unpackaged tablets, etc. The target product R may be a final product generated after the manufacturing process is completed, an intermediate product generated in the middle of the manufacturing process, or an initial product prepared before the manufacturing process. Defects may be, for example, damage, dirt, cracks, indentations, burrs, uneven colors, foreign matter mixed in, etc.
[0058] §2 Example of composition
[0059] [Hardware Configuration]
[0060] <Model generation device>
[0061] Figure 2 An example of the hardware configuration of the model generation device 1 according to the present embodiment is schematically illustrated. Figure 2 As shown, the model generation device 1 according to the present embodiment is a computer electrically connected to a control unit 11, a storage unit 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a driver 17. Figure 2 In the specification, the communication interface and the external interface are recorded as "communication I / F" and "external I / F".
[0062] The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. as a hardware processor, and is configured to perform information processing according to a program and various data. The storage unit 12 is an example of a memory, and is composed of, for example, a hard disk drive, a solid state drive, etc. In the present embodiment, the storage unit 12 stores various information such as the model generation program 81, the first learning data 30, the second learning data 35, the first learning result data 121, and the second learning result data 125.
[0063] The model generation program 81 is used to cause the model generation device 1 to execute the information processing (described later) for generating each estimation model (5, 6). Figure 6 and Figure 7) program. The model generation program 81 includes a series of instructions for the information processing. The first learning data 30 is used to generate the first inference model 5. The second learning data 35 is used to generate the second inference model 6. The first learning result data 121 represents information related to the generated first inference model 5. The second learning result data 125 represents information related to the generated second inference model 6. In the present embodiment, each learning result data (121, 125) is generated as a result of executing the model generation program 81. The details will be described later.
[0064] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for wired or wireless communication via a network. The model generation device 1 can use the communication interface 13 to perform data communication with other information processing devices via a network. The external interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, etc., and is an interface for connecting to an external device. The type and number of the external interface 14 can be selected arbitrarily. The model generation device 1 can be connected to a camera for obtaining a learning image via at least one of the communication interface 13 and the external interface 14.
[0065] The input device 15 is a device for inputting, such as a mouse and a keyboard. The output device 16 is a device for outputting, such as a display and a speaker. An operator such as a user can operate the model generating device 1 using the input device 15 and the output device 16 .
[0066] The drive 17 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading various information such as programs stored in the storage medium 91. The storage medium 91 is a medium that stores various information such as programs stored in it through electrical, magnetic, optical, mechanical or chemical effects in a manner that enables computers and other devices, machines, etc. to read the stored programs. At least any one of the above-mentioned model generation program 81, the first learning data 30 and the second learning data 35 may also be stored in the storage medium 91. The model generation device 1 may also obtain at least any one of the above-mentioned model generation program 81, the first learning data 30 and the second learning data 35 from the storage medium 91. In addition, in Figure 2 In the embodiment, as an example of the storage medium 91, a disk storage medium such as a CD or a DVD is illustrated. However, the type of the storage medium 91 is not limited to the disk type, and may be a type other than the disk type. As a storage medium other than the disk type, for example, a semiconductor memory such as a flash memory may be cited. The type of the drive 17 may be arbitrarily selected according to the type of the storage medium 91.
[0067] In addition, regarding the specific hardware configuration of the model generation device 1, the components may be appropriately omitted, replaced, and added according to the implementation method. For example, the control unit 11 may also include multiple hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), etc. The storage unit 12 may also be composed of a RAM and a ROM included in the control unit 11. At least one of the communication interface 13, the external interface 14, the input device 15, the output device 16, and the drive 17 may also be omitted. The model generation device 1 may also be composed of multiple computers. In this case, the hardware configuration of each computer may be consistent or inconsistent. In addition, in addition to being designed as an information processing device dedicated to the service provided, the model generation device 1 may also be a general-purpose server device, a PC (Personal Computer), etc.
[0068] <Inspection equipment>
[0069] Figure 3 An example of the hardware configuration of the inspection device 2 according to this embodiment is schematically illustrated. Figure 3 As shown, the inspection device 2 according to the present embodiment is a computer electrically connected to a control unit 21 , a storage unit 22 , a communication interface 23 , an external interface 24 , an input device 25 , an output device 26 , and a driver 27 .
[0070] The control unit 21 to the driver 27 and the storage medium 92 of the inspection device 2 can be respectively configured in the same manner as the control unit 11 to the driver 17 and the storage medium 91 of the above-mentioned model generation device 1. The control unit 21 is configured to include a CPU, RAM, ROM, etc. as a hardware processor, and performs various information processing according to programs and data. The storage unit 22 is composed of, for example, a hard disk drive, a solid state drive, etc. In this embodiment, the storage unit 22 stores various information such as the inspection program 82, the first learning result data 121, and the second learning result data 125.
[0071] The inspection program 82 is used to cause the inspection device 2 to execute the information processing (described later) related to the appearance inspection of the target product R. Figure 8 and Fig. 9 ). The inspection program 82 includes a series of commands for the information processing. At least one of the inspection program 82, the first learning result data 121, and the second learning result data 125 may also be stored in the storage medium 92. In addition, the inspection device 2 may also obtain at least one of the inspection program 82, the first learning result data 121, and the second learning result data 125 from the storage medium 92.
[0072] In addition, Figure 3 In the example, the inspection device 2 is connected to the camera S (photographing device) via the external interface 24. Thus, the inspection device 2 is configured to be able to obtain the object image 221 from the camera S. However, the connection method with the camera S may not be limited to such an example. For example, when the camera S has a communication interface, the inspection device 2 may also be connected to the camera S via the communication interface 23. The type of camera S can be appropriately selected according to the implementation method. The camera S can be, for example, a common RGB camera, a depth camera, an infrared camera, etc.
[0073] In addition, regarding the specific hardware configuration of the inspection device 2, the components can be appropriately omitted, replaced, and added according to the implementation method. For example, the control unit 21 may also include multiple hardware processors. The hardware processor may be composed of a microprocessor, an FPGA, a DSP, etc. The storage unit 22 may also be composed of a RAM and a ROM contained in the control unit 21. At least one of the communication interface 23, the external interface 24, the input device 25, the output device 26, and the driver 27 may also be omitted. The inspection device 2 may also be composed of multiple computers. In this case, the hardware configuration of each computer may be consistent or inconsistent. In addition, in addition to being designed as an information processing device dedicated to the services provided, the inspection device 2 may also be a general-purpose server device, a general-purpose PC, a PLC (programmable logic controller), etc. In addition, the inspection device 2 and the camera S may also be integrally configured as an image sensor.
[0074] [Software composition]
[0075] <Model generation device>
[0076] Figure 4 An example of the software configuration of the model generation device 1 according to the present embodiment is schematically illustrated. The control unit 11 of the model generation device 1 loads the model generation program 81 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the commands included in the model generation program 81 loaded into the RAM through the CPU, thereby controlling each component. Figure 4 As shown, the model generation device 1 according to the present embodiment operates as a computer including a first acquisition unit 111, a first learning unit 112, a second acquisition unit 113, a first learning unit 114, and a storage processing unit 115 as software modules. That is, in the present embodiment, each software module of the model generation device 1 is implemented by the control unit 11 (CPU).
[0077] The first acquisition unit 111 acquires first learning data 30 including a plurality of good product images 31. The first learning unit 112 implements machine learning of the first inference model 5 using the first learning data 30. The second acquisition unit 113 acquires second learning data 35 including a plurality of defective images 371. The second learning unit 114 implements machine learning of the second inference model 6 using the second learning data 35. The storage processing unit 115 generates information related to the first inference model 5 generated by machine learning as first learning result data 121. Similarly, the storage processing unit 115 generates information related to the second inference model 6 generated by machine learning as second learning result data 125. Moreover, the storage processing unit 115 saves each generated learning result data (121, 125) to a specified storage area.
[0078] (An example of the structure of each model and the method of machine learning)
[0079] As long as the above-mentioned calculation processing can be installed, the configuration of each inference model (5, 6) may not be particularly limited and may be appropriately determined according to the implementation method. The first inference model 5 may be appropriately configured to extract an area with a probability of capturing a defect as a region of interest. The second inference model 6 may be appropriately configured to generate multiple feature maps 63 of different dimensions during the calculation process through multiple projection processing, and to determine the likelihood of a defect.
[0080] like Figure 4 As shown, in the present embodiment, the first inference model 5 is configured to include an encoder 51 and a decoder 53, the encoder 51 converts the provided image (input image) into a feature quantity, and the decoder 53 decodes the image based on the feature quantity. The second inference model 6 is configured to include an encoder 61 and a decoder 65, the encoder 61 converts the provided image (input image) into a feature quantity, and the decoder 65 decodes the image based on the feature quantity. The image decoded by the decoder 65 is configured to represent the likelihood of a defect for each pixel. That is, each pixel of the decoded image is configured to represent information related to the degree of whether the object photographed in each pixel is a defect. The encoder 61 includes a plurality of encoding units 62 connected in series from the input side, and each encoding unit 62 is configured to convert the input data into a feature map 63 having a lower dimension than the input data.
[0081] In addition, in the present embodiment, each inference model (5, 6) that obtains the above-mentioned capabilities is generated by machine learning. Therefore, each inference model (5, 6) uses a machine learning model with operation parameters that can be adjusted by machine learning. The composition and type of the machine learning model can be appropriately selected according to the implementation method. In the present embodiment, each inference model (5, 6) uses a neural network. Specifically, the encoder 51 and the decoder 53 of the first inference model 5 are respectively composed of a fully connected neural network. The encoder 61 of the second inference model 6 is composed of a convolutional neural network. According to this composition, each inference model (5, 6) can be easily installed.
[0082] (1) First Speculative Model
[0083] The encoder 51 and the decoder 53 of the first estimation model 5 respectively include an input layer (511, 531), an intermediate (hidden) layer (512, 532), and an output layer (513, 533). Figure 4 In the example of , the encoder 51 and the decoder 53 are composed of three layers, but the number of layers of the encoder 51 and the decoder 53 can be arbitrarily selected. For example, at least one of the encoder 51 and the decoder 53 may include two or more intermediate layers.
[0084] In this embodiment, an image such as the object image 221 is input to the input layer 511 of the encoder 51, and a feature quantity calculated based on the input image is output from the output layer 513. In addition, the feature quantity calculated by the encoder 51 is input to the input layer 531 of the decoder 53, and an image (decoded image) decoded from the input feature quantity is output from the output layer 533. However, as long as the above-mentioned inference processing can be performed, the input and output forms of the encoder 51 and the decoder 53 are not limited to such examples. For example, each input layer (511, 531) may also be configured to receive other information than the above. In addition, each output layer (513, 533) may also be configured to output other information than the above.
[0085] Each layer (511, 512, 513, 531, 532, 533) has one or more neurons (nodes). The number of neurons contained in each layer (511, 512, 513, 531, 532, 533) may not be particularly limited and may be appropriately selected according to the implementation method. The input layer (511, 531) may be appropriately determined based on the input data such as the input image and feature quantity and its form. The output layer (513, 533) may be appropriately determined based on the output data such as the feature quantity and the inference result (such as the decoded image) and its form. Figure 4In the example of , each neuron included in each layer (511, 512, 513, 531, 532, 533) is connected to all neurons in the adjacent layer. However, the connection relationship between each neuron is not limited to such an example and can be appropriately set according to the implementation method. For example, each neuron can also be connected to a specific neuron in the adjacent layer, or to a neuron in a layer other than the adjacent layer.
[0086] Weights (connection weights) are set for each connection of each layer (511, 512, 513, 531, 532, 533). A threshold is set in each neuron. Basically, the output of each neuron is determined by whether the sum of the products of each input and each weight exceeds the threshold. The threshold can also be expressed by an activation function. In this case, the output of each neuron is determined by inputting the sum of the products of each input and each weight into the activation function and performing the operation of the activation function. The type of activation function can be selected arbitrarily. The weights of the connections between the neurons contained in each layer (511, 512, 513, 531, 532, 533) and the threshold of each neuron are examples of operation parameters used in the operation processing of the encoder 51 and the decoder 53.
[0087] In the machine learning of the above-mentioned first inference model 5, the first learning unit 112 uses each good image 31 as training data and correct answer data. That is, the first learning unit 112 inputs the good image 31 into the input layer 511 of the encoder 51, and performs the forward propagation operation processing of the encoder 51 and the decoder 53. As a result of this operation processing, the first learning unit 112 obtains the generated decoded image from the output layer 533 of the decoder 53. The first learning unit 112 calculates the error (reconstruction error) between the obtained decoded image and the input good image 31. The first learning unit 112 repeatedly adjusts the values of each operation parameter of the encoder 51 and the decoder 53 for each good image 31 to reduce the calculated error. In this way, the learned first inference model 5 can be generated.
[0088] According to the first inference model 5 involved in the present embodiment, the image 31 of a good product used in machine learning and images similar thereto, that is, images of the appearance of a non-defective product (good product) obtained in the environment of the object to be inspected, can be reconstructed with high accuracy. In contrast, for images other than these, such as images in which defects are captured, the accuracy of reconstruction decreases. In other words, an area with a large reconstruction error is an area with a high probability of capturing a defect. Therefore, the first inference model 5 involved in the present embodiment can obtain the ability to extract an area with a high probability of capturing a defect based on the reconstruction error through the above-mentioned machine learning.
[0089] The storage processing unit 115 generates the first learning result data 121 for reproducing the first inference model 5 generated by the above-mentioned machine learning. As long as the first inference model 5 that has been learned can be reproduced, the composition of the first learning result data 121 is not particularly limited and can be appropriately determined according to the implementation method. For example, the first learning result data 121 may also include information representing the values of each operation parameter of the first inference model 5 obtained by the adjustment of the above-mentioned machine learning. Depending on the situation, the first learning result data 121 may also include information representing the structure of the first inference model 5 (encoder 51 and decoder 53). The structure can be determined, for example, based on the number of layers from the input layer to the output layer in the neural network, the type of each layer, the number of neurons contained in each layer, the connection relationship between neurons in adjacent layers, etc. The storage processing unit 115 saves the generated first learning result data 121 to a specified storage area.
[0090] (2) Second Speculative Model
[0091] The encoder 61 of the second estimation model 6 includes a plurality of encoding units 62. Each encoding unit 62 is composed of one or more convolutional layers 621 and pooling layers 623.
[0092] The convolution layer 621 is configured to perform a convolution operation on the provided data. The convolution operation is equivalent to a process of calculating the correlation between the provided data and a specified filter. For example, by performing image convolution, a shading pattern similar to the shading pattern of the filter can be detected from the input image. The pooling layer 623 is configured to perform a pooling process. The pooling process discards the information portion of the position where the response of the provided data is strong relative to the filter, and achieves the invariance of the response relative to the slight position change of the feature appearing in the data. For example, in the pooling process, the maximum value in the filter can also be extracted and the other values can be deleted. Each encoding unit 62 involved in this embodiment is configured to convert the input data into a feature map 63 with a lower dimension through such a convolution layer 621 and a pooling layer 623.
[0093] The number of convolutional layers 621 and pooling layers 623 included in each encoding unit 62 can be determined arbitrarily. Figure 4 In the example of , the encoder 61 does not include components other than the encoding unit 62, and each encoding unit 62 is connected in series. In each encoding unit 62, the convolution layer 621 is arranged at a position closer to the input side than the pooling layer 623. The convolution layer 621 of the encoding unit 62 arranged closest to the input side constitutes the input layer of the encoder 61, and the pooling layer 623 of the encoding unit 62 arranged closest to the output side constitutes the output layer of the encoder 61.
[0094] However, the structure of the encoder 61 and each encoding unit 62 may not be limited to such an example. The configuration and connection relationship of the convolution layer 621 and the pooling layer 623 may be appropriately determined according to the implementation method. For example, when each encoding unit 62 has multiple convolution layers 621 and multiple pooling layers 623, the convolution layers 621 and the pooling layers 623 may be configured alternately. Alternatively, one or more pooling layers 623 may be configured after multiple convolution layers 621 are configured continuously. The encoder 61 may also include constituent elements other than the encoding unit 62. The encoder 61 may also include other types of layers such as normalization layers, exit layers, and fully connected layers. Other types of layers may also be configured in the encoding unit 62.
[0095] On the other hand, the decoder 65 includes an inverse pooling layer 651 and a deconvolution layer 653. The inverse pooling layer 651 is configured to perform the inverse operation of the above-mentioned pooling process, and the deconvolution layer 653 is configured to perform the inverse operation of the above-mentioned convolution process. The number of inverse pooling layers 651 and deconvolution layers 653 included in the decoder 65 can be arbitrarily determined. Figure 4 In the example of , the depooling layer 651 and the deconvolution layer 653 are alternately configured. The depooling layer 651 configured on the input side constitutes the input layer of the decoder 65, and the deconvolution layer 653 configured on the output side constitutes the output layer of the decoder 65. However, the structure of the decoder 65 is not limited to such an example as the encoder 61 described above. The configuration and connection relationship of the depooling layer 651 and the deconvolution layer 653 can be appropriately determined according to the implementation method. The decoder 65 may also include other types of layers other than the depooling layer 651 and the deconvolution layer 653.
[0096] In this embodiment, an image such as the object image 221 is input to the input layer of the encoder 61, and a feature amount (feature map) calculated based on the input image is output from the output layer. In addition, a feature amount calculated by the encoder 61 is input to the input layer of the decoder 65, and an image (determination image) of the estimated result of the likelihood of the defect represented by each pixel, which is decoded from the input feature amount, is output from the output layer. However, as long as the above-mentioned estimation process can be performed, the input and output forms of the encoder 61 and the decoder 65 are not limited to such examples. For example, each input layer may be configured to receive other information than the above-mentioned information. Each output layer may also be configured to output other information than the above-mentioned information.
[0097] The convolution layer 621 has neurons (nodes) corresponding to the convolution operation, and the neurons (nodes) are connected to a part of the area of the output of the layer that is arranged in front of the layer itself (input side). The deconvolution layer 653 also has neurons corresponding to the deconvolution operation. The number of neurons contained in each layer (621, 653) is not particularly limited and can be appropriately selected according to the implementation method. In addition, each neuron of the convolution layer 621 can also be connected to neurons of layers other than adjacent layers (for example, the corresponding deconvolution layer 653 of the decoder 65), such as U-net.
[0098] As with each layer of the first inference model 5, weights (connection weights) are set for each connection of the convolution layer 621 and the deconvolution layer 653. Thresholds are set for each neuron. The threshold can also be expressed by an activation function. The weights of the connections between the neurons included in the convolution layer 621 and the deconvolution layer 653 and the threshold of each neuron are examples of operation parameters used in the operation processing of the encoder 61 and the decoder 65.
[0099] In the present embodiment, the second acquisition unit 113 acquires second learning data 35 that can be used in machine learning of the second inference model 6 having such a structure. As an example, the second learning data 35 is composed of a plurality of learning data sets 36. Each learning data set 36 is composed of a combination of a learning image 37 and a correct answer image 38. In machine learning, the learning image 37 is used as training data, and the correct answer image 38 is used as correct answer data. At least a portion of the learning image 37 uses a defect image 371. Another portion of the learning image 37 may also use an image in which the defect is not captured. In the case where a defect is captured in the learning image 37, the corresponding correct answer image 38 is constructed to represent the range of the defect.
[0100] In the present embodiment, the second learning unit 114 implements machine learning of the second inference model 6 using such second learning data 35. Specifically, the second learning unit 114 inputs the learning images 37 of each learning data set 36 into the input layer of the encoder 61, and performs the forward propagation operation processing of the encoder 61 and the decoder 65. As a result of the operation processing, the second learning unit 114 obtains the generated judgment image from the output layer of the decoder 65. The second learning unit 114 calculates the error between the obtained judgment image and the corresponding correct image 38. The second learning unit 114 repeatedly adjusts the values of each operation parameter of the encoder 61 and the decoder 65 for each learning data set 36 to reduce the calculated error. In this way, the learned second inference model 6 can be generated. In addition, the composition of the correct answer data can be appropriately selected according to the output form of the second inference model 6. For example, in the case where the second inference model 6 is configured to output the recognition result of the category of the defect, the correct answer data may also include information indicating the category of the defect in addition to or instead of the correct answer image 38.
[0101] The storage processing unit 115 generates the second learning result data 125 for reproducing the learned second inference model 6 generated by the above-mentioned machine learning. As long as the learned second inference model 6 can be reproduced, the structure of the second learning result data 125 is not particularly limited, just like the first learning result data 121, and can be appropriately determined according to the implementation method. For example, the second learning result data 125 may also include information representing the values of each operation parameter of the second inference model 6 obtained by the adjustment of the above-mentioned machine learning. Depending on the situation, the second learning result data 125 may also include information representing the structure of the second inference model 6 (encoder 61 and decoder 65). The storage processing unit 115 saves the generated second learning result data 125 to a specified storage area.
[0102] <Inspection equipment>
[0103] Figure 5 An example of the software configuration of the inspection device 2 according to the present embodiment is schematically illustrated. The control unit 21 of the inspection device 2 loads the inspection program 82 stored in the storage unit 22 into the RAM. The control unit 21 interprets and executes the commands contained in the inspection program 82 loaded into the RAM through the CPU, thereby controlling each component. Figure 5 As shown in FIG. 1 , the inspection device 2 according to the present embodiment operates as a computer including a data acquisition unit 211, a region extraction unit 212, a model calculation unit 213, a determination unit 214, an output unit 215, a selection unit 216, and a setting unit 217 as software modules. That is, in the present embodiment, each software module of the inspection device 2 is also implemented by the control unit 21 (CPU) as in the model generation device 1.
[0104] The data acquisition unit 211 acquires an object image 221 of the object product R to be inspected. The region extraction unit 212 has a first inference model 5 by holding the first learning result data 121. The region extraction unit 212 extracts the region of interest from the object image 221 using the first inference model 5. In the present embodiment, the region extraction unit 212 inputs the object image 221 into the encoder 51, and performs forward propagation calculation processing of the encoder 51 and the decoder 53. Thus, the region extraction unit 212 generates an object decoded image 223 for the object image 221 using the encoder 51 and the decoder 53. The region extraction unit 212 calculates the difference region between the generated object decoded image 223 and the object image 221, and generates an extracted image 225 based on the calculation result of the difference region. The extracted image 225 is an example of the extraction result of the region of interest. In the present embodiment, the extracted image 225 can be composed of an attention map that represents the degree of probability of being a defect for each pixel. The feature map 63 and the attention map are both image data. Therefore, by using the attention map to construct the extraction result of the attention area, the merging process of the extraction result can be easily installed.
[0105] The model operation unit 213 has the second inference model 6 by holding the second learning result data 125. The model operation unit 213 uses the extraction result of the interest region (that is, the interest region extracted from the object image 221) to perform the operation processing of the second inference model 6. The operation processing of the second inference model 6 includes performing multiple projection processing on the object image 221 to a low-dimensional space, thereby generating a plurality of feature maps 63 of different dimensions. The extraction result of the interest region is merged into at least one of the plurality of feature maps 63 during the operation processing of the second inference model 6.
[0106] In the present embodiment, the model operation unit 213 inputs the object image 221 into the encoder 61, and performs the forward propagation operation processing of the encoder 61 and the decoder 65. In the process of this operation processing, multiple projection processing is performed on the object image 221 by sequentially performing the operation processing of each encoding unit 62 from the input side. The extraction result of the focus area is merged into the feature map 63 generated by at least one of the multiple encoding units 62 during the operation processing of the second inference model 6. As an example of a merging method, the model operation unit 213 can also appropriately adjust the size of the extracted image 225 to match the size of the feature map 63 of the merged object. In addition, the model operation unit 213 can also merge the extraction result of the focus area into the feature map 63 of the object by multiplying the resized extracted image 225 with the feature map 63 of the object. If not necessary, the resizing process can be omitted.
[0107] The determination unit 214 determines whether the target product R has a defect based on the calculation result of the second estimation model 6. The output unit 215 outputs information indicating the result of the determination whether the target product R has a defect.
[0108] The selection unit 216 selects at least one target coding unit from the plurality of coding units 62. The extraction result of the region of interest can be incorporated into the feature map 63 generated by the selected at least one target coding unit during the operation process of the second estimation model 6. By selecting the target coding unit, the accuracy of the appearance inspection of the two estimation models (5, 6) can be optimized. In addition, the evaluation data set 227 composed of the combination of the evaluation image 228 and the correct answer information 229 can be used in the selection of the target coding unit.
[0109] The setting unit 217 sets the value of a parameter that specifies the degree to which the area where the defect is captured is extracted as the area of interest. The area extraction unit 212 can extract the area of interest based on the value of the set parameter. According to this configuration, the degree to which the inference result of the first inference model 5 is reflected can be adjusted, thereby optimizing the accuracy of the appearance inspection of the two inference models (5, 6) in the object environment. In this embodiment, in order to extract the area of interest, the differential area between the object image 221 and the object decoded image 223 is calculated. Therefore, the parameter can include an expansion parameter that specifies the degree to which the differential area is expanded. The area extraction unit 212 can also expand the calculated differential area according to the value of the set expansion parameter to obtain the extraction result of the area of interest (the extracted image 225 in this embodiment).
[0110] <Other>
[0111] The software modules of the model generation device 1 and the inspection device 2 are described in detail in the action examples described later. In addition, in the present embodiment, an example is described in which the software modules of the model generation device 1 and the inspection device 2 are implemented by a general-purpose CPU. However, part or all of the above-mentioned software modules can also be implemented by one or more dedicated processors. That is, the above-mentioned modules can also be implemented as hardware modules. In addition, regarding the software configuration of the model generation device 1 and the inspection device 2, the software modules can be appropriately omitted, replaced, and added according to the implementation method.
[0112] §3 Action Examples
[0113] [Model generation device]
[0114] (1) Generation of the first inference model
[0115] Figure 6This is a flowchart showing an example of a processing sequence related to the generation (machine learning) of the first inference model 5 performed by the model generation device 1 involved in this embodiment. However, the following processing sequence related to the generation of the first inference model 5 is only an example, and each step can be changed within a possible range. In addition, regarding the following processing sequence related to the generation of the first inference model 5, steps can be appropriately omitted, replaced, and added according to the embodiment.
[0116] (Step S101).
[0117] In step S101 , the control unit 11 operates as the first acquisition unit 111 and acquires the first learning data 30 including a plurality of good product images 31 .
[0118] Each good product image 31 constituting the first learning data 30 can be appropriately generated. For example, a camera S or a camera of the same type is prepared. In addition, a product of the same type as the target product R and without defects is prepared in the target environment for inspection. And, the prepared product is photographed using a camera. Thus, each good product image 31 can be generated. The good product image 31 can also be an original image obtained by a camera. Alternatively, the good product image 31 can also be generated by applying some kind of image processing to the image obtained by a camera.
[0119] The first learning data 30 can be automatically generated by the action of the computer, or can be manually generated by at least partially including the operation of the operator. In addition, the generation of the first learning data 30 can be performed by the model generation device 1, or by other computers other than the model generation device 1. In the case where the first learning data 30 is generated by the model generation device 1, the control unit 11 automatically or manually performs the above-mentioned generation process through the operation of the operator via the input device 15, thereby acquiring the first learning data 30. On the other hand, in the case where the first learning data 30 is generated by other computers, the control unit 11 acquires the first learning data 30 generated by other computers, for example, via a network, a storage medium 91, etc. It is also possible that a part of the good product image 31 of the first learning data 30 is generated by the model generation device 1, and the other good product images 31 are generated by one or more other computers.
[0120] The number of good product images 31 constituting the first learning data 30 may not be particularly limited and may be appropriately determined according to the implementation. In addition, the first learning data 30 is preferably constituted only by good product images 31, but images of defective products may be included as learning images in a part of the first learning data 30. When the first learning data 30 is acquired, the control unit 11 advances the processing to the next step S102.
[0121] (Step S102)
[0122] In step S102, the control unit 11 acts as the first learning unit 112 and implements machine learning of the first inference model 5 using the acquired first learning data 30. Through the machine learning, the control unit 11 trains the first inference model 5 to obtain the ability to extract an area with a probability of capturing a defect as a focus area. In addition, it is not necessary to use all acquired good product images 31 for machine learning of the first inference model 5. The good product images 31 used in the machine learning can be appropriately selected.
[0123] As an example of the processing of machine learning, first, the control unit 11 prepares the neural network of the encoder 51 and the decoder 53 constituting the first inference model 5 as the processing object of machine learning. The structure of each neural network (for example, the number of layers, the type of each layer, the number of neurons contained in each layer, the connection relationship between neurons in adjacent layers, etc.), the initial value of the weight of the connection between each neuron, and the initial value of the threshold of each neuron can be provided by a template or by operator input. In addition, in the case of re-learning, the control unit 11 can also prepare the encoder 51 and the decoder 53 based on the learning result data obtained by machine learning in the past.
[0124] Next, the control unit 11 uses each good product image 31 of the first learning data 30 as training data (input data) and correct answer data (teacher signal, label), and performs training processing of the encoder 51 and the decoder 53. In this training processing, a stochastic gradient descent method, a mini-batch gradient descent method, etc. can be used.
[0125] As an example of training processing, the control unit 11 inputs each good image 31 into the encoder 51, and performs forward propagation calculation processing of the encoder 51 and the decoder 53. That is, the control unit 11 inputs each good image 31 into the input layer 511 of the encoder 51, and performs calculation processing such as ignition determination of neurons included in each layer in sequence from the input side. As a result of this calculation processing, the control unit 11 obtains a decoded image generated corresponding to each good image 31 from the output layer 533 of the decoder 53. The control unit 11 calculates the error between the obtained decoded image and the corresponding good image 31. A loss function can be used in the calculation of the error (loss). The type of loss function used to calculate the error can be appropriately selected according to the implementation method.
[0126] Next, the control unit 11 calculates the gradient of the calculated error. The control unit 11 calculates the error of the value of each operation parameter (the weight of the connection between each neuron, the threshold value of each neuron, etc.) of the encoder 51 and the decoder 53 in sequence from the output side by using the calculated error backpropagation method. The control unit 11 updates the value of each operation parameter of the encoder 51 and the decoder 53 according to each calculated error. The degree of update of the value of each operation parameter can be adjusted according to the learning rate. The learning rate can be provided by the operator's designation or as a set value in the program.
[0127] The control unit 11 adjusts the values of the operation parameters of the encoder 51 and the decoder 53 for each good product image 31 through the above series of update processing so that the sum of the calculated errors becomes smaller. For example, the control unit 11 may repeatedly adjust the values of the operation parameters of the encoder 51 and the decoder 53 through the above series of update processing until the prescribed conditions such as the prescribed number of times of execution and the calculated sum of the errors being below a threshold value are satisfied.
[0128] As a result of the machine learning, the control unit 11 can generate a trained first estimation model 5 that has acquired the ability to extract an area with a probability of capturing a defect as a focus area. Specifically, in the present embodiment, a trained first estimation model 5 is generated that has a high ability to reconstruct the appearance of a non-defective product that appears in the target environment and a low ability to reconstruct other appearances. When the machine learning of the first estimation model 5 is completed, the control unit 11 advances the processing to the next step S103.
[0129] (Step S103)
[0130] In step S103, the control unit 11 operates as the storage processing unit 115 to generate information related to the trained first estimation model 5 (encoder 51 and decoder 53) generated by machine learning as the first learning result data 121. Then, the control unit 11 stores the generated first learning result data 121 in a predetermined storage area.
[0131] The prescribed storage area may be, for example, a RAM in the control unit 11, a storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, etc., and the control unit 11 may store the first learning result data 121 in the storage medium via the driver 17. The external storage device may be, for example, a data server such as NAS (Network Attached Storage). In this case, the control unit 11 may also store the first learning result data 121 in the data server via the network using the communication interface 13. In addition, the external storage device may also be, for example, an external storage device connected to the model generation device 1 via the external interface 14.
[0132] When the storage of the first learning result data 121 is completed, the control unit 11 ends the processing procedure related to the generation of the first estimation model 5 according to this operation example.
[0133] In addition, the generated first learning result data 121 can be provided to the inspection device 2 at any time. For example, the control unit 11 can also transmit the first learning result data 121 to the inspection device 2 as a process of step S103 or separately from the process of step S103. The inspection device 2 can also obtain the first learning result data 121 by receiving the transmission. In addition, for example, the inspection device 2 can also use the communication interface 23 and access the model generation device 1 or the data server via the network to obtain the first learning result data 121. In addition, for example, the inspection device 2 can also obtain the first learning result data 121 via the storage medium 92. In addition, for example, the first learning result data 121 can also be pre-embedded in the inspection device 2.
[0134] Furthermore, the control unit 11 may also update or regenerate the first learning result data 121 by repeating the processing of the above-mentioned steps S101 to S103 regularly or irregularly. When performing the repetition, at least a part of the first learning data 30 used in the machine learning may be appropriately changed, modified, added, deleted, etc. Furthermore, the control unit 11 may also update the first learning result data 121 retained by the inspection device 2 by providing the updated or regenerated first learning result data 121 to the inspection device 2 using any method.
[0135] (2) Generation of the Second Inference Model
[0136] Figure 7This is a flowchart showing an example of a processing sequence related to the generation (machine learning) of the second inference model 6 performed by the model generation device 1 involved in this embodiment. However, the following processing sequence related to the generation of the second inference model 6 is only an example, and each step can be changed within a possible range. In addition, regarding the following processing sequence related to the generation of the second inference model 6, steps can be appropriately omitted, replaced, and added according to the embodiment.
[0137] (Step S151)
[0138] In step S151, the control unit 11 operates as the second acquisition unit 113 to acquire the second learning data 35 including the plurality of defect images 371. In the present embodiment, the second learning data 35 is composed of a plurality of learning data sets 36. Each learning data set 36 is composed of a combination of a learning image 37 and a correct image 38. At least a portion of the learning images 37 uses the defect image 371.
[0139] Each learning data set 36 constituting the second learning data 35 can be appropriately generated. As with the above-mentioned first learning data 30, for example, a camera S or a camera of the same type is prepared. In addition, a product with a defect is prepared. And, the prepared product is photographed by using a camera. In this way, a defect image 371 that can be used as a learning image 37 can be generated. It is also possible to extract only the defective part in the obtained image through image processing. In other words, it is also possible to generate an image in which only the defect is photographed as a defect image 371. In the defect image 371, the appearance of the product may or may not be photographed. In addition, it is also possible to generate an image in which the defect is not photographed as a learning image 37. The learning image 37 can also be appropriately generated by any image processing. In addition, it is also possible to generate a learning image 37 in which the defect is not photographed by photographing a non-defective product with a camera. The learning image 37 can be either an original image obtained by a camera, or it can be generated by applying some image processing to the image obtained by the camera. Furthermore, the generated learning image 37 is associated with a correct image 38 indicating the range of the defect. The correct image 38 can also be generated from the learning image 37 according to the form of the judgment image. In addition, the correct answer image 38 may be generated by manual operation of an operator. Through these processes, each learning data set 36 can be generated.
[0140] The second learning data 35 can be automatically generated by the action of the computer, or can be manually generated by at least partially including the operation of the operator. In addition, the generation of the first learning data 35 can be performed by the model generation device 2, or by other computers other than the model generation device 1. In the case where the second learning data 35 is generated by the model generation device 1, the control unit 11 automatically or manually performs the above-mentioned generation process through the operation of the operator via the input device 15, thereby acquiring the second learning data 35. On the other hand, in the case where the second learning data 35 is generated by other computers, the control unit 11 acquires the second learning data 35 generated by other computers, for example, via a network, a storage medium 91, etc. A part of the learning data set 36 of the second learning data 35 can also be generated by the model generation device 1, and the other learning data sets 36 can be generated by one or more other computers.
[0141] The number of learning data sets 36 constituting the second learning data 35 is not particularly limited and may be appropriately determined according to the embodiment. When the second learning data 35 is acquired, the control unit 11 advances the process to the next step S152.
[0142] (Step S152)
[0143] In step S152, the control unit 11 acts as the second learning unit 114, and uses the acquired second learning data 35 to perform machine learning of the second inference model 6. Through the machine learning, the control unit 11 trains the second inference model 6 to obtain the ability to determine the likelihood of a defect for the provided image. In addition, it is not necessary to use all the acquired learning data sets 36 for the machine learning of the second inference model 6. The learning data set 36 used in the machine learning can be appropriately selected.
[0144] As an example of the processing of machine learning, first, the control unit 11 prepares the neural network of the encoder 61 and the decoder 65 constituting the second inference model 6 as the processing object of machine learning. The structure of each neural network, the initial value of the weight of the connection between each neuron, and the initial value of the threshold of each neuron can be provided by a template or by operator input. In addition, in the case of re-learning, the control unit 11 can also prepare the encoder 61 and the decoder 65 based on the learning result data obtained by machine learning in the past.
[0145] Next, the control unit 11 uses the learning images 37 of each learning data set 36 as training data and the correct answer images 38 as correct answer data to perform training processing of the encoder 61 and the decoder 65. In this training processing, the stochastic gradient descent method, the mini-batch gradient descent method, etc. can be used.
[0146] As an example of training processing, the control unit 11 inputs the learning images 37 of each learning data set 36 into the encoder 61, and performs forward propagation calculation processing of the encoder 61 and the decoder 65. That is, the control unit 11 inputs each learning image 37 into the input layer of the encoder 61, and performs calculation processing such as firing judgment of neurons contained in each layer in sequence from the input side. As a result of this calculation processing, the control unit 11 obtains the judgment image generated corresponding to each learning image 37 from the output layer of the decoder 65. The control unit 11 calculates the error between the obtained judgment image and the corresponding correct image 38 for each learning data set 36. The loss function can be used in the calculation of the error (loss). For example, the error between the judgment image and the correct image 38 can be calculated by IoU (Intersection over Union).
[0147] Next, the control unit 11 calculates the gradient of the calculated error. The control unit 11 calculates the errors of the values of the operation parameters of the encoder 61 and the decoder 65 in order from the output side by using the calculated error gradient through the error back propagation method. The control unit 11 updates the values of the operation parameters of the encoder 61 and the decoder 65 according to the calculated errors. As with the first estimation model 5 described above, the degree of update of the values of the operation parameters can be adjusted according to the learning rate.
[0148] The control unit 11 adjusts the values of the operation parameters of the encoder 61 and the decoder 65 for each learning data set 36 through the above-mentioned series of update processing so as to reduce the sum of the calculated errors. Similar to the above-mentioned first estimation model 5, the control unit 11 may repeatedly adjust the values of the operation parameters of the encoder 61 and the decoder 65 through the above-mentioned series of update processing until the prescribed conditions are satisfied.
[0149] As a result of the machine learning, the control unit 11 can generate a trained second estimation model 6 that has acquired the ability to determine the likelihood of a defect for a given image. When the machine learning of the second estimation model 6 is completed, the control unit 11 advances the process to the next step S153.
[0150] (Step S153)
[0151] In step S153, the control unit 11 acts as the storage processing unit 115 to generate information related to the trained second inference model 6 (encoder 61 and decoder 65) generated by machine learning as the second learning result data 125. In addition, the control unit 11 saves the generated second learning result data 125 to a specified storage area. As in the above-mentioned step S103, the specified storage area may be, for example, a RAM in the control unit 11, a storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage destination of the second learning result data 125 may be the same as or different from the first learning result data 121.
[0152] When the saving of the second learning result data 125 is completed, the control unit 11 ends the processing sequence related to the generation of the second inference model 6 involved in this action example. In addition, similar to the above-mentioned first learning result data 121, the second learning result data 125 can be provided to the inspection device 2 at any time. In addition, the processing of the above-mentioned steps S151 to S153 can be repeated regularly or irregularly. During this repetition, changes, modifications, additions, deletions, etc. of at least a part of the second learning data 35 can be appropriately performed. In addition, the second learning result data 125 maintained by the inspection device 2 can also be updated by providing the second learning result data 125 updated or regenerated by this repetition to the inspection device 2 using any method.
[0153] [Inspection device]
[0154] (1) Selection of target coding unit
[0155] Figure 8 2 is a flowchart showing an example of a processing sequence related to the selection of the object coding unit performed by the inspection device 2 according to the present embodiment. The control unit 21 acts as the selection unit 216 and performs the following processing of steps S201 to S210. However, the following processing sequence related to the selection of the object coding unit is only an example, and each step can be changed within a possible range. In addition, regarding the following processing sequence related to the selection of the object coding unit, steps can be appropriately omitted, replaced, and added according to the embodiment.
[0156] (Step S201)
[0157] In step S201, the control unit 21 obtains an evaluation image 228 of an evaluation product containing a specified defect. In the present embodiment, the control unit 21 obtains an evaluation data set 227 composed of a combination of the evaluation image 228 and the correct answer information 229. The evaluation image 228 can be generated by photographing an evaluation product having a specified defect with a camera, similar to the good product image 31, the learning image 37, etc. Alternatively, the evaluation image 228 can also be generated by any image processing. The correct answer information 229 is appropriately constructed to indicate the range in which the specified defect is photographed. For example, the correct answer information 229 can be constructed in the same manner as the above-mentioned correct answer image 38. The number of evaluation data sets 227 obtained can be determined arbitrarily. When the evaluation data set 227 is obtained, the control unit 21 causes the processing to proceed to the next step S102.
[0158] (Step S202)
[0159] In step S202, the control unit 21 extracts the region of interest from the acquired evaluation image 228 using the first inference model 5. In the present embodiment, the control unit 21 sets the trained first inference model 5 with reference to the first learning result data 121. Next, the control unit 21 inputs the evaluation image 228 into the input layer 511 of the encoder 51 of the first inference model 5, and performs forward propagation calculation processing of the encoder 51 and the decoder 53. As a result of the calculation processing, the control unit 21 obtains a decoded image generated corresponding to the evaluation image 228 from the output layer 533 of the decoder 53.
[0160] The control unit 211 calculates the difference area between the generated decoded image and the evaluation image 228. The control unit 21 can obtain the calculation result of the difference area directly as the extracted image (the extraction result of the focus area). When the expansion parameter is set, the control unit 21 can also expand the calculated difference area according to the value of the set expansion parameter to obtain the extracted image. The expansion processing of the difference area can be composed of, for example, blur processing (i.e., smoothing using a filter), dilation processing, etc. As an example, the expansion parameter can be composed of a filter such as a specified Gaussian. In this case, the control unit 21 can also smooth the difference area using a filter specified according to the value of the expansion parameter to obtain the extracted image. When the focus area in the evaluation image 228 is extracted, the control unit 21 causes the processing to enter the next step S203.
[0161] (Step S203 to Step S209)
[0162] In step S203 , the control unit 21 sets a variable n to 1. The variable n is used to specify the coding unit 62 that is assumed to be selected as the target coding unit.
[0163] In step S204, the control unit 21 sets the trained second inference model 6 with reference to the second learning result data 125. The setting of the second inference model 6 may also be performed before the processing of step S203. The control unit 21 inputs the evaluation image 228 to the input layer of the encoder 61 of the second inference model 6, and performs the forward propagation operation processing up to the n-th encoding unit 62. Thus, the control unit 21 obtains the feature map 63 from the n-th encoding unit 62.
[0164] In step S205, the control unit 21 merges the extracted image obtained by the processing of step S202 into the feature map 63 generated by the nth encoding unit 62. As long as the extraction result of the focus area of the first inference model 5 can be reflected in the determination of the defect likelihood of the second inference model 6, the method of merging the extraction results of the focus area is not particularly limited and can be appropriately determined according to the implementation method. As an example of a simple merging method, the control unit 21 can also appropriately resize the extracted image to match the size of the feature map 63 of the merged object, and multiply the resized extracted image with the feature map 63 of the object. Through this series of processing, the control unit 21 can merge the extraction result of the focus area into the feature map 63 of the object. When this merging method is adopted, the control unit 21 inputs the above-mentioned merging result into the next layer.
[0165] In step S206, the control unit 21 performs forward propagation calculation processing of the remaining part of the second inference model 6 (i.e., the components of the n-th coding unit 62 and thereafter). As a result, the control unit 21 obtains the judgment image for the evaluation image 228 from the output layer of the decoder 65 as the calculation result of the second inference model 6. The control unit 21 determines whether the evaluation product is defective based on the judgment image (the calculation result of the second inference model 6). Through the processing of steps S204 to S206, the control unit 21 can obtain a trial result of determining whether the evaluation product is defective for the evaluation image 228 when the n-th coding unit is selected as the target coding unit.
[0166] In step S207, the control unit 21 uses the correct answer information 229 to evaluate the trial results obtained by the processing up to step S206. The higher the judgment accuracy (i.e., the detection accuracy of the defect), the higher the control unit 21 evaluates the coding unit 62 assumed to be the object coding unit in the trial processing. As long as the evaluation can be performed in this way, the evaluation method and scale can be set arbitrarily. As an example, the control unit 21 can also score the overlap between the correct answer range and the detection range as an evaluation value. The correct answer range is the range in which the specified defect is captured, which is represented by the correct answer information 229. In addition, the detection range is a range that is determined to have captured a defect (or its likelihood is high) based on the judgment image.
[0167] In step S208, the control unit 21 determines the branch target of the process according to the value of the variable n. When the value of the variable n is not N (2 or more), the control unit 21 advances the process to step S209. N represents the number of encoding units 62 included in the encoder 61. In step S209, the control unit 21 increments the value of the variable n by 1. And, the control unit 21 executes the process from step S204 again according to the new value of the variable n. On the other hand, when the value of the variable n is N, the control unit 21 advances the process to step S210.
[0168] In the present embodiment, the control unit 21 repeatedly performs the processes of step S204 to step S206 while changing the value of the variable n through the above-mentioned loop process. Through this series of processes, the control unit 21 can repeatedly perform the calculation process of the second estimation model 6 using the extraction result of the focus area for the evaluation image 228 while changing the encoding unit 62 assumed to be selected as the target encoding unit. And as a result of this iterative process, the control unit 21 can try to determine whether there is a defect in the evaluation product for the evaluation image 228 when each encoding unit 62 is assumed to be selected as the target encoding unit (defect detection) for each encoding unit 62.
[0169] In addition, this trial method is an example. The trial method for defect detection of evaluation products is not limited to such an example, and can be appropriately changed according to the implementation method. In addition, in the above example, in each trial, it is assumed that only one coding unit 62 is the target coding unit. However, in each trial, the number of coding units 62 assumed to be the target coding units is not limited to such an example, and it can also be assumed that two or more coding units 62 are respectively the target coding units.
[0170] (Step S210)
[0171] In step S210, the control unit 21 selects the best coding unit 62 as the target coding unit based on the trial result. In this embodiment, the control unit 21 selects the coding unit 62 with the highest evaluation (i.e., the trial result that gives the highest accuracy) obtained in step S207 as the target coding unit. When the selection of the target coding unit is completed, the control unit 21 ends the processing related to the selection of the target coding unit involved in this operation example. According to this processing, the accuracy of the appearance inspection of the two estimation models (5, 6) can be automatically optimized.
[0172] (2) Appearance inspection
[0173] Fig. 91 is a flowchart showing an example of a processing sequence related to the appearance inspection performed by the inspection device 2 involved in the present embodiment. The following processing sequence related to the appearance inspection is an example of an inspection method. However, the following processing sequence related to the appearance inspection is only an example, and each step can be changed within a possible range. In addition, for the following processing sequence related to the appearance inspection, steps can be appropriately omitted, replaced, and added according to the embodiment.
[0174] (Step S301)
[0175] In step S301, the data acquisition unit 211 of the control unit 21 acquires the object image 221 of the object product R to be inspected. In the present embodiment, the control unit 21 directly acquires the object image 221 from the camera S via the external interface 24. The object image 221 may be either a dynamic image or a static image. In addition, the path for acquiring the object image 221 may not be limited to such an example. As an example of other paths, the camera S may be connected to other computers. In this case, the control unit 21 may also indirectly acquire the object image 221 from the camera S via other computers. When the object image 221 is acquired, the control unit 21 causes the processing to proceed to the next step S302.
[0176] (Step S302)
[0177] In step S302 , the control unit 21 operates as the region extraction unit 212 and extracts a region of interest from the target image 221 using the first estimation model 5 .
[0178] In this embodiment, the control unit 21 sets the trained first inference model 5 with reference to the first learning result data 121 in the same manner as in the above-mentioned step S202. Next, the control unit 21 inputs the object image 221 into the input layer 511 of the encoder 51 of the first inference model 5, and performs the forward propagation operation of the encoder 51 and the decoder 53. As a result of the operation, the control unit 21 obtains the object decoded image 223 generated corresponding to the object image 221 from the output layer 533 of the decoder 53. The control unit 21 calculates the difference area between the generated object decoded image 223 and the object image 221, and generates an extracted image 225 based on the calculation result of the difference area. The extracted image 225 can be constructed as an attention map that represents the degree of probability of being a defect for each pixel.
[0179] Similar to the above-mentioned step S202, the control unit 21 may also obtain the calculation result of the differential area directly as the extracted image 225. Alternatively, when the expansion parameter is set, the control unit 21 may also expand the calculated differential area according to the value of the set expansion parameter to obtain the extracted image 225. According to the method of using the expansion parameter, the degree to which the inference result of the first inference model 5 is reflected in the appearance inspection can be easily adjusted. In addition, the expansion parameter is an example of a parameter (hereinafter also recorded as an extraction parameter) that specifies the degree to which the area where the defect is photographed is extracted as the extraction area. The time of setting the value of the parameter can be arbitrary. For example, the control unit 21 may also accept the specification of the value of the parameter via the input device 25 before or during the execution of the processing related to the appearance inspection involved in this action example. When the extracted image 225 (the extraction result of the area of interest) is obtained, the control unit 21 causes the processing to enter the next step S303.
[0180] (Step S303)
[0181] In step S303 , the control unit 21 operates as the model calculation unit 213 and executes calculation processing of the second estimation model 6 using the extraction result (extracted image 225 ) of the target image 221 based on the first estimation model 5 .
[0182] In the present embodiment, the control unit 21 sets the trained second inference model 6 with reference to the second learning result data 125. The control unit 21 inputs the target image 221 to the input layer of the encoder 61 of the second inference model 6, and performs the forward propagation operation processing up to the coding unit 62 selected as the target coding unit. As a result of the operation processing, the control unit 21 obtains the feature map 63 from the selected coding unit 62. Next, the control unit 21 merges the extracted image 225 into the feature map 63 generated by the selected coding unit 62. The merging method may be the same as the above-mentioned step S205. When a plurality of coding units 62 are selected as the target coding units, the control unit 21 sequentially performs the forward propagation operation processing and the merging processing up to the selected coding unit 62 from the input side. Moreover, the control unit 21 performs the forward propagation operation processing of the remaining part of the second inference model 6. As a result, the control unit 21 obtains the judgment image for the target image 221 from the output layer of the decoder 65 as the operation result of the second inference model 6.
[0183] In the present embodiment, during the operation process of the second inference model 6, the control unit 21 sequentially performs the operation process of each encoding unit 62 of the encoder 61 from the input side. Through the operation process of each encoding unit 62, multiple projection processes are performed on the target image 221, and multiple feature maps 63 with different dimensions are generated as a result. The extracted image 225 is merged into the feature map 63 generated by at least one selected target encoding unit during the operation process of the second inference model 6. When the operation process of the second inference model 6 is completed, the control unit 21 advances the process to the next step S304.
[0184] (Step S304)
[0185] In step S304 , the control unit 21 operates as the determination unit 214 to determine whether the target product R has a defect based on the calculation result of the second estimation model 6 .
[0186] Determining whether there is a defect means: determining the defect detection with reference to the output (calculation result) of the second inference model 6. The form of determination can be appropriately determined based on the output form of the second inference model 6, etc. As an example, determining whether there is a defect can be constituted by determining whether there is a defect, identifying the type of defect, extracting the area where the defect exists (or has the probability), estimating the likelihood of the defect, or a combination thereof. The control unit 21 can obtain the determination image directly as the determination result, or can obtain the determination result by performing some information processing (such as threshold determination) on the determination image. When the determination of whether there is a defect in the object product R is completed, the control unit 21 advances the processing to the next step S305.
[0187] (Step S305)
[0188] In step S305 , the control unit 21 outputs information indicating a result of determination as to whether the target product R has a defect.
[0189] The output destination and the content of the output information can be appropriately determined according to the embodiment.
[0190] The output destination may be, for example, the RAM in the control unit 21, the output device 26, or other computers. The information representing the determination result may also include information obtained by performing certain information processing according to the determination result (e.g., a specific message, a control instruction for the control object device, etc.). As an example, the control unit 21 may also output the determination result of whether the object product R has a defect directly to the output device 26 or other computers. As another example, when it is determined that there is a defect in the object product R, the control unit 21 may also output a warning to the output device 26 or other computers to notify the situation. As yet another example, when the inspection device 2 is connected to a conveyor device that transports the object product R, the control unit 21 may also control the conveyor device according to the determination result to transport defective object products R and non-defective object products R on different production lines.
[0191] When the output of the information indicating the determination result is completed, the control unit 21 ends the processing sequence related to the appearance inspection involved in this action example. In addition, the control unit 21 can also continue to repeatedly perform a series of information processing from step S301 to step S305. The timing of the repetition can be arbitrary. During the repetition, the target product R can be replaced in sequence by a conveyor device, etc. Thus, the inspection device 2 can continuously perform the appearance inspection on the target product R.
[0192] [feature]
[0193] As described above, in the present embodiment, in the appearance inspection of the target product R in steps S301 to S305, in addition to the second estimation model 6 having the ability to determine the likelihood of a defect, the first estimation model 5 specialized for the target environment is used. By using the first estimation model 5, it is expected that the accuracy of the appearance inspection of the target product R will be improved. In addition, in the present embodiment, through the processing of step S303, the extraction result (extracted image 225) of the focus area of the first estimation model 5 is merged into at least one of the multiple feature maps 63 with different dimensions during the operation process of the second estimation model 6. That is, the operation processing of the second estimation model 6 is configured to be able to change the dimension in which the extraction result of the focus area of the first estimation model 5 is merged. As a result, the range of determining the likelihood of a defect can be adjusted at the dimensional level, and as a result, it can be expected that a merging process suitable for the appearance of the target product R can be implemented. For example, for a product with a patterned appearance, a relatively wide range can be referenced, and for a product with a simple appearance, the referenced range can be relatively narrowed. Therefore, according to the present embodiment, for the target product R that may have a variety of appearances, it can be expected that the accuracy of the appearance inspection can be improved by two estimation models (5, 6).
[0194] §4 Variations
[0195] Above, the embodiments of the present invention have been described in detail, but the above description is only an illustration of the present invention in all aspects. It is needless to say that various improvements or modifications can be made without departing from the scope of the present invention. For example, the following changes can be made. In addition, the same reference numerals are used below for the same components as the above-mentioned embodiments, and the description is appropriately omitted for the same points as the above-mentioned embodiments. The following modifications can be appropriately combined.
[0196] <4.1>
[0197] In the above embodiment, the encoder 51 and the decoder 53 of the first inference model 5 use a fully connected neural network. In addition, the encoder 61 of the second inference model 6 uses a convolutional neural network. However, the type of neural network used in each inference model (5, 6) is not limited to such an example, and can be appropriately selected according to the embodiment. For example, the encoder 51 of the first inference model 5 can use a convolutional neural network.
[0198] In addition, in the above-mentioned embodiment, each inference model (5, 6) uses a neural network as a machine learning model. However, the type of machine learning model used in each inference model (5, 6) is not limited to a neural network and can be appropriately selected according to the embodiment. The machine learning method can be appropriately selected according to the type of machine learning model. For example, the encoder 51 and the decoder 53 of the first inference model 5 can be composed of an orthogonal projection matrix using eigenvectors derived by principal component analysis. In this case, the machine learning of the first inference model 5 can include performing principal component analysis.
[0199] In addition, in the above-mentioned embodiment, each inference model (5, 6) is generated by machine learning. However, the generation method of each inference model (5, 6) may not be limited to such an example, and may be appropriately determined according to the embodiment. As an example, each inference model (5, 6) may be generated by manual operation according to a rule base. In this case, each inference model (5, 6) may be constructed by known image processing such as feature extraction. Each encoding unit 62 may be constructed to appropriately include a projection process of dimensionality reduction such as pooling processing and reduction processing. The first inference model 5 may be constructed, for example, to extract a region of interest with a high probability of a defect by a method other than a method based on a reconstruction error by a method such as Japanese Patent Laid-Open No. 2010-203845. In addition, in the above-mentioned embodiment, in the encoder 61 of the second inference model 6, each encoding unit 62 is connected in series. However, the configuration of the encoding unit 62 may not be limited to such an example. As long as a plurality of feature maps 63 of different dimensions can be generated, at least a portion of the encoding units 62 may be configured in parallel. In an extreme case, all encoding units 62 may also be configured in parallel.
[0200] Fig.10 An example of the second inference model 6A involved in this modification is schematically illustrated. The second inference model 6A involved in this modification is configured to perform at least a portion of the multiple projection processing in parallel by having three detection units arranged in parallel. Specifically, the first detection unit includes a first reduction unit 661, one or more feature extraction units 671, and a detection unit 681 in sequence from the input side. The second detection unit includes a second reduction unit 662, one or more feature extraction units 672, and a detection unit 682 in sequence from the input side. The third detection unit includes one or more feature extraction units 673 and a detection unit 683 in sequence from the input side.
[0201] The first reduction unit 661 and the second reduction unit 662 are configured to reduce the size of the input image (i.e., reduce the dimension of the input image). In the method of reducing the size of the input image, for example, a method of thinning out the pixels in the image (e.g., pooling processing), an average pixel method, etc. can be used. The calculation processing of each reduction unit (661, 662) is an example of a projection processing to a low-dimensional space. Between the first reduction unit 661 and the second reduction unit 662, it is configured to generate images of different sizes. As an example, the first reduction unit 661 can be configured to reduce the size of the input image to 1 / 2, and the second reduction unit 662 can be configured to reduce the size of the input image to 1 / 4.
[0202] Each feature extraction unit 671 to 673 is configured to extract features of an image. The process of extracting image features can be, for example, configured by a well-known filter such as a Sobel filter or a Laplace filter. The number of each feature extraction unit 671 to 673 can be appropriately determined according to the implementation method. As an example, one or more feature extraction units 671 to 673 can be configured to extract features of defects and exclude features other than these (i.e., extract candidate regions of defects).
[0203] Each of the detection units 681 to 683 is configured to detect a defect region based on the features of the image obtained by each of the feature extraction units 671 to 673. As an example, when each of the feature extraction units 671 to 673 is configured to extract a candidate region for a defect, each of the detection units 681 to 683 may be configured to detect a candidate region whose size exceeds a threshold value among the extracted candidate regions as a defect region.
[0204] According to each detection unit, the defect detection result can be obtained from multiple images of different dimensions. In addition, in this modification, the reduced image generated by each reduction unit (661, 662) and the image output from each feature extraction unit 671 to 673 are examples of feature maps. Fig.10As shown by the dotted arrow in FIG. 5 , the extracted image 225 obtained by the first inference model 5 can be merged into at least one image (feature map). In the third detection unit without a reduction unit, the feature extraction unit 673 can also perform the calculation processing after merging the extracted image 225 into the object image 221. Fig.10 As shown, the merging process can be implemented by multiplication of images. The selection unit 216 of the inspection device 2 can be configured to set the validity or invalidity of the merging process for each image as a candidate.
[0205] The second inference model 6A involved in this modification further includes a merge determination unit 69, which is configured to merge the defect detection results of the three detection units and determine whether there is a defect based on the result. As an example, the merge determination unit 69 can be configured to adjust the defect areas detected by each detection unit 681 to 683 to match the size of the image and then overlap them, and determine whether there is a defect and the position based on the overlap result. The overlap of defect areas can be configured, for example, by AND operation (extracting only the overlapping part), OR operation (adding the areas), etc.
[0206] The second inference model 6A involved in this variation is configured to output the final defect determination result. As an example, the second inference model 6A can be configured to directly output the determination result of the merge determination unit 69. As another example, the second inference model 6A may also be provided with a defect emphasis unit (not shown), which is configured to process the object image 221 in a manner that emphasizes the defective portion when the merge determination unit 69 determines that a defect exists. The emphasis method can be appropriately determined. As an example, the defect emphasis unit may also adjust the brightness value of each pixel of the object image 221 so that the contrast of the peripheral area of the detected defect becomes higher and the contrast of the area outside the periphery of the defect becomes lower. In this case, the second inference model 6A may be configured to output an image obtained by the defect emphasis unit that emphasizes the defect.
[0207] In addition, in the second inference model 6A involved in this modification, the number of detection units arranged in parallel is not limited to three, and may be two, or may be four or more. In addition, in each detection unit, the image size reduction process may be performed multiple times. That is, the detection unit may also have multiple reduction units.
[0208] In addition, in the above-mentioned embodiment, for example, the dimensions of a part of the plurality of feature maps 63 generated in the second estimation model 6 may be made consistent based on the reasons such as performing a process without dimensionality reduction (e.g., a simple filtering process) or making the number of dimensions reduced consistent when generating feature maps in parallel. In this case, the feature map 63 to which the extraction results of the focus area of the first estimation model 5 are merged may be changed to another feature map 63 of the same dimension. Thus, the feature amount to merge the extraction results of the focus area of the first estimation model 5 may be changed.
[0209] In addition, in the above-mentioned embodiment, the output form of each inference model (5, 6) can be appropriately determined according to the embodiment. For example, the second inference model 6 can also be provided with a determiner instead of the decoder 65. The determiner can be configured to determine whether there is a defect based on the feature quantity obtained by the encoder 61. In this case, each learning data set 36 of the second learning data 35 used in the machine learning can be configured to include correct answer information indicating whether there is a defect instead of the correct answer image 38. The model generation device 1 can appropriately train the second inference model 6 by using such second learning data 35 in the above-mentioned step S152. In addition, in the above-mentioned embodiment, each inference model (5, 6) can be generated by different computers. The first inference model 5 can be generated based on the first learning data 30 obtained at one site. In contrast, the second inference model 6 can be generated based on the second learning data 35 collected from multiple sites.
[0210] <4.2>
[0211] In the above-mentioned embodiment, the extraction parameter may not be limited to the expansion parameter. As an example, the inspection device 2 may also generate a differential image by calculating the difference between the object decoded image 223 and the object image 221. Next, the inspection device 2 may also binarize each pixel of the differential image according to a threshold value (binarization threshold value). Moreover, the inspection device 2 may also delete the set of pixels whose area is below the threshold value (area threshold value) from the set of pixels that appear to have defects through binarization. Through this series of processing, the inspection device 2 may also calculate the differential area between the object decoded image 223 and the object image 221. In this case, the extraction parameter may include at least one of the binarization threshold value and the area threshold value. In addition, in the above-mentioned embodiment, the process of setting the value of the parameter may be omitted. In this case, the setting unit 217 may be omitted from the software configuration of the inspection device 2.
[0212] <4.3>
[0213] In the above-mentioned embodiment, the object coding unit most suitable for the object environment can be automatically selected by the processing of steps S201 to S210. However, the method of selecting the object coding unit may not be limited to such an example. The control unit 21 may also select at least one object coding unit from the plurality of coding units 62 by any method. As an example of another method, the control unit 21 may omit the processing of step S207 in the processing of steps S201 to S210. Furthermore, the control unit 21 may also output the trial result of step S206 to the operator instead of the processing of step S210. The output destination may be selected arbitrarily. Moreover, the control unit 21 may also accept the selection of at least one object coding unit from the plurality of coding units 62 by the designation of the operator. For example, the control unit 21 may also output a list of determination images obtained by each trial to the output device 26. The control unit 21 may also accept the selection of at least one object coding unit from the plurality of coding units 62 by the operator appropriately designating the determination image via the input device 25. The processing of step S201 to step S206, step S208 and step S209 can be performed in the same manner as in the above-mentioned embodiment. According to this processing, the accuracy of the appearance inspection of the two inference models (5, 6) can be optimized by operation. As another example of a method, the control unit 21 can also omit a series of processing of step S201 to step S210 and simply accept the selection of the object coding unit by the operator's designation. In addition, in the above-mentioned embodiment, a series of processing of selecting the object coding unit can be omitted. In this case, the selection unit 216 can be omitted from the software configuration of the inspection device 2.
[0214] <4.4>
[0215] In the above-mentioned embodiment, the extraction result of the area of interest can be merged into the feature map 63 of the object by multiplying the feature map 63 of the object by the extracted image. However, the merging method may not be limited to such an example. As another example, the feature map 63 of the object and the extracted image can be respectively input into other nodes in the next layer of the layer where the feature map 63 of the object is obtained. In this case, the extraction result of the area of interest can be merged into the feature map 63 of the object through the calculation processing of the next layer. In addition, in this case, in the machine learning processing of step S152, the control unit 11 can also input the extracted image obtained by the trained first inference model 5 for the learning image 37 into the layer of the object.
[0216] <4.5>
[0217] In the above-described embodiment or modification, the process of selecting the target coding unit may be executed by a computer other than the inspection device 2 .
[0218] Fig.11An example of the hardware configuration of the unit selection device 7 according to this modification is schematically illustrated. Fig.11 As shown, the unit selection device 7 according to the present modification is a computer electrically connected to a control unit 71, a storage unit 72, a communication interface 73, an external interface 74, an input device 75, an output device 76, and a driver 77. The control unit 71 to the driver 77 and the storage medium 97 can be configured similarly to the control unit 21 to the driver 27 and the storage medium 92 of the above-mentioned inspection device 2.
[0219] In this modification, the storage unit 72 stores various information such as the selection program 87, the first learning result data 121, and the second learning result data 125. The selection program 87 is a program for causing the unit selection device 7 to perform information processing related to the selection processing of the above-mentioned object encoding unit. The selection program 87 includes a series of commands for the information processing. At least any one of the selection program 87, the first learning result data 121, and the second learning result data 125 may also be stored in the storage medium 97. In addition, the unit selection device 7 may also obtain at least any one of the selection program 87, the first learning result data 121, and the second learning result data 125 from the storage medium 92.
[0220] In addition, similar to the inspection device 2, etc., regarding the hardware configuration of the unit selection device 7, it is possible to appropriately omit, replace, and add components according to the implementation method. The unit selection device 7 may also be composed of a plurality of computers. In this case, the hardware configurations of the computers may be consistent or inconsistent. In addition, in addition to being designed as an information processing device dedicated to the service provided, the unit selection device 7 may also be a general-purpose server device, a general-purpose PC, a PLC, etc.
[0221] Fig.12 An example of the software structure of the unit selection device 7 involved in this modification is schematically illustrated. Similar to the above-mentioned inspection device 2, etc., the software structure of the unit selection device 7 is implemented by the control unit 71 executing the selection program 87. By executing the selection program 87, the unit selection device 7 involved in this modification is operated as a computer having a model acquisition unit 219 and a selection unit 216 as a software module. In addition, part or all of the software module can also be implemented by one or more dedicated processors. That is, each of the above modules can also be implemented as a hardware module. In addition, regarding the software structure of the unit selection device 7, the software module can be appropriately omitted, replaced, and added according to the implementation method.
[0222] The control unit 71 operates as the model acquisition unit 219, and acquires each learning result data (121, 125) by an arbitrary method. Thus, the control unit 71 acquires each estimation model (5, 6) as the object of the process of selecting the target coding unit. Next, the control unit 71 operates as the selection unit 216, and selects at least one target coding unit from the plurality of coding units 62. The method of selecting the target coding unit can adopt any of the above methods.
[0223] The result of selecting the object coding unit can be reflected to the inspection device for performing the appearance inspection of the object product R in the object environment by any method. As an example, the control unit 71 can also act as an output unit to notify the inspection device of the selection result of the object coding unit via a network or the like. In the inspection device, the result of selecting the object coding unit can be reflected according to the reception of the notification. As another example, the control unit 71 can also output the selection result of the object coding unit to the output device 76 or the output device of another computer. The operator can also operate the inspection device by referring to the output, so that the result of selecting the object coding unit is reflected to the inspection device. According to this modified example, the load of the inspection device can be reduced.
[0224] Description of Reference Numerals
[0225] 1…model generation device, 11…control unit, 12…storage unit, 13…communication interface, 14…external interface, 15…input device, 16…output device, 17…driver, 81…model generation program, 91…storage medium, 111…first acquisition unit, 112…first learning unit, 113…second acquisition unit, 114…second learning unit, 115…storage processing unit, 121…first learning result data, 125…second learning result data, 2…inspection device, 21…control unit, 22…storage unit, 23…communication interface, 24…external interface, 25…input device, 26…output device, 27…driver, 82…inspection program, 92…storage medium, 211…data acquisition unit, 212…region extraction unit , 213…model calculation unit, 214…determination unit, 215…output unit, 216…selection unit, 217…setting unit, 221…object image, 223…object decoded image, 225…extracted image, 227…data set for evaluation, 228…evaluation image, 229…correct answer information, 30…first learning data, 31…good product image, 35…second learning data, 36…learning data set, 37…learning image, 371…defective image, 38…correct answer image, 5…first inference model, 51…encoder, 53…decoder, 6…second inference model, 61…encoder, 62…encoding unit, 621…convolution layer, 623…pooling layer, 65…decoder, 651…depooling layer, 653…deconvolution layer.
Claims
1. An inspection device comprising: A data acquisition unit, which acquires an object image of a target product to be inspected; a region extraction unit that extracts a region of interest from the acquired object image using a first inference model, wherein the first inference model is generated in a manner that obtains a capability of extracting a region with a probability of capturing a defect as the region of interest based on first learning data, wherein the first learning data includes a plurality of first learning images of non-defective products captured in an environment where inspection is to be performed; a model operation unit, which uses the extraction result of the focus area for the object image based on the first inference model to perform operation processing of a second inference model, and generates the second inference model in a manner of obtaining the ability to determine the likelihood of the defect based on second learning data including a plurality of second learning images in which the defect is captured, wherein the operation processing of the second inference model includes performing multiple projection processing on the object image to a low-dimensional space respectively, thereby generating a plurality of feature maps of different dimensions, and the extraction result of the focus area is merged into at least any one of the plurality of feature maps during the operation processing of the second inference model; a determination unit, which determines whether the target product has a defect based on a calculation result of the second estimation model; as well as The output unit outputs information indicating a result of determining whether the target product has a defect.
2. The inspection device according to claim 1, wherein: The second estimation model includes an encoder and a decoder, the encoder converting an image into a feature quantity, the decoder decoding the image from the feature quantity, and the decoded image indicating the likelihood of a defect for each pixel. The encoder includes a plurality of encoding units connected in series from the input side. Each of the encoding units is configured to convert the input data into a feature map having a lower dimension than the input data, The multi-projection processing is performed on the target image by sequentially performing the arithmetic processing of each of the encoding units from the input side. The extraction result of the focus area is merged into the feature map generated by at least one of the multiple encoding units during the operation processing of the second inference model.
3. The inspection device according to claim 2, wherein: The inspection device further includes a selection unit configured to select at least one target encoding unit from among the plurality of encoding units. The extraction result of the target area is incorporated into the feature map generated by the selected at least one object encoding unit during the operation process of the second inference model.
4. The inspection device according to claim 3, wherein: The selection unit acquires an evaluation image of an evaluation product including a predetermined defect. extracting a focus area from the acquired evaluation image using the first inference model, The calculation processing of the second inference model is repeatedly performed using the extraction result of the focus area for the evaluation image while changing the coding unit assumed to be selected as the target coding unit, thereby trying to determine whether the evaluation product is defective for the evaluation image when each coding unit is selected as the target coding unit for each coding unit, and An optimal coding unit is selected as the target coding unit based on the result of the trial.
5. The inspection device according to claim 3, wherein: The selection unit acquires an evaluation image of an evaluation product including a predetermined defect. extracting a focus area from the acquired evaluation image using the first inference model, While changing the coding unit assumed to be selected as the object coding unit, the operation processing of the second inference model is repeatedly performed using the extraction result of the focus area for the evaluation image, thereby trying to determine whether the evaluation product of the evaluation image when each coding unit is selected as the object coding unit is defective for each coding unit. outputting the results of the trial to an operator, and, A selection of the at least one target encoding unit from among the plurality of encoding units is accepted based on a designation by the operator.
6. The inspection device according to any one of claims 2 to 5, wherein: The encoder is composed of a convolutional neural network. Each of the encoding units is composed of one or more convolutional layers and pooling layers.
7. The inspection device according to any one of claims 1 to 5, wherein: The inspection device further includes a setting unit that sets a value of a parameter that specifies the degree to which a region where the defect is captured is extracted as a region of interest. The region extraction unit extracts the region of interest according to the set parameter value.
8. The inspection device according to claim 7, wherein: The first estimation model includes an encoder and a decoder, the encoder converting an image into a feature quantity, and the decoder decoding the image from the feature quantity. The region extraction unit generates a target decoded image for the target image using the encoder and the decoder of the first estimation model, and calculating a difference area between the generated target decoded image and the target image, The parameters include an expansion parameter, the expansion parameter specifies the degree to which the differential region is to be expanded, The region extraction unit enlarges the calculated difference region according to the set value of the expansion parameter, thereby acquiring an extraction result of the target region.
9. The inspection device according to any one of claims 1 to 5, wherein: The extraction result of the focus area is composed of an attention map.
10. The inspection device according to any one of claims 1 to 5, wherein: The first estimation model is generated by machine learning using the first learning data, and is trained so as to acquire the ability to extract a region with a probability of capturing the defect as a region of interest.
11. The inspection device according to any one of claims 1 to 5, wherein: The second estimation model is generated by machine learning using the second learning data and trained so as to acquire the ability to determine the likelihood of a defect.
12. A method for checking, wherein a computer executes the following steps: Acquire an object image of the object product to be inspected; extracting a region of interest from the acquired object image using a first inference model, the first inference model being generated in a manner capable of extracting a region with a probability of capturing a defect as the region of interest based on first learning data, the first learning data comprising a plurality of first learning images of non-defective products captured in an environment where inspection is to be performed; executing a calculation process of a second estimation model using the extraction result of the target area of the target image based on the first estimation model, generating the second estimation model in a manner that obtains the ability to determine the likelihood of a defect based on second learning data including a plurality of second learning images in which defects are captured, The calculation processing of the second inference model includes performing multiple projection processing on the object image to a low-dimensional space respectively, thereby generating a plurality of feature maps with different dimensions, The extraction result of the focus area is merged into at least any one of the plurality of feature maps during the operation process of the second inference model; determining whether the target product has a defect according to the calculation result of the second inference model; as well as Information indicating a result of determining whether the target product has a defect is output.
13. A checking program for causing a computer to execute the following steps: Acquire an object image of the object product to be inspected; extracting a region of interest from the acquired object image using a first inference model, the first inference model being generated in a manner capable of extracting a region with a probability of capturing a defect as the region of interest based on first learning data, the first learning data comprising a plurality of first learning images of non-defective products captured in an environment where inspection is to be performed; executing a calculation process of a second estimation model using a result of extracting the target area of the target image based on the first estimation model, generating the second estimation model in a manner that obtains the ability to determine the likelihood of a defect based on second learning data including a plurality of second learning images in which defects are captured, The calculation processing of the second inference model includes performing multiple projection processing on the object image to a low-dimensional space respectively, thereby generating a plurality of feature maps with different dimensions, The extraction result of the focus area is merged into at least any one of the plurality of feature maps during the operation process of the second inference model; determining whether the target product has a defect according to the calculation result of the second inference model; as well as Information indicating a result of determining whether the target product has a defect is output.
