Method, program, information processing device, and trained model

Abstract

The objective of the present invention is to improve determination accuracy in product inspection while reducing the workload for model generation in the case of using defective pseudo-generated product images when inspecting multiple types of products. A quality determination device 101 inputs an inspection image I22 of a product of product type B to a quality determination model 10122 trained using training data, and acquires, from the quality determination model 10122, information indicating whether the product of product type B is a non-defective product or a defective product, wherein the quality determination model 10122 takes, as input, a training image set including a defective product image I11 of a product of product type A which is a defective product, a non-defective product image I20 of a product of product type B which is a non-defective product, and a pseudo-defective product image I21 of a product of product type B in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality that occurred in the product of product type B synthesized with the non-defective product image I20, and outputs information indicating whether the product of product type B is a non-defective product or a defective product.

Description

Method, program, information processing device, and trained model

[0001] The present invention relates to a method, a program, an information processing device, and a trained model.

[0002] Visual inspection is one method for inspecting the quality of a product. Visual inspection is sometimes performed automatically by a computer, for example, by inputting photographed images of the product into a trained model to obtain results. For some products, for example, due to a small number of products manufactured, it may be difficult to prepare sufficient images of defective products required for learning to generate a trained model. As a technology for compensating for the lack of images of defective products, Patent Literature 1 describes an image data generation system capable of generating image data of various types of defective products.

[0003] Japanese Patent Application Laid-Open No. 2022-136583

[0004] Generating a trained model requires a large number of steps, such as collecting and annotating images of defective products, training to generate a trained model, and verifying the accuracy of the trained model. In particular, collecting and annotating images of defective products requires a large number of steps, which can lead to insufficient preparation of defective product images or to the need to reduce the number of steps required. When inspecting multiple different product varieties contained in a product, a trained model must be generated for each of the different product varieties. Therefore, it is necessary to collect and annotate images of defective products for each different product variety, and the number of steps increases as the number of product varieties increases.

[0005] Furthermore, the defective product images generated by the image data generation system described in Patent Document 1 are merely pseudo-defective product images. Therefore, if pseudo-defective product images are generated for each of different types of products, the generated images may contain parts that differ from images containing defects that actually occurred in each of the different types of products. If training is performed to generate a trained model using pseudo-defective product images generated in this way, the accuracy of detecting images containing actual defects may decrease.

[0006] Therefore, the present invention aims to improve judgment accuracy while reducing the amount of work required to generate a trained model when using pseudo-generated defective product images during inspection of multiple types of products using a trained model.

[0007] A method according to one aspect of the present invention includes: inputting an inspection image of a second product variety to be inspected into a first trained model trained using training data; the first trained model receives as input a training image set including a first defective product image of a first product variety that is a defective product, a second good product image of a second product variety that is a good product, and a pseudo-defective product image of the second product variety, in which a pseudo-abnormal image indicating a pseudo-abnormality different from an abnormality occurring in the second product variety is superimposed on the second good product image; and outputting information indicating whether the second product variety included in the training image set is a good product or a defective product; and obtaining, from the first trained model, information indicating whether the second product variety to be inspected is a good product or a defective product as an output.

[0008] A program according to another aspect of the present invention causes a computer to execute the following steps: input an inspection image of a second product variety to be inspected into a first trained model that has been trained using training data that takes as input a training image set including a first defective product image of a first product variety that is a defective product, a second good product image of a second product variety that is a good product, and a pseudo-defective product image of the second product variety in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the second product variety has been superimposed on the second good product image, and outputs information indicating whether the second product variety included in the training image set is a good product or a defective product; and obtain, as output, information indicating whether the second product variety to be inspected is a good product or a defective product from the first trained model.

[0009] An information processing device according to another aspect of the present invention includes a quality determination unit that inputs an inspection image of the second variety of product to be inspected into a first trained model that has been trained using training data that receives as input a training image set including a first defective product image of a first variety of product that is a defective product, a second good product image of a second variety of product that is a good product, and a pseudo-defective product image of the second variety of product, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality that has occurred in the second variety of product has been superimposed on the second good product image, and outputs information indicating whether the second variety of product included in the training image set is a good product or a defective product, and obtains as output from the first trained model information indicating whether the second variety of product to be inspected is a good product or a defective product.

[0010] A trained model according to another aspect of the present invention is trained using training data that receives as input a training image set including a first defective product image of a first variety of product that is a defective product, a second good product image of a second variety of product that is a good product, and a pseudo-defective product image of the second variety of product, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the second variety of product is superimposed on the second good product image, and outputs information indicating whether the second variety of product included in the training image set is a good product or a defective product, and causes a computer to function so that, in response to an input inspection image of the second variety of product that is the subject of inspection, the computer outputs information indicating whether the second variety of product that is the subject of inspection is a good product or a defective product.

[0011] According to the present invention, when pseudo-generated defective product images are used to inspect multiple types of products using a trained model, it is possible to improve the accuracy of judgment in product inspection while reducing the amount of work required to generate the trained model.

[0012] FIG. 1 is a block diagram of a quality assessment device according to the present embodiment. FIG. 2 is a diagram illustrating the concept of processing using a quality assessment model according to the present embodiment. FIG. 3 is a flowchart outlining processing using a quality assessment device according to the present embodiment. FIG. 4 is a flowchart illustrating quality assessment model generation processing according to the present embodiment. FIG. 5 is a flowchart illustrating pseudo-abnormal image generation processing according to the present embodiment. FIG. 6 is a diagram illustrating abnormality information according to the present embodiment. FIG. 7 is a diagram illustrating an example of a pseudo-abnormal image generation method according to the present embodiment. FIG. 8 is a diagram illustrating an example of a pseudo-abnormal image generation method according to the present embodiment. FIG. 9 is a diagram illustrating generation of a pseudo-defective image according to the present embodiment. FIG. 10 is a flowchart illustrating pseudo-defective segment information generation processing according to the present embodiment. FIG. 11 is a flowchart illustrating quality assessment processing according to the present embodiment. FIG. 12 is a diagram illustrating another example of a pseudo-abnormal image generation method according to the present embodiment. FIG. 13 is a diagram illustrating the concept of processing using a quality assessment model as another example according to the present embodiment.

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A preferred embodiment of the present invention will be described with reference to the accompanying drawings. In the drawings, components with the same reference numerals have the same or similar configurations.

[0014] 1 shows a block diagram of a quality assessment device 101 according to this embodiment. The quality assessment device 101 includes a display unit 1011, a storage unit 1012, an image acquisition unit 1013, a pseudo-defective product image generation unit 1014, a quality assessment model generation unit 1015, and a quality assessment unit 1016. Each unit of the quality assessment device 101 can be realized by a processor executing a program stored in a storage device in an information processing device such as a personal computer.

[0015] The quality assessment device 101 is a device used for visual inspection of products. The products targeted by the quality assessment device 101 include subtype products that share the same materials for the components that make up the product but have different component shapes and component arrangement patterns, and these similar products are distinguished as being of different "product types." An example of a product is a printed circuit board, and products with different wiring on the printed circuit board are distinguished as different products. Note that the products and product types targeted by the quality assessment device 101 are not limited to this and can include, for example, mechanical parts such as screws.

[0016] For example, a certain product may include a product of type A (first type) and a product of type B (second type). When performing a visual inspection of a product, it is preferable to determine whether the product is good or defective for each type. Here, a defective product refers to a product that has an abnormality, such as a product with foreign matter attached, a product with a chip, a product with a flaw, or a product with dirt or stains. In a visual inspection by the quality determination device 101, a determination of whether the product is good or defective is made by image processing using a trained model.

[0017] In this case, it is assumed that for products of type A, sufficient images of good products (first good product images) and defective product images (first defective product images) have been collected to enable model training, while for products of type B, the number of defective product images is insufficient for training. This situation occurs, for example, when the number of defective product images is small to begin with, or when collection of defective product images is omitted to reduce the man-hours required for generating a trained model. The quality assessment device 101 supplements the number of defective product images of type B using pseudo-generated defective product images of products of type B (pseudo-defective product images). The quality assessment device 101 is a device that determines whether products of type B are good or defective using a trained model trained using an image set including pseudo-defective product images.

[0018] The display unit 1011 is a display of the quality assessment device 101. The storage unit 1012 stores various types of information used in processing by the quality assessment device 101. The storage unit 1012 stores an image generation model 10121 and a quality assessment model 10122 used in processing by the quality assessment device 101. The storage unit 1012 also stores data generated by the quality assessment device 101.

[0019] Image generation model 10121 (second trained model) is a trained model that receives, as input, segment information in which an image is labeled on a region or pixel basis by a segmentation method such as semantic segmentation or instance segmentation, and is trained to generate an image based on the segment information. For example, an image generation model such as label2image is used as the base model for image generation model 10121. Image generation model 10121 is trained using segment information in which, for example, non-defective product images and defective product images of a product type A are segmented.

[0020] The quality determination model 10122 (first trained model) is a trained model that has been trained to output information indicating whether the product of type B being inspected is a pass or fail product in response to an input of an inspection image of the product of type B being inspected. The quality determination model 10122 will be described later.

[0021] The image acquisition unit 1013 acquires, for example, an external image of a product captured by an imaging device installed on a product production line from an external device. The image acquisition unit 1013 communicates with, for example, an information processing device that captures the image, and acquires the external image of the product. The image acquisition unit 1013 also acquires, from the external information processing device, a defective product image of a type A product and a non-defective product image of a type B product (second non-defective product image), and stores them in the memory unit 1012 of the quality assessment device 101.

[0022] The pseudo-defective product image generation unit 1014 generates a pseudo-defective product image of the product of type B by combining a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality that actually occurred in the product of type B with a second non-defective product image of the product of type B. The generation of the pseudo-defective product image will be described later.

[0023] The quality determination model generation unit 1015 generates a quality determination model 10122 that outputs a determination result of whether a product of type B is good or bad, based on the pseudo-defective product image generated by the pseudo-defective product image generation unit 1014. Specifically, the quality determination model generation unit 1015 generates the quality determination model 10122 based on the defective product image of type A, the good product image of type B, and the pseudo-defective product image.

[0024] The quality judgment unit 1016 inputs an inspection image of the product of type B to be inspected into the quality judgment model 10122, and obtains information from the quality judgment model 10122 indicating whether the product of type B to be inspected is good or defective.

[0025] In the quality assessment device 101, the quality assessment device 101 is described as performing both the generation of the quality assessment model 10122 by the quality assessment model generation unit 1015 and the judgment of good or bad products by the quality assessment unit 1016, but the generation of the quality assessment model 10122 may be performed, for example, by another information processing device, and the generated quality assessment model 10122 may be stored in the memory unit 1012.

[0026] Referring to Figure 2, the quality assessment device 101 will now describe the pass / fail assessment of a product of type B. In step (A) of Figure 2, the quality assessment model generation unit 1015 uses a training image set including a defective product image I11 of a product of type A, a good product image I20 of a product of type B, and a pseudo-defective product image I21 of a product of type B to generate a quality assessment model 10122 that has been trained to output information indicating whether a product of type B included in the training image set is pass or defective. Here, the defective product image I11 of the product of type A is an image that includes an actual foreign object O1. The pseudo-defective product image I21 of the product of type B is an image generated by the pseudo-defective product image generation unit 1014 (described below) to include a pseudo-generated foreign object O2.

[0027] In step (B), the quality judgment unit 1016 inputs the inspection image I22 of the product of type B to the quality judgment model 10122. In step (C), the quality judgment unit 1016 obtains a judgment result indicating whether the product of type B shown in the inspection image is good or defective from the quality judgment model 10122. In the example of Figure 2, the inspection image I22 of the product of type B contains a foreign object O3, and therefore the judgment result is "defective."

[0028] As another example, the quality assessment model 10122 may be trained to output, for each pixel of the inspection image, the type of defect (also referred to as an anomaly) (e.g., foreign matter, chipping, flaw, etc.) to which the defective product belongs. In this case, the assessment result may include information indicating the location of the defect on a pixel-by-pixel basis. The assessment result may also include the pixel-by-pixel defect assessment result and the type of defect. Furthermore, the information indicating whether the inspected product of type B is a pass / fail product may be information indicating the location of an abnormal region in the inspection image. Furthermore, the information indicating whether the inspected product of type B is a pass / fail product may include information indicating the location of a non-abnormal region. Furthermore, the information indicating whether the inspected product of type B is a pass / fail product may include an inspection image binarized into an abnormal region indicating an anomaly and a non-abnormal region that is not an abnormal region. Note that the inspection image binarized into an abnormal region and a non-abnormal region that is not an abnormal region has values ​​that change discontinuously, but the inspection image may have values ​​that change continuously between the abnormal region and the non-abnormal region. In this case, the continuous change in the numerical value will depict the difference between the abnormal area and the non-abnormal area with different shading. Furthermore, the information indicating whether the product of type B being inspected is a pass or a fail product may include a determination result indicating whether the product of type B shown in the inspection image is a pass or a fail product.

[0029] The quality determination model 10122 can also perform other classifications, such as indicating whether the product of type B is a good product or a defective product. For example, the quality determination model 10122 may be trained to output information indicating the type of defect (e.g., foreign matter, chipping, flaws, etc.) to which the product of type B belongs. The quality determination model 10122 may also be trained to output information indicating which pixels of the inspection image are defective. The quality determination model generation unit 1015 can appropriately label the training image set according to the results output by the quality determination model 10122, thereby training the quality determination model 10122.

[0030] 3 shows a flowchart of the processing by the quality determining device 101. In step S301, the quality determining device 101 performs a quality determination model generation process. In step S302, the quality determining unit 1016 performs a quality determination process.

[0031] FIG. 4 shows a flowchart of the quality determination model generation process.

[0032] In step S401, the image acquisition unit 1013 acquires a non-defective product image of type B product.

[0033] In step S402, the pseudo-defective image generating unit 1014 executes pseudo-abnormal image generating processing, which will be described with reference to FIG.

[0034] 5, the pseudo-defective image generation unit 1014 acquires the abnormality type (e.g., flaw, foreign matter, dirt, etc.) indicated by the pseudo-abnormal image to be synthesized. Here, the pseudo-defective image generation unit 1014 may acquire the abnormality type based on an input from a user operating the quality assessment device 101, or may acquire the abnormality type by randomly selecting it by the pseudo-defective image generation unit 1014, for example.

[0035] In step S502, the pseudo-defective image generating unit 1014 refers to the anomaly information and selects a pseudo-abnormal image generating method associated with the anomaly type. The anomaly information is stored, for example, in the storage unit 1012. An example of the anomaly information is shown in FIG. 6.

[0036] The anomaly information has the following items: "anomaly type," "pseudo-abnormal image generation method," "location of anomaly occurrence," and "abnormal conditions." The anomaly information is information that has been generated in advance, for example, based on defective product images of type A products for which a sufficient number of defective product images have been collected. The "anomaly type" item stores information indicating the type of abnormality represented by the pseudo-abnormality. The "pseudo-abnormal image generation method" item stores a method for generating a pseudo-abnormal image. The "location of anomaly occurrence" item stores information indicating the location where an abnormality may occur. The "abnormal conditions" item stores size conditions that are set for the pseudo-abnormal image.

[0037] For example, if the abnormality type is a flaw, the pseudo-abnormal image generation method is either random walk method 1 or use of defective product images of type A, and the method is selected by the pseudo-defective product image generation unit 1014 according to a setting from the user. Furthermore, it is stored that the flaw abnormality occurs in a gold-plated portion of the product. Furthermore, it is stored that the abnormal condition for the flaw is that the width is 3 pixels or less and the length is in the range of 10 to 30 pixels.

[0038] When the abnormality type is a foreign substance, the pseudo-abnormal image generation method is either random walk method 2 or use of defective product images of type A, and the method is selected by the pseudo-defective product image generation unit 1014 according to settings made by the user. It is also stored that a foreign substance abnormality occurs in the solder resist portion of the product. It is also stored that the abnormal condition for a foreign substance is an area of ​​1200 pixels or less.

[0039] When the abnormality type is dirt or stain, the pseudo-abnormal image generation method is to use a noise image or a defective product image of type A, and the method is selected by the pseudo-defective product image generation unit 1014 according to a setting from the user. Also, it is stored that dirt or stain abnormalities occur in the solder resist area of ​​the product. Also, it is stored that the abnormal condition for dirt or stains is an area of ​​3000 pixels or less.

[0040] In step S503, the pseudo-defective image generating unit 1014 generates a pseudo-abnormal image based on the pseudo-abnormal image generating method selected in step S502.

[0041] Returning to FIG. 4, in step S403, the pseudo-defective image generating unit 1014 generates pseudo-abnormal segment information in which regions showing pseudo-abnormalities are segmented based on the pseudo-abnormal image.

[0042] An example of pseudo-abnormal image generation processing and generation of pseudo-abnormal segment information will be described with reference to FIGS.

[0043] 7 shows an example in which the anomaly type is a flaw and the pseudo-abnormal image generation method is random walk method 1. In random walk method 1, a graph G70 is generated by arranging a sequence of random walk points on a two-dimensional coordinate system and continuously connecting the points with a spline curve. The pseudo-defective image generation unit 1014 then generates (7A) an image I70 including a curve D connecting six points including point P1 in graph G70. The pseudo-defective image generation unit 1014 then generates (7B) a pseudo-abnormal segment image S70 in which an area showing a pseudo-abnormal flaw is segmented based on image I70.

[0044] 8 shows an example in which the type of abnormality is a foreign object and the pseudo-abnormal image generation method is random walk method 2. In random walk method 2, a random walk point sequence including point P2, whose angle and distance from center C are randomly determined, is arranged on a two-dimensional coordinate system, and (8A) an image I81 is generated that includes a foreign object O represented by a closed curve (closed line) formed by connecting the points. The pseudo-defective image generation unit 1014 then (8B) generates a pseudo-abnormal segment image S80 in which a region showing a pseudo-abnormality due to a foreign object is segmented based on image I70.

[0045] FIG. 9 shows an example in which the type of anomaly is dirt or stain, and the pseudo-abnormal image generation method uses a noise image. The pseudo-defective image generation unit 1014 generates a noise image I90 using, for example, Perlin noise or fBM (fractional Brownian Motion) noise generation techniques. The pseudo-defective image generation unit 1014 (9A) converts the generated noise image into a multi-valued image I91 having two or more pixel values ​​based on at least one predetermined threshold, and (9B) extracts an image I92 corresponding to a pseudo-abnormality. Here, the extraction of image I92 in (9B) involves extracting an image I92 in which an area A1 formed by consecutive pixels of a first value in the multi-valued image I91 is surrounded by pixels of a second value different from the first value, as shown in area A2. The pseudo-defective image generating unit 1014 generates pseudo-abnormal segment information S90 by segmenting the area showing the pseudo-abnormality based on the (C) image I92.

[0046] 9, the case of binarization was described as an example of multi-value conversion, but the pseudo-defective image generation unit 1014 may convert the noise image to have three or more pixel values. For example, it may be converted into an image having four pixel values, such as a pixel indicating a good product, a pixel indicating a flaw, a pixel indicating a chip, and a pixel indicating a dent. This makes it possible to generate pseudo-abnormal segment information while distinguishing between types of defects.

[0047] 4, in step S404, the pseudo-defective product image generation unit 1014 performs segmentation processing on the non-defective product image of the product type B to generate non-defective product segment information. An example of this processing is shown in step (10A) of FIG. 10. Non-defective product segment information S20 is generated from the non-defective product image I20 of the product type B. The non-defective product segment information S20 includes, depending on the material of the product, for example, a segment S201 corresponding to gold plating and a segment S202 corresponding to solder resist.

[0048] In step S405, the pseudo-defective image generation unit 1014 performs a pseudo-defective segment information generation process. The pseudo-defective segment information generation process will be described with reference to Fig. 11. In step S1101, the pseudo-defective image generation unit 1014 acquires the generated pseudo-abnormal image and the abnormality type indicated by the pseudo-abnormal segment.

[0049] In step S1101, the pseudo-defective image generation unit 1014 converts the pseudo-abnormal segments to satisfy the abnormality conditions associated with the anomaly type. For example, if the anomaly type is a scratch, the pseudo-defective image generation unit 1014 modifies the size of the pseudo-abnormal segments so that the width of the pseudo-abnormal segments is 3 pixels or less and the length is in the range of 10 to 30 pixels. For example, if the anomaly type is a foreign object, the pseudo-defective image generation unit 1014 modifies the size of the pseudo-abnormal segments so that the area of ​​the pseudo-abnormal segments is 1200 pixels or less. Furthermore, if the anomaly type is a stain or blemish, the pseudo-defective image generation unit 1014 modifies the size of the pseudo-abnormal segments so that the area of ​​the pseudo-abnormal segments is 3000 pixels or less.

[0050] In step S1101, the pseudo-defective product image generation unit 1014 acquires the abnormality location associated with the abnormality type. For example, if the abnormality type is a flaw, "gold plating" is acquired as the abnormality location. Also, if the abnormality location is a foreign substance, dirt, or stain, "soldering resist" is acquired as the abnormality location.

[0051] In step S1101, the pseudo-defective image generating unit 1014 generates pseudo-defective segment information by combining a pseudo-abnormal segment with a segment corresponding to the location of the abnormality in the non-defective segment information. For example, if the pseudo-abnormal segment indicates a flaw, the pseudo-defective image generating unit 1014 combines the pseudo-abnormal segment with the segment corresponding to the gold plating, since the gold plating is the location of the abnormality.

[0052] 10, if the pseudo-abnormal segment indicates a foreign object, the pseudo-defective image generating unit 1014 combines the pseudo-abnormal segment with the segment S202 corresponding to the solder resist, since the solder resist is the location where the abnormality occurred. By this process, pseudo-defective segment information S21 is generated in which the pseudo-abnormal segment information S90 is combined with the non-defective segment information S20.

[0053] In step S406, the pseudo-defective image generation unit 1014 inputs the pseudo-defective segment information into the image generation model 10121 to generate a pseudo-defective image of the product type B. An example of this process is shown in step (10C) of FIG. 10 . This process generates a pseudo-defective image I21 that includes a foreign object O2 (pseudo-abnormal image) from the pseudo-defective segment information S21. The pseudo-defective image generation unit 1014 can generate multiple types of pseudo-defective images by repeatedly generating pseudo-defective images. The pseudo-defective image generation unit 1014 can generate pseudo-defective images that include abnormalities in abnormality locations where an abnormality is highly likely to occur, depending on the type of abnormality. This makes it possible to generate pseudo-defective images that are closer to actual defects.

[0054] In step S407, the quality determination model generation unit 1015 trains a quality determination model based on a training image set including defective product images of product type A, non-defective product images of product type B, and pseudo-defective product images of product type B.

[0055] The quality judgment process will be described with reference to Fig. 12. In step S1201, the image acquisition unit 1013 acquires an inspection image of a product of type B. In step S1202, the quality judgment unit 1016 inputs the inspection image of the product of type B to the quality judgment model 10122. In step S1203, the quality judgment unit 1016 acquires a judgment result from the quality judgment model 10122. This makes it possible to perform an appearance inspection of the product using the quality judgment model 10122.

[0056] In determining whether a product is good or bad using conventional machine learning, learning is often performed using a training image set that includes images of good products of type B and images of pseudo-defective products of type B. The pseudo-defective images of type B products are not images of abnormalities that have occurred in actual products, but are pseudo-generated images. Therefore, for example, if a quality determination model is trained using only images of good products and pseudo-defective products of type B products, the model may be trained to determine whether a product is defective based on features generated by the pseudo-defective images. As a result, the model may make false positives or overdetections, resulting in low accuracy.

[0057] On the other hand, the quality judgment model 10122 in the quality judgment device 101 includes in its training image set not only images of good products and pseudo-defective products of product type B, but also images of defective products of product type A. In the quality judgment model 10122, the defective product images of product type A are images that show actual abnormalities that have occurred in the actual product. Therefore, the quality judgment model 10122 can also use feature quantities based on actual abnormalities that have occurred in the actual product, and is trained to judge defective products while reflecting the features of actual defective products. As a result, the quality judgment device 101 can use the quality judgment model 10122 to improve the judgment accuracy in product inspection.

[0058] In the above-described embodiment, examples have been described in which pseudo-defective product images are generated based on a random walk method or noise images. However, the method of generating pseudo-defective product images is not limited to this. For example, a pseudo-defective product image may be generated by extracting a pseudo-abnormal image, which is a region corresponding to an abnormality, from a defective product image of a product type A and combining it with a non-defective product image of a product type B. For example, as shown in FIG. 13 , (13A) a pseudo-abnormal image, which is a region corresponding to a foreign object O1, may be extracted from a defective product image of a product type A, and (13B) an image I23 may be generated in which a foreign object O4 based on the foreign object O1 is placed in segment S202 where the foreign object should be placed in the non-defective product segment information of product type B. Image processing techniques such as Poisson blending may be used when combining images. Furthermore, to ensure that the extracted image blends well with the non-defective product image, the combination may be performed by taking a weighted average based on the distance from the center of the defective area or the color tone. Alternatively, a defective product image of a product type A may be segmented to extract a region corresponding to a defect. A pseudo-defective product image may be generated by combining a non-defective product image of product type B with segment information that has been subjected to segmentation processing. In this case, the pseudo-defective product image is generated by the image generation model 10121 based on the combined segment information.

[0059] Another example of the quality assessment device 101 according to this embodiment will be described. A process for determining whether a product of type B is good or bad by the quality assessment device 101 according to this embodiment will be described with reference to FIG. 14 . This example differs from the example shown in FIG. 2 in that a good product image I10 (first good product image) of a product of type A is used to train the quality assessment model 10122A. In this example, the image acquisition unit 1013 acquires an inspection image, a defective product image of a product of type A, and a good product image of a product of type B from an external information processing device, and stores the good product image of the product of type A in the storage unit 1012 of the quality assessment device 101.

[0060] In step (A) of FIG. 14 , the quality judgment model generation unit 1015 uses a training image set including a non-defective product image I10 of a product of type A, a defective product image I11 of a product of type A, a non-defective product image I20 of a product of type B, and a pseudo-defective product image I21 of a product of type B to generate a quality judgment model 10122 trained to output information indicating whether a product of type B included in the training image set is non-defective or defective. The processing in step (C) of FIG. 14 is similar to that in step (C) of FIG. 2 , but the processing in step (B) of FIG. 14 differs from that in step (B) of FIG. 2 . Specifically, the quality judgment unit 1016 inputs the non-defective product image I20 and the inspection image I22 of a product of type B to the quality judgment model 10122. As a result, even when images of a product of type A are used in training the quality judgment model 10122 that performs quality judgment on products of type B, the influence of differences in product structure, i.e., differences in product types, is reduced, improving the judgment accuracy in product inspection. Furthermore, by performing a comparison process, such as aligning the image coordinates of the non-defective product image and the inspection image so that the same portions of the product in the non-defective product image and the inspection image have the same coordinates and calculating the difference between the non-defective product image and the inspection image, it becomes easier to identify defective portions present in the inspection image. Note that in step (B) of Fig. 14, unlike the process in step (B) of Fig. 2, in the process in step (C) of Fig. 14, the quality judgment unit 1016 inputs the non-defective product image I20 of the product type B in addition to the inspection image I22 to the quality judgment model 10122. This is because it is expected that the image format when the quality judgment unit 1016 inputs the image to the quality judgment model 10122 and the image format when the quality judgment model generation unit 1015 generates the quality judgment model 10122 will be the same image format.

[0061] The quality determination device 101 inputs an inspection image I22 of a product of type B to the quality determination model 10122, which is trained using training data that receives as input a training image set including a defective product image I11 of a defective product of type A, a good product image I20 of a good product of type B, and a pseudo-defective product image I21 of a product of type B, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the product of type B is superimposed on the good product image I20, and outputs information indicating whether the product of type B included in the training image set is a good product or a defective product. The quality determination device 101 obtains, as an output, information indicating whether the product of type B being inspected is a good product or a defective product from the quality determination model 10122.

[0062] As a result, the quality assessment device 101 can assess inspection images using the quality assessment model 10122 that has been trained to include defective product images of product type A, which are images showing actual abnormalities that have occurred in actual products. The quality assessment device 101 can also use feature amounts based on actual abnormalities that occur in actual products, and is trained to assess defective products while reflecting the features of actual defective products. This improves the assessment accuracy in product inspection compared to when assessing defective products based on feature amounts generated by pseudo-defective product images. Furthermore, the use of pseudo-defective product images reduces the amount of work required to collect defective product images, etc.

[0063] In the above aspect, the pseudo-defective product image generation unit 1014 may generate a pseudo-defective product image by synthesizing a pseudo-abnormal image at a position in the second good product image that corresponds to the location of the abnormality, based on the type of abnormality in the pseudo-abnormal image and the location of the abnormality associated with the type of abnormality. This allows the pseudo-defective product image generation unit 1014 to generate a pseudo-defective product image that includes an abnormality at a location where the abnormality is likely to occur, depending on the type of abnormality, and thereby enables the generation of a pseudo-defective product image that is closer to an actual defect.

[0064] In the above aspect, the pseudo-defective image generation unit 1014 may generate pseudo-abnormal segment information by segmenting the pseudo-abnormal image, generate good segment information S20 by segmenting the good image I20, and generate pseudo-defective segment information S21 based on the good segment information S20 and the pseudo-abnormal segment information S50. The pseudo-defective image generation unit 1014 may also generate the pseudo-defective image I21 based on the pseudo-defective segment information S21.

[0065] By generating pseudo-defective product images via segment information, noise can be generated and a large number of abnormalities can be reproduced, so that pseudo-defective product images can be generated more efficiently.

[0066] In the above aspect, the pseudo-defective product image generation unit 1014 inputs defective product segment information obtained by segmenting a defective product image, inputs pseudo-defective product segment information S21 into an image generation model 10121 that has been trained to output a defective product image, and generates a pseudo-defective product image I21 as an output.

[0067] This allows pseudo-defective product images to be generated more efficiently.

[0068] In the above aspect, the pseudo-defective image generation unit 1014 acquires a noise image I90, converts the noise image I90 into a multi-value image having two or more pixel values ​​based on at least one predetermined threshold, and extracts an image I92 from the converted noise image I91, in which an area A1 formed by successive pixels of a first value in the converted noise image I91 is surrounded by pixels of a second value different from the first value. The pseudo-defective image generation unit 1014 may generate pseudo-abnormal segment information S90 in which the area A1 is segmented as an area indicating a pseudo-abnormality.

[0069] This makes it possible to extract a large number of noise images when one noise image contains a plurality of closed regions, thereby enabling more efficient generation of pseudo-defective product images.

[0070] In the above aspect, the image acquisition unit 1013 may acquire a non-defective product image I10 of a product of type A that is a non-defective product, and the learning image set may further include a non-defective product image of the first type of product.

[0071] This allows for more accurate discrimination of non-defective products of type B, improving the accuracy of judgment in product inspection.

[0072] In the above aspect, the quality determination unit 1016 may input the non-defective product image I10 of the product of type A, which is a non-defective product, and the inspection image to the quality determination model 10122.

[0073] The above-described embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The elements of the embodiments, as well as their arrangement, conditions, shape, size, etc., are not limited to those illustrated and can be modified as appropriate. Furthermore, configurations shown in different embodiments can be partially substituted or combined with each other.

[0074] 101...quality determination device, 1011...display unit, 1012...storage unit, 1013...image acquisition unit, 1014...pseudo-defective product image generation unit, 1015...quality determination model generation unit, 1016...quality determination unit, 10121...image generation model, 10122, 10122A...quality determination model

Claims

1. A method comprising: a computer receives as input a training image set of a trained model including a first defective image of a first variety of product that is defective, a second good image of the product of a second variety that is good, and a pseudo-defective image of the second variety of product, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the second variety of product is synthesized onto the second good image; inputting an inspection image of the second variety of product that is to be inspected into a first trained model trained using teacher data that outputs information indicating whether the second variety of product included in the training image set is a good product or a defective product; and obtaining as output from the first trained model information indicating whether the second variety of product that is to be inspected is a good product or a defective product.

2. A method according to claim 1, further comprising the computer synthesizing the pseudo-abnormal image at a position in the second good product image that corresponds to the location of the abnormality based on the type of abnormality in the pseudo-abnormal image and the location of the abnormality associated with the type of abnormality, thereby generating the pseudo-defective product image.

3. A method according to claim 1 or 2, further comprising the computer generating good segment information by segmenting the second good image, generating pseudo-abnormal segment information by segmenting the pseudo-abnormal image, and generating pseudo-defective segment information based on the good segment information and the pseudo-abnormal segment information, wherein generating the pseudo-defective image includes generating the pseudo-defective image based on the pseudo-defective segment information.

4. A method according to claim 3, wherein generating the pseudo-defective product image comprises inputting defective product segment information obtained by segmenting the first defective product image, inputting the pseudo-defective product segment information into a second trained model trained to output the first defective product image, and generating the pseudo-defective product image as an output.

5. A method according to claim 3, further comprising the computer acquiring a noise image, and generating the pseudo-abnormal segment information comprising: converting the noise image into a multi-value image having two or more pixel values ​​based on at least one predetermined threshold; extracting from the converted noise image an image in which a first region formed by successive pixels of a first value in the converted noise image is surrounded by pixels of a second value different from the first value; and generating the pseudo-abnormal segment information in which the first region is segmented as a region indicating the pseudo-abnormality.

6. The method of claim 1, further comprising: the computer acquiring a first quality image of a first variety of product that is a quality product; and the learning image set further includes the first quality image of the first variety of product.

7. The method according to claim 6, wherein inputting the inspection image includes inputting the first good product image and the inspection image to the trained model.

8. A program that causes a computer to execute the following steps: input a training image set including a trained model including a first defective image of a first variety of product that is defective, a second good image of the product of a second variety that is good, and a pseudo-defective image of the second variety of product, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the second variety of product is synthesized onto the second good image; input an inspection image of the second variety of product that is to be inspected to a first trained model trained using teacher data that outputs information indicating whether the second variety of product included in the training image set is a good product or a defective product; and obtain, as an output, information indicating whether the second variety of product that is to be inspected is a good product or a defective product from the first trained model.

9. An information processing device comprising: a trained model, the information processing device comprising: a training image set including: a first defective image of a first variety of product that is defective; a second good image of the second variety of product that is good; and a pseudo-defective image of the second variety of product, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the second variety of product is synthesized onto the second good image; and a quality judgment unit that inputs an inspection image of the second variety of product that is to be inspected into a first trained model trained using teacher data that outputs information indicating whether the second variety of product included in the training image set is a good product or a defective product, and obtains information indicating whether the second variety of product that is to be inspected is a good product or a defective product as an output from the first trained model.

10. A trained model that receives as input a training image set including a first defective image of a first variety of product that is defective, a second good image of the product of a second variety that is a good product, and a pseudo-defective image of the second variety of product, in which a pseudo-abnormal image showing a pseudo-abnormality different from an abnormality occurring in the second variety of product is synthesized onto the second good image, and is trained using training data that outputs information indicating whether the second variety of product included in the training image set is a good product or a defective product, and causes a computer to function in such a way that, in response to an input of an inspection image of the second variety of product that is the subject of inspection, the computer outputs information indicating whether the second variety of product that is the subject of inspection is a good product or a defective product.

Images