Model learning device, trained model, trained model generation device, and determination device

Abstract

Improves the performance of trained models compared to conventional technologies. [Solution] The model learning device includes a learning unit that learns a model using training data comprising information on specific partial images that affect the quality determination of an image of a surface to be inspected, and information indicating the quality of the part of the surface to be inspected corresponding to the partial image. Since the training data uses information on specific partial images that affect the quality determination of an image of a surface to be inspected, this information can contain less information unnecessary for quality determination than in conventional technology, and the performance of the trained model can be improved compared to conventional technology.

Description

Technical Field

[0001] The technology of the present disclosure relates to a model learning device, a learned model, a learned model generation device, and a determination device.

Background Art

[0002] Patent Document 1 discloses a technique for determining a deterioration factor of a pier from an image of an area of a pier in an image of a structure, such as a bridge. In this conventional technique, a plurality of small piece images of a predetermined size are extracted from an image of the pier area, each of the extracted small piece images is input into a learned model, an identification result of the deterioration factor of the location corresponding to each small piece image is output, and based on the identification results of each of the output small piece images, the deterioration factor within the above area is determined.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The learned model of the above conventional technology is a model learned to output an identification result that identifies the deterioration factor of the location corresponding to the small piece image when the small piece image is input. Also, in the above conventional technology, a plurality of small piece images are extracted while sliding the small piece images so that they partially overlap within the image of the pier area. Therefore, the small piece images extracted for learning the model are likely to contain more unimportant information (image information that is not the deterioration part around the deterioration part), and the performance of the learned model may deteriorate.

[0005] The present invention aims to provide a model learning device, a trained model, and a trained model generation device that can improve the performance of trained models compared to conventional technologies, as well as a judgment device that can improve judgment performance compared to conventional technologies. [Means for solving the problem]

[0006] To achieve the above objective, a model learning device according to a first aspect of the technology of this disclosure includes a learning unit that learns a model using training data comprising information on specific partial images that affect the quality determination of an image of a surface to be inspected, and information indicating the quality of the part of the surface to be inspected corresponding to the partial image.

[0007] Here, the training data includes at least one of the following: first training data comprising information on a partial image of the same size as the defective area in an image of the surface of a good product to be inspected (i.e., a specific partial image that affects the pass / fail determination) and information indicating a good product; and second training data comprising information on a partial image of a defective area in an image of the surface of a defective product to be inspected (i.e., a specific partial image that affects the pass / fail determination) and information indicating a defective product.

[0008] In the second embodiment of the model learning apparatus, the learning unit determines information about the partial image from the partial image and predetermined rules, as in the first embodiment.

[0009] The third embodiment of the trained model is a model trained with training data comprising information on specific partial images within an image of the surface to be inspected that have an effect on determining whether it is good or bad, and information indicating whether the part of the surface to be inspected corresponding to the partial image is good or bad.

[0010] The trained model generation apparatus of the fourth embodiment includes a generation unit that generates a trained model by training a model using training data comprising information of a specific partial image in an image of a surface to be inspected that affects the quality determination, and information indicating the quality of the part of the surface to be inspected corresponding to the partial image.

[0011] The fifth embodiment of the determination device comprises an imaging unit that photographs the surface to be inspected, a trained model learned by the model learning device of the first embodiment, and a determination unit that determines whether the part of the surface to be inspected corresponding to the partial image is good or bad based on the information of the partial image in the image of the surface to be inspected obtained by the imaging.

[0012] In the sixth embodiment, the determination device, in the fifth embodiment, uses a rule-based initial filter to determine whether the surface to be inspected is good or bad, and determines whether the surface is good or bad based on the information of a partial image of the surface to be inspected that is determined to be defective and a trained model learned by the model learning device of the first embodiment or the second embodiment.

[0013] In the seventh embodiment, the determination device, in the fifth embodiment, includes a determination unit that performs a first quality determination to determine whether the surface to be inspected is good or bad using a rule base, and a second determination to determine whether the part of the surface to be inspected corresponding to the partial image is good or bad based on information of a partial image of the surface to be inspected and a trained model learned by the model learning device of the first embodiment or the second embodiment, and determines whether the surface to be inspected is good or bad based on the results of each determination. [Effects of the Invention]

[0014] The first to fourth aspects of the technology of this disclosure train a model using training data comprising information on specific partial images within an image of a surface to be inspected that have an effect on determining whether it is good or bad, and information indicating whether the part of the surface to be inspected corresponding to the partial image is good or bad. Since the training data uses information on specific partial images within an image of a surface to be inspected that have an effect on determining whether it is good or bad, this information can contain less information unnecessary for determining whether it is good or bad than in the conventional technology, and the performance of the trained model can be improved compared to the conventional technology.

[0015] The fifth to seventh embodiments determine whether the object to be inspected is good or bad based on the trained model of the first embodiment and information from a partial image of the surface of the object to be inspected obtained by the aforementioned photography. Therefore, the performance of determining whether the object to be inspected is good or bad can be improved compared to the conventional technology. [Brief explanation of the drawing]

[0016] [Figure 1] Figure 1 shows an example of the configuration of the information processing device 10 according to the first embodiment. [Figure 2] Figure 2 is a flowchart showing an example of a model learning program 36P1 executed by the CPU 22 in the first embodiment. [Figure 3] Figure 3 shows an example of how the image d to be examined is divided into a predetermined number of segmented images, each of the same size. [Figure 4] Figure 4 shows an example of how an operator determines that an item is defective because it contains a void, and then crops an image of a predetermined size centered on the designated defective area p1. [Figure 5] Figure 5 shows an example of how an operator determines that an item is defective because it has a scratch, and then crops an image of a predetermined size centered on the designated defective area p2. [Figure 6] Figure 6 shows an example of how an operator determines that an item is defective because it has a stain, and then crops an image of a predetermined size centered on the designated defective area p3. [Figure 7] Figure 7 is a flowchart showing an example of the pass / fail judgment program 36P2 of the first embodiment. [Figure 8] Figure 8 shows an example of how cropped images dim1, dim2, ... are acquired while moving over the captured image PM of a modified example of the first embodiment so that the area of ​​the suspected defective part partially overlaps. [Figure 9] Figure 9 is a flowchart showing an example of a model learning program 36P1 executed by the CPU 22 in the second embodiment. [Figure 10] Figure 10 is a flowchart showing an example of a pass / fail judgment program 36P2 in the third embodiment. [Figure 11]FIG. 11 is a flowchart showing an example of a pass / fail determination program 36P2 of a modification of the third embodiment. [Figure 12] FIG. 12 is a diagram showing a display screen 35D of a display device 35 that displays a comparison between a cut-out image (i.e., a partial image) 17 and a photographed image 15 together with the pass / fail determination result.

Embodiments for Carrying Out the Invention

[0017] [Embodiment] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings.

[0018] [First Embodiment] (Configuration) The configuration of the information processing apparatus 10 of the present embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the information processing apparatus 10. As shown in FIG. 1, the information processing apparatus 10 includes a computer 20, an input device 32, a photographing device 34, a display device 35, and a storage device 36. The computer 20 includes a CPU 22, a ROM 24, a RAM 26, and an input / output (I / O) port 28. The CPU 22, the ROM 24, the RAM 26, and the input / output (I / O) port 28 are interconnected by a bus 30. The input / output (I / O) port 28 is connected to the input device 32, the photographing device 34, the display device 35, and the storage device 36.

[0019] The storage device 36 includes a model learning program 36P1 and a pass / fail determination program 36P2.

[0020] The storage device 36 includes an image storage area 36G that stores images IM1, IM2,... obtained by the photographing device 34 photographing a plurality of inspection targets. Each of the plurality of images IM1, IM2,... is an image obtained by photographing the entire surface of each of the plurality of inspection targets.

[0021] The storage device 36 includes a training data storage area 36T for storing training data. The training data is data that combines partial images (also called cropped images) pim1, pim2, pim3, pim4, ... of defective parts in an image of the surface of the object to be inspected, and information indicating whether the part of the object to be inspected corresponding to the partial images pim1, pim2, pim3, pim4, ... is good or bad. The information indicating whether the object is good or bad is information G indicating a good product in the case of a good product, and information indicating the type of defect in the case of a defective product. Information indicating the type of unwanted material includes, for example, information indicating a single type such as V indicating a void (i.e., a cavity), K indicating a scratch, and S indicating a stain, as well as information indicating a combination of multiple types. For example, if there are voids and scratches, there is information indicating the combination of voids and scratches (V+K); if there are scratches and stains, there is information indicating the combination of scratches and stains (K+S); and if there are voids, scratches, and stains, there is information indicating a combination of these (V+K+S).

[0022] The memory device 36 stores the trained model 36M.

[0023] The information processing device 10 is an example of the "model learning device," "trained model generation device," and "decision device" of the technology disclosed herein. CPU22 is an example of the "learning unit" and the "generation unit" of the technology disclosed herein.

[0024] (action) Next, the operation of the first embodiment will be described.

[0025] <Model training process, model training method, and generation of trained models> Figure 2 is a flowchart showing an example of a model learning program 36P1 executed by the CPU 22 of the first embodiment. Figure 3 shows an example of dividing the image d of the object to be inspected into a predetermined number of segmented images pim0, each of the same size. Figure 4 shows an example of when the operator determines that the object to be inspected is defective because a void has occurred in it, and cuts out an image pim1 of a predetermined size centered on the specified defective area p1. Figure 5 shows an example of when the operator determines that the object to be inspected is defective because a scratch has occurred in it, and cuts out an image pim2 of a predetermined size centered on the specified defective area p2. Figure 6 shows an example of when the operator determines that the object to be inspected is defective because a stain has occurred in it, and cuts out a partial image pim3 of a predetermined size centered on the specified defective area p3.

[0026] When the CPU 22 executes the model learning program 36P1, the model learning process and model learning method, and the model generation process and model generation method are executed. The model learning program 36P1 is started by operating the model learning process start button (not shown) on the input device 32.

[0027] In step 52, the CPU 22 initializes a variable d that identifies multiple images IM1, IM2, ... stored in the image memory area 36G to 0 (d←0), and in step 54, the CPU 22 increments the variable d by 1 (d←d+1).

[0028] In step 56, the CPU 22 reads image d, identified by variable d, from the image storage area 36G. For example, if image IM1 is identified by variable d=1, image IM1 is read as image d identified by variable d=1.

[0029] In step 58, the CPU 22 displays image d on the display device 35. That is, the display device 35 has a display screen, and the CPU 22 controls the display device 35 so that image d is displayed on the display screen.

[0030] In step 60, the CPU 22 determines whether information indicating a good product has been input. If it is determined that information indicating a good product has been input, the model learning process proceeds to step 62. If it is determined that information indicating a good product has not been input, the model learning process proceeds to step 68. In step 68, the CPU 22 determines whether a defective part has been specified. If it is determined that a defective part has been specified, the model learning process proceeds to step 70. If it is determined that a defective part has not been specified, the model learning process returns to step 60.

[0031] As described above, when image d is displayed on the display device 35 in step 58, the operator checks the displayed image d and, if the inspected item is determined to be a good product, inputs information indicating a good product via the input device 32. If the item is determined to be defective, the operator specifies the defective area via the input device 32. Specifically, the operator specifies the center of the defective area as the defective area.

[0032] If the operator determines from the displayed image d that the item to be inspected is a good product and inputs information indicating a good product via the input device 32, step 60 becomes a positive determination, and the model learning process proceeds to step 62.

[0033] In step 62, the CPU 22 divides image d (for example, image IM4) into a predetermined number of divided images pim0, each of the same size, as shown in Figure 3. The size of the divided images corresponds to the size of the area expected to be defective. In the example shown in Figure 3, the predetermined number is 9.

[0034] In step 66, the CPU 22 creates training data by combining each image with a good product. Specifically, the CPU 22 creates training data by combining information from a partial image of the same size as the defective area in the surface image of the product to be inspected, with information indicating a good product. As described above, for example as shown in Figure 3, the CPU 22 creates nine sets of training data by combining information from each of the nine divided images pim0 obtained by dividing image d (for example, image IM4) into nine parts, with information indicating a good product (G).

[0035] The training data created in step 66 is an example of the “first training data” of the technology of this disclosure.

[0036] On the other hand, if the operator determines from the displayed image d that the item to be inspected is defective and specifies the defective part via the input device 32, then step 60 becomes a negative determination and step 68 becomes a positive determination, and the model learning process proceeds to step 70.

[0037] In step 70, the CPU 22 extracts a partial image of a predetermined size from image d, centered on the specified location. The size of the partial image is the size of the defective area on the surface of the object being inspected, which is determined by numerous inspections.

[0038] In step 72, the CPU 22 receives input for defective items from the operator via the input device 32. The types of defective items include information indicating a single type and information indicating a combination of multiple types, as described above. Information indicating a single type includes, for example, information V indicating a void (i.e., a cavity), information K indicating a scratch, and information S indicating a stain. Other examples include plating defects, plating spreading, black spots, white spots, broken wires, etc. Information indicating a combination of multiple types includes, for example, if there are voids and scratches, information on the combination of voids and scratches (V+K), if there are scratches and stains, information on the combination of scratches and stains (K+S), and if there are voids, scratches, and stains, information on the combination of these (V+K+S), etc.

[0039] In step 74, the CPU 22 creates training data by combining information from the extracted partial images with information about the defective items. Specifically, the CPU 22 creates training data by combining information from partial images of the defective areas in the surface image of the object to be inspected for defects with information indicating the defective product.

[0040] For example, as shown in Figure 4, if the operator determines that the product is defective because a void has occurred in the product being inspected, they specify the defective area p1 via the input device 32. As a result, step 68 becomes a positive determination, a partial image pim1 is extracted in step 70, and information V indicating the void is input in step 72. Training data is then created by combining the information from the partial image pim1 and the information V indicating the void.

[0041] Furthermore, as shown in Figure 5, for example, if the operator determines that the item to be inspected is defective because it has a scratch, they specify the defective area p2 via the input device 32. As a result, step 68 becomes a positive determination, a partial image pim2 is extracted in step 70, and information K indicating the scratch is input in step 72. Training data is then created by combining the information from the partial image pim2 and the information K indicating the scratch.

[0042] Furthermore, for example, as shown in Figure 6, if the operator determines that the item to be inspected is defective because it has a stain, they specify the defective area p3 via the input device 32. As a result, step 68 becomes a positive determination, a partial image pim3 is extracted in step 70, and information S indicating the stain is input in step 72. Training data is then created by combining the information of the partial image pim3 and the information S indicating the stain.

[0043] The training data created in step 74 is an example of “second training data” for the technology of this disclosure. If an item is being inspected for good quality, any part of the image's surface will affect the determination of whether it is good or not. However, if an item is being inspected for defective products, parts other than the defective area will not affect the determination of whether it is defective.

[0044] Once the training data is created in step 64 or step 74, the model training process proceeds to step 76.

[0045] In step 76, the CPU 22 trains a model using the training data created as described above. The training data, as described above, is data that combines information about specific partial images within an image of the surface to be inspected that have an effect on determining whether it is good or bad, with information indicating whether the part of the surface to be inspected corresponding to that partial image is good or bad. In step 76, the CPU 22 uses this training data to train a model so that when a partial image of the surface to be inspected is input, it outputs information indicating whether the part of the surface to be inspected corresponding to that partial image is good or bad.

[0046] The information in the above-mentioned partial image of the training data is the image data of that partial image itself.

[0047] The algorithm used for model learning is not particularly limited. For example, deep learning or other machine learning algorithms can be applied. An example of deep learning is a convolutional neural network (CNN). Other machine learning algorithms include Support Vector Machines (SVMs), decision trees, and random forests.

[0048] In step 78, the CPU 22 determines whether the variable d is equal to the total number D of multiple images IM1, IM2, ... stored in the image memory area 36G (d=D). If it is determined that the variable d is not equal to the total number D, there are still images available for training the model, so the model training process returns to step 54 and executes the above process (steps 54-78). If it is determined that the variable d is not equal to the total number D, there are no more images available for training the model, so the model training process terminates.

[0049] The above model learning process is performed on multiple images IM1, IM2, ... stored in the image storage area 36G, but the technology of this disclosure is not limited thereto. For example, the model learning program 36P1 may start when at least one captured image for model learning is input and the model learning start button is operated. In this case, the model learning process is performed with the captured image as image d.

[0050] <Perfect / failure determination process and method for determining whether the product is good or bad> Figure 7 is a flowchart showing an example of the pass / fail judgment program 36P2. The CPU 22 executes the pass / fail judgment program 36P2, thereby executing the pass / fail judgment process and pass / fail judgment method. The pass / fail judgment program 36P2 is started by operating the pass / fail judgment process start button (not shown) on the input device 32.

[0051] In step 102, the CPU 22 determines whether the object to be inspected has been photographed by the imaging device 34 to determine its quality, and whether the resulting image has been input. If it is determined that the image has not been input, the quality determination process returns to step 102 and repeats the determination until it is determined that the image has been input. If it is determined that the image has been input, the quality determination process proceeds to step 104.

[0052] In step 104, the CPU 22 divides the captured image into a predetermined number of divided images, each of the same size. The size of the divided images is the size of the area that is expected to be defective, as described above.

[0053] In step 106, the CPU 22 initializes the variable r, which identifies the segmented image, to 0 (r←0), and in step 108, the CPU 22 increments the variable r by 1 (r←r+1).

[0054] In step 110, the CPU 22 inputs the segmented image r into the trained model 36M. This provides information indicating the quality of the part of the inspected object corresponding to the segmented image. This information includes information indicating good products and information indicating defective items.

[0055] In step 112, the CPU 22 displays good or defective items on the display device 35.

[0056] In step 114, the CPU 22 determines whether the variable r is equal to the total number of segmented images R of the captured image (r=R).

[0057] If the variable r is not determined to be equal to the total number R of segmented images of the captured image, then there are segmented images whose quality has not been determined, and the quality determination process returns to step 108 and executes the above process (steps 108 to 114).

[0058] If the variable r is determined to be equal to the total number of segmented images R of the captured image, the pass / fail judgment process is terminated because the pass / fail judgment has been determined based on all parts (segmented images) of the captured image.

[0059] (effect) As explained above, in this embodiment, the model is trained using training data consisting of information on a specific partial image within the surface image of the object to be inspected that affects the quality determination, and information indicating the quality of the corresponding part of the object to be inspected. For a good product to be inspected, any part of the surface in the image affects the quality determination, but for a defective product to be inspected, parts other than the defective area do not affect the defect determination. In this way, since information on a specific partial image within the surface image of the object to be inspected that affects the quality determination is used as training data, the amount of information unnecessary for quality determination can be reduced compared to conventional technology, and the performance of the trained model 36M can be improved compared to conventional technology.

[0060] Furthermore, since the quality of the object being inspected is determined based on the trained model 36M, which was trained as described above, and all parts (segmented images) of the image being inspected, the quality of the object can be determined with greater accuracy than conventional techniques.

[0061] (First to third modified examples of the first embodiment) Next, a first and a second modified example of the above embodiment will be described. The configuration of each modified example is the same as that of the above embodiment, so its description will be omitted. In addition, since the operation of each modified example has some parts that are the same as those of the above embodiment, the same reference numerals will be used for the same parts and their descriptions will be omitted, and the parts that are mainly different will be described.

[0062] (First modification of the first embodiment) <Perfect / failure determination process and method for determining whether the product is good or bad> In the above embodiment, the captured image is divided into a predetermined number of segmented images, each of the same size, and the quality of the object to be inspected is determined based on each segmented image and the trained model 36M. The predetermined number of segmented images do not overlap with each other. The technology of this disclosure is not limited thereto.

[0063] Figure 8 shows an example of how to obtain cropped images dim1, dim2, ... by moving the captured image PM over an area that is expected to be the size of the defective part.

[0064] In the first modification, as shown in Figure 8, cropped images dim1, dim2, ... are acquired from top to bottom while moving the area described above over the captured image PM from left to right at predetermined intervals, partially overlapping. Each of these acquired cropped images dim1, dim2, ... is input to the trained model 36M, and information indicating the quality of the part of the inspection target corresponding to each cropped image dim1, dim2, ... is obtained. The information indicating quality includes information indicating good products and information indicating defective items, as described above.

[0065] In the first modification, the area described above is moved from the left edge to the right edge on the captured image PM, and while moving at predetermined intervals so as to partially overlap, cropped images dim1, dim2, ... are acquired. Therefore, the number of times the image of the defective part is divided and acquired as a divided image can be reduced compared to the above embodiment, and the quality of the inspected object can be determined with greater accuracy than in the above embodiment.

[0066] (Second modification of the first embodiment) In the first embodiment, the information of the portion image in the training data is the image data of the portion image, but the technology of this disclosure is not limited thereto. The information of the portion image in the training data (in particular, the second training data) may be, for example, the type of shape of the defective area, its area, its perimeter, or the ratio of the defective area to the whole image. Accordingly, in step 110, the information input to the trained model is the type of shape of the identified defective area, its area, its perimeter, or the ratio of the defective area to the whole image, obtained by identifying the defective area from the segmented image through image matching or the like.

[0067] (Third modified example of the first embodiment) In the first embodiment, the model is trained using the training data created in step 66 (an example of the first training data) and the training data created in step 74 (an example of the second training data), but the technology of this disclosure is not limited thereto. For example, the model may be trained using the training data created in step 66 or the training data created in step 74.

[0068] [Second Embodiment] Next, a second embodiment will be described.

[0069] (composition) The configuration of the second embodiment is the same as that of the first embodiment, so its description will be omitted. In the second embodiment, the storage device 36 stores images of multiple defective items (defective item images (templates)).

[0070] (action) Since the operation of the second embodiment is similar in some respects to that of the first embodiment, the same reference numerals are used to denote these similar parts, and their descriptions are omitted. The explanation will focus primarily on the differences.

[0071] <Model training process, model training method, and generation of trained models> In the first embodiment, the operator checks the displayed image d and, if it determines that the item to be inspected is a good product, inputs information indicating a good product via the input device 32. If the operator determines that the item is defective, the operator specifies the defective area via the input device 32, and the CPU 22 cuts out a portion image of a predetermined size centered on the specified area, and the operator inputs the defective item. The technology of this disclosure is not limited thereto, and the CPU 22 may perform all of these operations as described below.

[0072] Figure 9 is a flowchart showing an example of a model learning program 36P1 executed by the CPU 22 in the second embodiment.

[0073] In the model learning program 36P1 of the second embodiment, the following processes are performed instead of steps 56-60 and 68-72 in Figure 2.

[0074] After steps 52 and 54, in step 120, the CPU 22 reads image d and each defective item image from the storage device 36. In step 122, the CPU 22 performs template matching between image d and each defective item image and calculates the similarity between the two images.

[0075] In the first method of template matching between image d and defective item images, the CPU 22 arranges each of the same multiple defective item images in a grid pattern on image d so as to cover the entire area of ​​image d. For each arranged defective item image, a match is performed between that defective item image and the corresponding partial image of image d.

[0076] In the second method of template matching between image d and the defective item image, the CPU 22 moves the defective item image within the region of image d and performs matching with the overlapping portion of the image (partial image).

[0077] In step 124, the CPU 22 determines whether the object to be inspected in image d is a good product or not. Specifically, the CPU 22 determines that the object is a good product if the similarity to all defective item images obtained in step 122 is less than a predetermined value, and determines that the object is defective due to a defective item if there is a defective item image with a similarity of more than the predetermined value.

[0078] If the object being inspected in image d is determined to be a good product, the model training process proceeds to step 62. If the object being inspected in image d is not determined to be a good product, the model training process proceeds to step 126.

[0079] In step 126, CPU 22 extracts a partial image of the matched portion. In step 128, CPU 22 retrieves the defective items from the matched template (defective item image).

[0080] After the processing in step 126, the model training process proceeds to step 74.

[0081] <Perfect / failure determination process and method for determining whether the product is good or bad> The pass / fail judgment process and pass / fail judgment method are the same as in the first embodiment, so their explanation will be omitted.

[0082] (effect) In the second embodiment, the CPU 22 performs the model learning process, the model learning method, and the generation of the trained model without operator intervention. Therefore, the operator's workload can be reduced.

[0083] [Third Embodiment] Next, a third embodiment will be described.

[0084] (composition) The configuration of the third embodiment is the same as that of the first embodiment, so its description will be omitted.

[0085] (action) Since the operation of the third embodiment is similar in some respects to that of the first embodiment, the same reference numerals are used to denote these similar parts, and their descriptions are omitted. The explanation will focus primarily on the differences.

[0086] <Model training process, model training method, and generation of trained models> The model training process, model training method, and generation of the trained model are the same as those in the first embodiment, the various modifications of the first embodiment, or the second embodiment, so their description will be omitted.

[0087] <Perfect / failure determination process and method for determining whether the product is good or bad> In the first embodiment, when a captured image is input, the captured image is divided and each divided image is input to the trained model 36M, but the technology of this disclosure is not limited thereto.

[0088] Figure 10 is a flowchart showing an example of the pass / fail judgment program 36P2 of the third embodiment. The pass / fail judgment program 36P2 of the third embodiment uses a rule-based initial filtering. This will be explained in detail below.

[0089] In the pass / fail judgment program 36P2 of the third embodiment, if it is determined that a captured image has been input in step 102 of Figure 7 of the first embodiment, the CPU 22 performs a pass / fail judgment based on rules in step 132.

[0090] Specifically, as described above, a predetermined standard is established that an inspected item with the above-mentioned defects such as voids and scratches is considered a defective product, and an inspected item without the above-mentioned defects is considered a good product. Then, as described above, template matching (using either the first or second method) is performed between the captured image and the image of the defective item, and the similarity between the two images is calculated.

[0091] In step 134, the CPU 22 determines whether the item to be inspected is defective or not. Specifically, the CPU 22 determines that the item is good if the similarity to all defective item images obtained in step 132 is less than a predetermined value, and determines that the item is defective due to a defective item if there is a defective item image with a similarity of more than the predetermined value.

[0092] If the item being inspected is not determined to be defective (i.e., it is determined to be good), the quality determination process proceeds to step 112. If the item being inspected is determined to be defective, the quality determination process proceeds to step 136.

[0093] In step 136, the CPU 22 uses the trained model 36M to determine whether the result is good or bad. The process in step 136 corresponds to steps 104-110 and 114 in Figure 7. After the process in step 136, the good / bad judgment process proceeds to step 112.

[0094] In step 112, the CPU 22 displays either a good item or a defective item on the display device 35. Specifically, if the item was not determined to be defective in step 134 (i.e., determined to be good), "good item" is displayed. If the item was determined to be defective in step 134, the output result of the trained model 36M in step 136 (good item or defective item) is displayed.

[0095] (effect) As explained above, in the third embodiment, rule-based filtering is used as the initial filtering, and if the product is not judged to be defective, i.e., judged to be good, good products are displayed and no good / bad judgment is made using the trained model 36M. Therefore, the processing can be simplified in the third embodiment.

[0096] (First and second modified examples of the third embodiment) Next, the first and second modified examples of the third embodiment will be described. Since the configuration of each modified example is the same as that of the third embodiment, their description will be omitted. Also, since the operation of these modified examples has some parts that are the same as those of the third embodiment, the same reference numerals will be used for the same parts and their descriptions will be omitted, and the parts that are mainly different will be described.

[0097] (First modified example of the third embodiment) In the third embodiment, rule-based filtering is used as the initial filtering, and if the product is not judged to be defective, i.e., judged to be good, good products are displayed and no quality determination is made using the trained model 36M. In contrast, in the first modification of the third embodiment, if the product is judged to be good using rule-based filtering, further quality determination is made using the trained model 36M, and if the product is judged to be defective using rule-based filtering, no quality determination is made using the trained model 36M.

[0098] In the first modification of the third embodiment, if a product is determined to be good using the rule-based approach, a further quality determination is made using the trained model 36M. That is, the result of determining a product to be good using the rule-based approach is verified by the quality determination using the trained model 36M. Therefore, the determination of whether a product is good can be made with greater accuracy than in the third embodiment.

[0099] (Second modified example of the third embodiment) <Perfect / failure determination process and method for determining whether the product is good or bad> In the third embodiment, rule-based filtering is used as the initial filtering, and if a product is determined to be defective, a pass / fail judgment is made using the trained model 36M. The technology of this disclosure is not limited thereto.

[0100] Figure 11 is a flowchart showing an example of a quality determination program 36P2 for a second modified example of the third embodiment.

[0101] If step 102 in Figure 10 results in a positive judgment, the process in step 132 (rule-based pass / fail judgment) is executed, followed by the process in step 136 (pass / fail judgment using the trained model 36M). Note that the processes in step 132 and step 136 may be executed in any order, or they may be executed simultaneously.

[0102] In step 142, the CPU 22 determines whether the product was judged as good based on either the process in step 132 or the process in step 136.

[0103] If the product is determined to be good in either step 132 or step 136, in step 144, the CPU 22 displays "Good Product" on the display device 35. If the product is not determined to be good in either step 132 or step 136, in step 146, the CPU 22 displays "Defective Product" on the display device 35 and also displays the defective items.

[0104] As described above, in the modified version of the third embodiment, rule-based quality determination is performed, and quality determination using the trained model 36M is performed. If the product is determined to be good in either determination, it is displayed as good. Therefore, quality determination can be performed with greater accuracy compared to when only rule-based quality determination or quality determination using the trained model 36M is performed.

[0105] [Other variations] In step 112 of Figures 7, 10, and 11 described above, the CPU 22 displays good or defective items on the display device 35, but the technology of this disclosure is not limited thereto. Figure 12 shows the display screen 35D of the display device 35, which displays the pass / fail judgment result along with the captured image 15 and the cropped image (i.e., partial image) 17 in comparison. As shown in Figure 12, in step 112, the CPU 22 displays the captured image 15 and the cropped image (i.e., partial image) 17 in correspondence on the display screen 35D. The CPU 22 also displays the quality judgment result 19, specifically whether it is a good product or a defective product, on the display screen 35D, and if it is a defective product, it also displays the defective item. In the example shown in Figure 12, the quality judgment result 19 is displayed as "defective product" and the defective item is displayed as "stain".

[0106] In the examples described above, the surface of the object to be inspected is used as an example, but the technology of this disclosure is not limited to this, and for example, the back surface or side surface of the object to be inspected may also be used. Each of the front, back, and side surfaces of the object to be inspected is a "surface of the object to be inspected" of the technology of this disclosure.

[0107] In the embodiments and modifications described above, examples of how the model learning process and pass / fail judgment process are performed by the information processing device 10 have been given, but the technology of this disclosure is not limited thereto. For example, the model learning process and pass / fail judgment process may be performed by a computer in an external device that is communicably connected to the information processing device 10 via a network such as a LAN (Local Area Network) or WAN (Wide Area Network). An example of a computer is a server computer for cloud services. The computer includes a processor, storage, and memory. The storage contains the model learning program 36P1 and the pass / fail judgment program 36P2. The computer is an example of an "information processing device" related to the technology of this disclosure.

[0108] The information processing device 10 requests an external device via the network to perform model learning processing and pass / fail judgment processing. In response, the processor of the external device reads the model learning program 36P1 and the pass / fail judgment program 36P2 from storage and executes the model learning program 36P1 and the pass / fail judgment program 36P2 in memory. The processor performs model learning processing and pass / fail judgment processing according to the model learning program 36P1 and the pass / fail judgment program 36P2 executed in memory.

[0109] The information processing device 10 performs the model learning process and the pass / fail judgment process, but this is merely one example. For example, the information processing device 10 and an external device may perform the model learning process and the pass / fail judgment process in a distributed manner, or the information processing device 10 and multiple devices including an external device may perform the model learning process and the pass / fail judgment process in a distributed manner.

[0110] In the embodiments and modifications described above, an example was given in which the model learning program 36P1 and the pass / fail judgment program 36P2 are stored in the storage device 36, but the technology of this disclosure is not limited thereto. For example, the model learning program 36P1 and the pass / fail judgment program 36P2 may be stored in a portable computer-readable non-temporary storage medium such as an SSD, USB memory, or magnetic tape. The model learning program 36P1 and the pass / fail judgment program 36P2 stored in the non-temporary storage medium are installed in the information processing device 10. The CPU 22 executes the model learning process and the pass / fail judgment process according to the model learning program 36P1 and the pass / fail judgment program 36P2.

[0111] Alternatively, the model learning program 36P1 and the pass / fail judgment program 36P2 may be stored in a storage device of another computer or server connected to the information processing device 10 via a network, and the model learning program 36P1 and the pass / fail judgment program 36P2 may be downloaded and installed on the information processing device 10 upon request.

[0112] Furthermore, it is not necessary to store all of the model learning program 36P1 and the pass / fail judgment program 36P2 in the storage device of another computer or server connected to the information processing device 10; it is acceptable to store only a portion of the model learning program 36P1 and the pass / fail judgment program 36P2.

[0113] In the embodiments and modifications described above, examples of how the technology of this disclosure is implemented by a software configuration are given. However, the technology of this disclosure is not limited thereto, and devices including ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), or PLDs (Programmable Logic Devices) may be applied. Furthermore, a combination of hardware and software configurations may be used.

[0114] The following types of processors can be used as hardware resources to perform the model learning process and pass / fail judgment process described in each of the above embodiments and modifications. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for performing the model learning process and pass / fail judgment process by executing software, i.e., a program. Other examples of processors include dedicated electronic circuits, which are processors with circuit configurations specifically designed to perform particular processes, such as FPGAs, PLDs, or ASICs. Each processor has built-in or connected memory, and each processor uses memory to perform the model learning process and pass / fail judgment process.

[0115] The hardware resources that perform the model learning process and the pass / fail judgment process may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform the model learning process and the pass / fail judgment process may consist of a single processor.

[0116] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs model learning and pass / fail judgment processing. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform model learning and pass / fail judgment processing, on a single IC (Integrated Circuit) chip, as exemplified by SoCs (System-on-a-chip). In this way, model learning and pass / fail judgment processing are realized using one or more of the above types of processors as hardware resources.

[0117] Furthermore, the hardware structure of these various processors can more specifically utilize electronic circuits that combine circuit elements such as semiconductor elements. Also, the model learning process and quality judgment process described above are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0118] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0119] [Note] Based on the above disclosures, the following addendum is proposed.

[0120] (Note 1) A model learning method in which the learning unit learns a model using training data comprising information on specific partial images within an image of a surface to be inspected that have an effect on determining whether it is good or bad, and information indicating whether the part of the surface to be inspected corresponding to the partial image is good or bad.

[0121] (Note 2) A method for generating a trained model, wherein the generation unit trains a model using training data comprising information on specific partial images that affect the quality determination of an image of a surface to be inspected, and information indicating the quality of the part to be inspected corresponding to said partial image.

[0122] (Note 3) A model learning program that causes a computer to perform a process that includes training a model using training data comprising information on specific partial images within an image of a surface to be inspected that affect the quality determination, and information indicating the quality of the part to be inspected corresponding to said partial image.

[0123] (Note 4) The photography department will photograph the surface of the object to be inspected, The determination unit determines whether the part of the inspection target corresponding to the partial image is good or bad, based on the trained model learned by the model learning device described in Appendix 1 and the information of the partial image in the image of the surface to be inspected obtained by the aforementioned photography. A method for determining whether something is good or bad, including the following. [Explanation of symbols]

[0124] 10 Information Processing Devices 20 Computers 32 Input devices 34 Imaging device 35 Display device 36 Storage device 36G image storage area 36M pre-trained model 36P1 Model Learning Program 36P2 Pass / Fail Judgment Program 36T training data storage area

Claims

1. A model learning device comprising a learning unit that learns a model using training data comprising information on specific partial images within an image of a surface to be inspected that have an effect on determining whether it is good or bad, and information indicating whether the part of the surface to be inspected corresponding to said partial image is good or bad.

2. The model learning device according to claim 1, wherein the learning unit determines information of the partial image from the partial image and predetermined rules.

3. A trained model trained using training data comprising information on specific partial images within an image of a surface to be inspected that influence the quality determination, and information indicating the quality of the corresponding part of the surface to be inspected.

4. A trained model generation device comprising a generation unit that generates a trained model by training a model using training data comprising information on specific partial images that affect the quality determination of an image of a surface to be inspected, and information indicating the quality of the part to be inspected corresponding to said partial image.

5. The camera unit takes pictures of the surface to be inspected, A determination unit that determines whether a part of the inspection target corresponding to a partial image is good or bad based on a trained model learned by the model learning device described in claim 1 and information from a partial image of the surface of the inspection target obtained by the photography, A determination device equipped with the following features.

6. The determination unit determines whether the surface to be inspected is good or bad using a rule-based initial filtering, and determines whether the surface is good or bad based on the information of a partial image of the surface to be inspected that is determined to be defective and a trained model learned by the model learning device described in claim 1, according to claim 5.

7. The determination unit performs a first quality determination using a rule base to determine whether the surface to be inspected is good or bad, and a second determination using information of a partial image of the surface to be inspected and a trained model trained by the model learning device described in claim 1 to determine whether the portion of the surface to be inspected corresponding to the partial image is good or bad, and the determination device according to claim 5 determines whether the surface to be inspected is good or bad based on the results of each determination.

Images