Computer program, information processing device, information processing method, and image generation model generation method

Abstract

A computer program, an information processing device, an information processing method, and an image generation model generation method are provided that are capable of evaluating the stability of the detection accuracy of a defect detection model. [Solution] The computer program causes a computer to execute the following process: acquire instruction information including an annotation image in which defect information is added to a product image; when the acquired instruction information is input into an image generation model, acquire a generated image generated by the image generation model; input the acquired generated image into a defect detection model; acquire defect detection information output by the defect detection model; and evaluate the defect detection model based on the acquired defect detection information and instruction information.

Description

[Technical Field]

[0001] The present invention relates to a computer program, an information processing device, an information processing method, and an image generation model generation method. [Background technology]

[0002] In recent years, labor shortages have become more serious, and labor-saving measures have become increasingly important at manufacturing sites in various fields. In this situation, automation of product inspection using artificial intelligence (AI) has attracted attention.

[0003] Patent Document 1 discloses a system in which original image data containing defects obtained by photographing an object to be inspected and image data in which the original image data is labeled with a predetermined color for each type of defect are used as first training data, and original image data without defects is prepared as second training data, and a defect detection model is trained using the first training data and the second training data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-60798 Summary of the Invention [Problem to be solved by the invention]

[0005] However, defect detection models such as those described in Patent Document 1 have difficulty dealing with unknown defects or defects not included in the training data (e.g., deformation of the defect shape or fluctuation in its position). For example, if a defect in the training data exists only in the upper right corner of an image, but the actual defect also exists in the lower left corner of the image, bias in the training data may cause fluctuations in defect detection accuracy. Furthermore, there is a high possibility that the user performing the inspection may not be aware of the bias in the training data, making it difficult to ensure the validity of the defect detection model and often making it difficult to grasp the stability of the detection accuracy of the defect detection model.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a computer program, an information processing device, an information processing method, and an image generation model generation method that are capable of evaluating the stability of the detection accuracy of a defect detection model. [Means for solving the problem]

[0007] The present application includes multiple means for solving the above-mentioned problems. As one example, a computer program causes a computer to execute the following process: acquire instruction information including an annotation image in which defect information is added to a product image; when the acquired instruction information is input into an image generation model, acquire a generated image generated by the image generation model; input the acquired generated image into a defect detection model; acquire defect detection information output by the defect detection model; and evaluate the defect detection model based on the acquired defect detection information and the instruction information. [Effects of the Invention]

[0008] According to the present invention, it is possible to evaluate the stability of the detection accuracy of a defect detection model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing apparatus according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an example of a processing scheme performed by an information processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an image generation model. [Figure 4] FIG. 10 is a diagram showing a first example of instruction information. [Figure 5] FIG. 10 is a diagram showing a second example of instruction information. [Figure 6] FIG. 10 is a diagram showing a third example of instruction information. [Figure 7] FIG. 10 is a diagram showing a fourth example of instruction information. [Figure 8] FIG. 10 is a diagram showing a fifth example of instruction information. [Figure 9] FIG. 10 is a diagram showing a sixth example of instruction information. [Figure 10] FIG. 10 is a diagram illustrating an example of defect detection information output by a defect detection model. [Figure 11] FIG. 10 is a diagram illustrating an example of an evaluation method when an instruction by instruction information is class information. [Figure 12] FIG. 10 is a diagram showing an example of precision and recall using a confusion matrix. [Figure 13] FIG. 10 is a diagram illustrating an example of an evaluation method when the instruction information indicates segmentation. [Figure 14] FIG. 10 is a diagram illustrating an example of an evaluation result of a defect detection model. [Figure 15] FIG. 10 is a diagram illustrating an example of a processing procedure of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will be described below. FIG. 1 is a diagram showing an example of the configuration of an information processing device 50 of this embodiment. The information processing device 50 can be connected to a defect inspection model DB (database) 100 via a communication network 1. The defect inspection model DB 100 records data of a plurality of different defect inspection models (such as programs and data required to execute the defect inspection models). The defect detection models recorded in the defect inspection model DB 100 are used as evaluation targets, the defect detection accuracy of which is evaluated by the information processing device 50. The defect detection models include, for example, models created by personnel of a company requesting the evaluation. The information processing device 50 can access the defect inspection model DB 100 and acquire the defect detection model to be evaluated.

[0011] The information processing device 50 includes a control unit 51 that controls the entire device, a communication unit 52, a memory 53, a display unit 54, an operation unit 55, a storage unit 56, and an evaluation unit 59. The information processing device 50 can evaluate the detection accuracy of the defect detection model to be evaluated. The information processing device 50 may be configured with multiple information processing devices with functions distributed.

[0012] The control unit 51 may be configured by incorporating a required number of central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), etc. The control unit 51 may also be configured by combining digital signal processors (DSPs), field-programmable gate arrays (FPGAs), etc.

[0013] The communication unit 52 includes a communication module and has a function of communicating with the defect inspection model DB 100 via the communication network 1.

[0014] The display unit 54 is configured with a liquid crystal display, an organic EL display, or the like, and provides a UI (user interface) by displaying required information. The display unit 54 is equipped with a microphone and a speaker, and can input and output audio.

[0015] The operation unit 55 is configured with, for example, a touch panel, and allows the user to operate icons displayed on the display unit 54, move and operate a cursor, input characters, etc. The operation unit 55 may be configured with buttons, switches, etc., or may be configured with a keyboard, mouse, etc. The operation unit 55 provides a UI by accepting user operations.

[0016] The storage unit 56 can be configured with a semiconductor memory, a hard disk, or the like, and stores a computer program 57 (program product), an image generation model 58, and required information. The image generation model 58 will be described in detail later.

[0017] The computer program 57 can be stored in the storage unit 56 by reading the computer program 57 recorded on a recording medium (e.g., an optically readable disk storage medium such as a CD-ROM) M using a recording medium reading unit (not shown). The computer program 57 may also be downloaded from an external device via the communication unit 52 and stored in the storage unit 56. The computer program 57 may also be read from a recording medium such as a storage device (semiconductor memory such as a solid-state drive (SSD)) connected by a standard for connecting to a computer (e.g., Universal Serial Bus (USB) or other standard) and stored in the storage unit 56. The computer program 57 can also be deployed to be executed on a single computer, or on multiple computers located at one site, or distributed across multiple sites and interconnected by a communication network.

[0018] The memory 53 can be configured with a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory. A computer program 57 can be loaded into the memory 53, and the control unit 51 can execute the computer program 57. The control unit 51 can execute processing defined by the computer program 57. In other words, processing by the control unit 51 is also processing by the computer program 57.

[0019] The evaluation unit 59 evaluates the detection accuracy of the defect detection model to be evaluated.

[0020] 2 is a diagram showing an example of a processing scheme by the information processing device 50. When the control unit 51 acquires a product image (annotation) and a prompt, the control unit 51 inputs the acquired product image (annotation) and prompt to the image generation model 58.

[0021] A product image (annotation) is an annotation image in which defect information is added to a product image, and is an image in which product defects are pseudo-added. Note that a product image (annotation) may also include a non-defective product image without defect information. Product defects include damage that would render the product defective, such as scratches, wrinkles, dirt, stains, tears, etc. Product defects may also include deformations such as folds and bends that cannot be classified as defects. Defect information includes information such as the type of defect (e.g., scratches, wrinkles, dirt, stains, tears, deformation, etc.), the shape of the defect, the position of the defect on the image, and the number of defects. Product images (annotations) can be prepared in advance by the user.

[0022] A prompt is an instruction input to the image generation model 58 to cause the image generation model 58 to generate an image. The prompt can be input to the information processing device 50 by a user via the operation unit 55, for example, and the information processing device 50 can acquire the input prompt. The product image (annotation) and the prompt are collectively referred to as instruction information.

[0023] When a product image (annotation) and a prompt are input, the image generation model 58 generates an image as instructed by the input prompt and product image (annotation), and outputs the generated image (also referred to as a "generated image"). For example, when a product image (annotation) with defects annotated and a prompt such as "Please generate an image in which defects similar to the product image exist" are input to the image generation model 58, the image generation model 58 outputs a generated image similar to the input product image (annotation).

[0024] The information output by the image generation model 58 includes defect information, deformation information, class information, etc. in addition to the generated image. The defect information includes the type, position, and shape of each of one or more defects included in the generated image. The deformation information includes the type, position, and shape of each of one or more deformations included in the generated image. The class information corresponds to one generated image and includes information such as "OK" or "NG." "OK" indicates that the generated image has no defect, and "NG" indicates that the generated image has a defect. The generated image also includes one or more areas where defects or deformations exist. The image generation model 58 may output the input product image (annotation) as the generated image as is. Details of the image generation model 58 will be described later.

[0025] The control unit 51 inputs the generated image generated by the image generation model 58 to the defect detection model 101. The defect detection model 101 performs inference on the input image and outputs defect detection information. The defect detection information includes areas of defects or deformations present on the image. The defect detection information may also include class information indicating whether or not at least one of a defect and a deformation exists in the input image (if present: "NG", if not present: "OK").

[0026] The control unit 51 inputs the defect detection information output by the defect detection model 101 and the product image (annotation) input to the image generation model 58 to the evaluation unit 59. The evaluation unit 59 outputs an evaluation result of the defect detection model 101 based on the defect detection information and the product image (annotation). The evaluation result includes, for example, the detection accuracy of the defect detection model 101. The method for evaluating the defect detection model 101 will be described in detail later.

[0027] As described above, the control unit 51 acquires instruction information including a product image (annotation) as an annotation image in which defect information is added to a product image, and when the acquired instruction information is input to the image generation model 58, it acquires a generated image generated by the image generation model 58, inputs the acquired generated image to the defect detection model 101, acquires defect detection information output by the defect detection model 101, and can evaluate the defect detection model 101 based on the acquired defect detection information and the product image (annotation) as instruction information.

[0028] This makes it possible to evaluate whether the defect detection model 101 correctly detects the defects contained in the defect information contained in the generated image generated by the image generation model 58 based on the instruction information, and to evaluate the stability of the detection accuracy of the defect detection model 101.

[0029] 3 is a diagram showing an example of the configuration of the image generation model 58. The image generation model 58 is configured, for example, with stable diffusion and uses a latent diffusion model, which is a type of diffusion model. The image generation model 58 includes a VAE (variational autoencoder) 581, an encoder 582, a decoder 583, a VAE 584, a text encoder 585, and the like.

[0030] The VAE 581 compresses the product images (annotations) from pixel space to a lower-dimensional latent space and maps the image data of the product images (annotations) to latent variables. The image generation model 58 can learn the representation in the latent space using the VAE 581.

[0031] The encoder 582 and decoder 583 constitute a U-Net block, which gradually adds Gaussian noise to the latent variables (latent representations) through a forward diffusion process, eventually resulting in pure noise. The encoder 582 and decoder 583 then use a backward diffusion process to gradually remove noise from the noisy data in the latent space, which is the output of the forward diffusion process, and reconstruct the original latent variables.

[0032] The VAE 584 transforms the latent variables reconstructed by the encoder 582 and decoder 583 back into pixel space to generate a generated image.

[0033] The text encoder 585 embeds text such as prompts to generate embedding vectors, and outputs the generated embedding vectors to each layer of the encoder 582 and the decoder 583. This allows the encoder 582 and the decoder 583 to add information representing the relationship between the text and latent variables to the noise data, thereby generating noise containing conceptual information about the image.

[0034] In this way, the image generation model 58 can generate a generated image by gradually adding noise to the product image (annotation) and then removing the noise, thereby reconstructing the original image data and generating a generated image, which allows new image data to be generated.

[0035] The image generation model 58 may be generated (trained) as follows. That is, the control unit 51 acquires training data including instruction information, which includes an annotation image in which defect information is added to a product image, and based on the acquired training data, the control unit 51 can generate the image generation model 58 so that, when the acquired instruction information is input, an image is generated to which the defect information included in the instruction information is added. Specifically, the noise data generated in the forward diffusion process is used to learn to accurately approximate the probability distribution in the dediffusion process.

[0036] Next, the instruction information to be input to the image generation model 58 will be described.

[0037] FIG. 4 is a diagram showing a first example of instruction information. The instruction information includes a product image (annotation) and a prompt. As shown in FIG. 4, the product is, for example, a retort pouch for retort foods, but is not limited to this. In the product image (annotation), a linear scratch (type of defect) is annotated as a defect on the surface in the upper right corner of the product, as shown by the dashed line. The prompt associated with the product image (annotation) can be, for example, "Please generate an image based on the defect in the product image." Note that there may be multiple areas where defects exist. Also, a normal area where no defects exist may be specified.

[0038] 4 is input, the image generation model 58 generates a generated image similar to the product image (annotation). The generated image shows a linear scratch (a type of defect) on the upper right surface of the product.

[0039] When the generated image is input, the defect detection model 101 infers defects contained in the generated image and outputs defect detection information. If the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a linear scratch (type of defect) exists as a defect on the upper right surface of the product.

[0040] The control unit 51 can compare the defect detection information output by the defect detection model 101 with the product image (annotation) input to the image generation model 58, and evaluate the detection accuracy of the defect detection model 101 based on the degree of match between the two.

[0041] As described above, the instruction information includes the type of defect, the generated image includes the defect of that type, and the control unit 51 can evaluate the defect detection model 101 based on the degree of match between the type of defect included in the defect detection information and the type of defect included in the instruction information.

[0042] FIG. 5 is a diagram showing a second example of instruction information. As shown in FIG. 5, the product image (annotation) is annotated with two defect locations, the top center and the left center of the product, as indicated by the dashed lines, and with circular regions that specify the shape of each defect. In this case, the specification is a so-called polygon specification, which specifies a shaped region. A prompt associated with the product image (annotation) can be, for example, "Please generate an image in which a defect exists around the annotated area of ​​the product image." Note that the number of defect areas may be one, or three or more. Alternatively, a normal region without any defects may be specified.

[0043] 5 is input, the image generation model 58 generates a generated image similar to the product image (annotation). The generated image has a circular defect in the center of the top of the product, and also has a circular defect in the center left of the product.

[0044] When the generated image is input, the defect detection model 101 infers defects contained in the generated image and outputs defect detection information. If the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a circular defect exists in the center of the top of the product and also in the center left of the product.

[0045] The control unit 51 can compare the defect detection information output by the defect detection model 101 with the product image (annotation) input to the image generation model 58, and evaluate the detection accuracy of the defect detection model 101 based on the degree of match between the two.

[0046] As described above, the instruction information includes the position of the defect, the generated image includes the defect at that position, and the control unit 51 can evaluate the defect detection model 101 based on the degree of correspondence between the position of the defect included in the defect detection information and the position of the defect included in the instruction information.

[0047] In addition, the instruction information includes the shape of the defect, and the generated image includes a defect of that shape, and the control unit 51 can evaluate the defect detection model 101 based on the degree of match between the shape of the defect included in the defect detection information and the shape of the defect included in the instruction information.

[0048] FIG. 6 is a diagram showing a third example of instruction information. As shown in FIG. 6, the product image (annotation) is annotated with a rectangular grid to specify the location of defects. In the example of FIG. 6, the grid specifies the approximate locations of two defects. A prompt associated with the product image (annotation) may be, for example, "Please generate an image in which there is a defect within the grid of the product image." The prompt may also be a complex instruction, for example, "Please generate an image in which there is a 100 px scratch within the top center grid and a 200 px stain within the left grid." Note that the number of areas containing defects may be one, three, or more. Normal areas in which no defects exist may also be specified.

[0049] 6 is input, the image generation model 58 generates a generated image similar to the product image (annotation). In the generated image, defects exist in each of the two grids.

[0050] When the generated image is input, the defect detection model 101 infers defects contained in the generated image and outputs defect detection information. If the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a defect exists in each of the two grids.

[0051] The control unit 51 can compare the defect detection information output by the defect detection model 101 with the product image (annotation) input to the image generation model 58, and evaluate the detection accuracy of the defect detection model 101 based on the degree of match between the two.

[0052] As described above, the instruction information includes the position of the defect, the generated image includes the defect at that position, and the control unit 51 can evaluate the defect detection model 101 based on the degree of correspondence between the position of the defect included in the defect detection information and the position of the defect included in the instruction information.

[0053] FIG. 7 is a diagram showing a fourth example of instruction information. As shown in FIG. 7, the product image (annotation) is annotated with a pseudo-deformation in which part of the packaging is folded, as indicated by a dashed line, at the bottom right of the product. The prompt associated with the product image (annotation) can be, for example, "Please generate an image with a pseudo-deformation in the annotated area of ​​the product image." Note that there may be multiple areas where a pseudo-deformation exists. Alternatively, a normal area where no pseudo-deformation exists may be specified.

[0054] 7 is input, the image generation model 58 generates a generated image similar to the product image (annotation). The generated image has a pseudo-deformation in which part of the packaging is folded at the bottom right of the product.

[0055] When the generated image is input, the defect detection model 101 infers the pseudo-deformation contained in the generated image and outputs defect detection information. If the defect detection model 101 makes a correct inference, the defect detection information includes information indicating that there is a pseudo-deformation in which part of the packaging is folded at the bottom right of the product.

[0056] The control unit 51 can compare the defect detection information output by the defect detection model 101 with the product image (annotation) input to the image generation model 58, and evaluate the detection accuracy of the defect detection model 101 based on the degree of match between the two.

[0057] As described above, the instruction information includes a pseudo-deformation, the generated image includes the pseudo-deformation, and the control unit 51 can evaluate the defect detection model 101 based on the degree of correspondence between the position and shape of the pseudo-deformation included in the defect detection information and the position and shape of the pseudo-deformation included in the instruction information.

[0058] FIG. 8 is a diagram showing a fifth example of instruction information. As shown in FIG. 8, a product image (annotation) has wrinkles annotated in the upper left of the product and stains annotated in the lower right of the product. A prompt associated with the product image (annotation) can be, for example, "Please generate an image of the product with wrinkles in the upper left and stains in the lower right." In this way, the prompt can provide contextual instructions that include multiple classes ("wrinkles" and "stains").

[0059] 8 is input, the image generation model 58 generates a generated image similar to the product image (annotation). The generated image has wrinkles in the upper left of the product and stains in the lower right of the product.

[0060] When the generated image is input, the defect detection model 101 infers the "wrinkles" and "stains" contained in the generated image and outputs defect detection information. If the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a wrinkle exists in the upper left of the product and that a stain exists in the lower right of the product.

[0061] The control unit 51 can compare the defect detection information output by the defect detection model 101 with the product image (annotation) input to the image generation model 58, and evaluate the detection accuracy of the defect detection model 101 based on the degree of match between the two.

[0062] As described above, the instruction information includes multiple classes based on the context, and the generated image includes multiple classes based on the context, and the control unit 51 can evaluate the defect detection model 101 based on the degree of correspondence between the positions and shapes of the "wrinkles" and "stains" included in the defect detection information and the positions and shapes of the "wrinkles" and "stains" included in the instruction information.

[0063] FIG. 9 is a diagram showing a sixth example of instruction information. As shown in FIG. 9, the product image (annotation) is annotated with a pseudo-deformation in which part of the packaging is folded at the bottom left of the product, and a scratch at the center of the product. A prompt associated with the product image (annotation) can be, for example, "Please generate an image in which the pseudo-deformation is at the bottom left of the product image and the scratch is at the center." In this way, the prompt can give instructions using a higher-level concept, a pseudo-scenario, which combines a specific pseudo-deformation and defect.

[0064] 9 is input, the image generation model 58 generates a generated image similar to the product image (annotation). The generated image has a pseudo-deformation in the lower left of the product and a scratch in the center of the product.

[0065] When the generated image is input, the defect detection model 101 infers the pseudo-deformation and flaws contained in the generated image and outputs defect detection information. If the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a pseudo-deformation exists in the lower left of the product and that a flaw exists in the center of the product.

[0066] The control unit 51 can compare the defect detection information output by the defect detection model 101 with the product image (annotation) input to the image generation model 58, and evaluate the detection accuracy of the defect detection model 101 based on the degree of match between the two.

[0067] As described above, the instruction information includes defects and pseudo-deformations, and the generated image includes the defects and pseudo-deformations, and the control unit 51 can evaluate the defect detection model 101 based on the degree of correspondence between the positions and shapes of the "defects" and "pseudo-deformations" included in the defect detection information and the positions and shapes of the "defects" and "pseudo-deformations" included in the instruction information.

[0068] Although not shown, the instruction information may include an instruction that there are no defects, and the generated image may not include any defects. The control unit 51 can evaluate the defect detection model 101 based on whether or not the defect detection information includes any defects. In this case, if the defect detection information indicates that there are no defects, it indicates that the defect detection model 101 has made a correct inference.

[0069] Next, the defect detection information output by the defect detection model 101 will be described.

[0070] 10 is a diagram showing an example of defect detection information output by the defect detection model 101. When a generated image generated by the image generation model 58 is input to the defect detection model 101, the defect detection model 101 outputs defect detection information by inferring defects and pseudo-deformations in the input generated image.

[0071] 10A, when a defect exists in the center left of the generated image, if the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a defect exists in the center left of the image. In the example of FIG. 10A, the defect detection information indicates that a defect exists that corresponds to the position and shape of the defect on the generated image.

[0072] 10B, when a defect exists at the top center of the generated image, if the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a defect exists at the top center of the image. In the example of FIG. 10B, the defect detection information indicates that a defect exists that corresponds to the position and shape of the defect on the generated image.

[0073] As shown in Fig. 10C, when a defect exists in the lower left of the generated image, if the defect detection model 101 can make a correct inference, the defect detection information includes information indicating that a defect exists in the lower left of the image. In the example of Fig. 10C, the defect detection information indicates that a defect exists that corresponds to the position and shape of the defect on the generated image.

[0074] The defect detection information includes information such as the type of defect (for example, scratches, wrinkles, stains, blemishes, tears, deformations, etc.), the shape of the defect, the position of the defect on the image, and the number of defects.

[0075] Next, a method for evaluating the defect detection model 101 will be described.

[0076] Fig. 11 is a diagram showing an example of an evaluation method when the instruction information indicates class information. The instruction information indicates the presence of a defect (which may include pseudo-deformation), for example, as shown in Figs. 4, 6, 7, 8, and 9. That is, the class information can be classified into "defect present" and "no defect present."

[0077] As shown in FIG. 11, correct labels can be classified into "defect present" and "no defect present." Furthermore, the defect detection information output by the defect detection model 101 can be classified into "defect present" and "no defect present." A state in which the correct label correctly infers "defect present" as "defect present" is called a TP (True Positive) class. A state in which the correct label erroneously infers "defect present" as "no defect present" is called an FN (False Negative) class. A state in which the correct label erroneously infers "no defect present" as "defect present" is called an FP (False Positive) class. A state in which the correct label correctly infers "no defect present" as "no defect present" is called a TN (True Negative) class. The table shown in FIG. 11 is called a confusion matrix.

[0078] Using the confusion matrix, precision and recall can be used as indices for evaluating the defect detection model 101. Precision can be expressed as precision = TP / (TP + FP), and indicates the ratio of data that was actually "defective" among data inferred to be "defective." Recall indicates whether "defective" data was actually correctly inferred to be "defective."

[0079] FIG. 12 is a diagram showing an example of precision and recall using a confusion matrix. In the example of FIG. 12, image generation model 58 generates nine generated images with defects and 12 generated images without defects, and each generated image is input to defect detection model 101 to show the inference results. Since all nine pieces of data for which the detection result inferred that there is a defect actually do have a defect, precision is 100%. Furthermore, since the nine pieces of data that actually had a defect were correctly inferred as having a defect, recall is 100%. Note that the number of pieces of data is shown for convenience's sake.

[0080] Fig. 13 is a diagram showing an example of an evaluation method when the instruction information specifies segmentation. When the instruction information specifies segmentation, it specifies an area (position and shape) where a defect (which may include pseudo-deformation) exists, as shown in Fig. 5. In other words, segmentation can specify the area of ​​the defect.

[0081] As shown in FIG. 13, the area P indicated by the dashed line is the annotation area (correct area). The annotation area P is the area indicated by the annotation. The area Q indicated by the solid line is the defect area indicated by the defect detection information resulting from the inference by the defect detection model 101. In this case, the defect detection model 101 can be evaluated by IoU, which is an accuracy index. IoU can be expressed as IoU=(P∩Q) / (P∪Q). IoU is an index that indicates how much the area P and the area Q overlap; in the case of a perfect match, IoU=1.0, and in the case of no overlap at all, IoU=0.0.

[0082] 14 is a diagram showing an example of the evaluation results of the defect detection model 101. The defect detection model 101 with ID=1001 is selected from multiple defect detection models. Assume that there are 100 generated images including defects (including pseudo-deformations) inferred by the defect detection model 101. Of the 100 generated images, for example, 50 generated images include pseudo-defect (defect) a1, 1 generated image includes pseudo-defect a2, 40 generated images include pseudo-deformation b1, and 9 generated images include pseudo-scenario c1.

[0083] Evaluation of the defect detection model 101 shows that for generated images containing 50 pseudo defects (defects) a1, the accuracy is 95% and the recall is 95%, for generated images containing one pseudo defect a2, the IoU is 0.92, for generated images containing 40 pseudo deformations b1, the accuracy is 93% and the recall is 97%, and for generated images containing nine pseudo scenarios c1, the accuracy is 90% and the recall is 85%.

[0084] 15 is a diagram showing an example of a processing procedure of the information processing device 50. The control unit 51 acquires instruction information including an annotation image and a prompt (S11). The annotation image is a product image (annotation) in which defect information is added to the product image. The control unit 51 inputs the acquired instruction information to the image generation model 58 (S12), and acquires a generated image generated by the image generation model 58 (S13).

[0085] The control unit 51 inputs the acquired generated image to the defect detection model 101 (S14), and acquires defect detection information output by the defect detection model 101 (S15). The control unit 51 compares the acquired defect detection information with the instruction information to evaluate the defect detection model 101 (S16), and ends the process.

[0086] According to this embodiment, pseudo defect images can be generated according to an assumed scenario, and the stability of the detection accuracy of defect detection model 101 can be automatically evaluated. In addition, any defect image can be generated using image generation model 58, so that defect detection model 101 can be evaluated while covering defects that the user of defect detection model 101 has not learned, and this can assist in building defect detection model 101 that can handle unknown defects.

[0087] (Supplementary Note 1) The computer program causes a computer to execute the following process: acquire instruction information including an annotation image in which defect information is added to a product image; when the acquired instruction information is input into an image generation model, acquire a generated image generated by the image generation model; input the acquired generated image into a defect detection model; acquire defect detection information output by the defect detection model; and evaluate the defect detection model based on the acquired defect detection information and the instruction information.

[0088] (Appendix 2) In Appendix 1, the computer program causes a computer to execute a process in which the instruction information includes a type of defect, the generated image includes a defect of the type, and the defect detection model is evaluated based on the degree of match between the type of defect included in the defect detection information and the type of defect included in the instruction information.

[0089] (Appendix 3) The computer program, in Appendix 1 or Appendix 2, causes a computer to execute a process in which the instruction information includes a position of a defect, the generated image includes a defect at the position, and the defect detection model is evaluated based on the degree of coincidence between the position of the defect included in the defect detection information and the position of the defect included in the instruction information.

[0090] (Appendix 4) A computer program, in any one of Appendices 1 to 3, causes a computer to execute a process in which the instruction information includes a shape of a defect, the generated image includes a defect of the shape, and the defect detection model is evaluated based on the degree of match between the shape of the defect included in the defect detection information and the shape of the defect included in the instruction information.

[0091] (Appendix 5) A computer program in any one of Appendices 1 to 4 causes a computer to execute a process in which the instruction information includes an instruction that there is no defect, the generated image does not include a defect, and the defect detection model is evaluated based on whether the defect detection information includes a defect.

[0092] (Appendix 6) The information processing device includes a control unit, which acquires instruction information including an annotation image in which defect information is added to a product image, and when the acquired instruction information is input to an image generation model, acquires a generated image generated by the image generation model, inputs the acquired generated image into a defect detection model, acquires defect detection information output by the defect detection model, and evaluates the defect detection model based on the acquired defect detection information and the instruction information.

[0093] (Appendix 7) An information processing method acquires instruction information including an annotation image in which defect information is added to a product image, and when the acquired instruction information is input into an image generation model, acquires a generated image generated by the image generation model, inputs the acquired generated image into a defect detection model, acquires defect detection information output by the defect detection model, and evaluates the defect detection model based on the acquired defect detection information and the instruction information.

[0094] (Appendix 8) The image generation model generation method acquires training data including instruction information, which includes an annotation image in which defect information is added to a product image, and generates an image generation model based on the acquired training data so that, when the acquired instruction information is input, an image to which the defect information included in the instruction information is added is generated.

[0095] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limited to this format. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used. [Explanation of symbols]

[0096] 1. Communication Network 50 Information processing equipment 51 Control section 52 Communications Department 53 Memory 54 Display section 55 Operation section 56 Memory section 57 Computer Programs 58 Image Generation Model 59 Evaluation Department 100 Defect detection model DB

Claims

1. Obtain instruction information including an annotation image in which defect information is added to a product image; When the acquired instruction information is input to an image generation model, a generated image generated by the image generation model is acquired; inputting the acquired generated image into a defect detection model, and acquiring defect detection information output by the defect detection model; evaluating the defect detection model based on the acquired defect detection information and the instruction information; A computer program that causes a computer to perform a process.

2. the indication includes a type of defect; the generated image includes the type of defect; evaluating the defect detection model based on the degree of coincidence between the type of defect included in the defect detection information and the type of defect included in the instruction information; 2. The computer program according to claim 1, which causes a computer to execute a process.

3. the indication includes a location of the defect; the generated image includes a defect at the location; evaluating the defect detection model based on the degree of coincidence between the positions of the defects included in the defect detection information and the positions of the defects included in the instruction information; 3. The computer program according to claim 1, which causes a computer to execute a process.

4. the indication includes a shape of the defect; the generated image includes the shape defect; evaluating the defect detection model based on the degree of coincidence between the shape of the defect included in the defect detection information and the shape of the defect included in the instruction information; 3. The computer program according to claim 1, which causes a computer to execute a process.

5. the indication includes an indication of no defect; the generated image is free of defects; evaluating the defect detection model based on whether the defect detection information includes a defect; 3. The computer program according to claim 1, which causes a computer to execute a process.

6. A control unit is provided, The control unit Obtain instruction information including an annotation image in which defect information is added to a product image; When the acquired instruction information is input to an image generation model, a generated image generated by the image generation model is acquired; inputting the acquired generated image into a defect detection model, and acquiring defect detection information output by the defect detection model; evaluating the defect detection model based on the acquired defect detection information and the instruction information; Information processing device.

7. Obtain instruction information including an annotation image in which defect information is added to a product image; When the acquired instruction information is input to an image generation model, a generated image generated by the image generation model is acquired; inputting the acquired generated image into a defect detection model, and acquiring defect detection information output by the defect detection model; evaluating the defect detection model based on the acquired defect detection information and the instruction information; Information processing methods.

8. Acquire training data including instruction information including annotation images in which defect information is added to product images; generating an image generation model based on the acquired training data so that, when the acquired instruction information is input, an image to which the defect information included in the instruction information is added is generated; Image generation model generation method.

Images