Electronic apparatus and method for generating images of defective products and performing visual inspection by using same

Abstract

An electronic apparatus according to an embodiment of the present invention may comprise a processor which: acquires a training data set including images of external structure of normal products and first prompt information corresponding thereto, images of internal structure of the normal products and second prompt information corresponding thereto, and images of defective product tests and third prompt information corresponding thereto; constructs an image generation model by applying a feature model generated on the basis of the training data set to a base model trained to generate images of defective products; inputs, into the image generation model, defect information in the form of at least one of text and an image representing a defect and related to the location of the defect in a product; and acquires, through the image generation model, images of defective products including the defects corresponding to the defect information.

Description

Electronic device and method for generating a defective product image and performing vision inspection using the same

[0001] The present invention relates to an electronic device and method for generating an image of a defective product and performing a vision inspection using the same.

[0002] With the recent advancements in related technologies like the Internet of Things (IoT) and Artificial Intelligence (AI), smart factories are becoming a hot topic. A smart factory is a manufacturing system that operates by applying Information and Communications Technology (ICT) combined with digital automation solutions throughout the production process, from design and development to manufacturing.

[0003] One technology related to smart factories is vision inspection, which detects product defects during the manufacturing process. To perform vision inspection, a vision inspection model trained to determine product defects using images of both normal and defective products must first be developed. The vision inspection model then determines whether a product is pass or fail based on images captured by cameras installed on the manufacturing line.

[0004] However, considering the ratio of good to defective products shipped from manufacturing production lines, the number of good products is overwhelmingly high. This biases the training data for vision inspection models toward good products, inevitably reducing the model's ability to detect defective products.

[0005] Currently, the model is created based on a limited number of defective cases, and the model is retrained and improved whenever an undetected defective product occurs.

[0006] This is inefficient and diminishes the value of introducing an automatic vision inspection solution, so there is a need to secure images of defective products for training vision inspection models.

[0007] An object of the present invention is to provide an electronic device and method for generating defective product images more diversely and randomly.

[0008] An object of the present invention is to provide an electronic device and method for generating a defective product image for a type of failure that reveals not only the appearance of the product but also the internal structure of the product.

[0009] An object of the present invention is to provide an electronic device that generates a defective product image and performs vision inspection using the image to train a higher-performance vision inspection model.

[0010] In one embodiment of the present invention, an electronic device may include a processor that acquires a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image, applies a feature model generated based on the learning data set to a base model trained to generate a defective product image to build an image generation model, inputs defect information related to a location of a defect in a product and in the form of at least one of text and an image representing the defect into the image generation model, and acquires a defective product image including a defect corresponding to the defect information through the image generation model.

[0011] The above feature model may be a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

[0012] The above defect information may further include text describing features to be excluded from the defective product image. The processor may receive defect information including a masking image indicating defects that may occur in the product.

[0013] The processor can adjust the external structure image, the internal structure image, and the defective product test image to each include a target area, and generate the feature model using the adjusted images.

[0014] The processor can input defect information including the adjusted internal structure image into the image generation model to obtain an adjusted defective product image, and can generate a defective product image by combining the adjusted defective product image and the remaining area of ​​the internal structure image excluding the target area.

[0015] The above processor can store the defect information by matching it with the acquired defective product image.

[0016] The above processor can filter the acquired defective product image by comparing the external structure image or the internal structure image with the acquired defective product image, depending on whether the degree of defect exceeds a threshold.

[0017] A method for generating a defective product image performed by an electronic device according to one embodiment of the present invention may include: acquiring a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image; applying a feature model generated based on the learning data set to a base model trained to generate a defective product image to build an image generation model; inputting defect information related to a location of a defect in a product and formed in the form of at least one of text and an image representing the defect into the image generation model; and acquiring a defective product image including a defect corresponding to the defect information through the image generation model.

[0018] The above defect information may further include text describing features to be excluded from the defective product image. The step of inputting the defect information into the image generation model may include a step of receiving defect information including a masking image indicating defects that may occur in the product.

[0019] The step of building the image generation model may include the step of adjusting the external structure image, the internal structure image, and the defective product test image to each include a target area; and the step of generating the feature model using the adjusted images.

[0020] A method comprising: a step of obtaining the defective product image, comprising: a step of obtaining an adjusted defective product image by inputting defect information including the adjusted internal structure image into the image generation model; and a step of generating a defective product image by combining the adjusted defective product image and the remaining area of ​​the internal structure image excluding the target area.

[0021] After the step of acquiring the defective product image, a step of comparing the acquired defective product image with the external structure image or the internal structure image and filtering the acquired defective product image depending on whether the degree of defect exceeds a threshold may be included.

[0022] According to one embodiment of the present invention, the scope of defective product images can be expanded by generating defective product images that are related to the internal structure of the product and include defects.

[0023] According to one embodiment of the present invention, defect inspection can be applied even when the inside of a product is visible while the outside of the product is damaged, thereby further improving the performance of a vision inspection model.

[0024] FIG. 1 is a schematic diagram illustrating an electronic device according to one embodiment of the present invention.

[0025] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0026] FIG. 3 is a diagram illustrating an operation flow diagram of an electronic device according to one embodiment of the present invention.

[0027] FIG. 4 is a drawing illustrating a construction of a feature model according to one embodiment of the present invention.

[0028] FIG. 5 is a drawing showing an external structural image and an internal structural image of a normal product according to one embodiment of the present invention.

[0029] FIG. 6 is a drawing illustrating a defective product image generation operation of an electronic device according to a first embodiment of the present invention.

[0030] FIG. 7 is a drawing illustrating a defective product image generation operation of an electronic device according to a second embodiment of the present invention.

[0031] FIG. 8 is a drawing illustrating a defective product image generation operation of an electronic device according to a third embodiment of the present invention.

[0032] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. In the drawings, portions irrelevant to the description may be omitted for clarity in describing the present invention, and the same reference numerals may be used throughout the specification for identical or similar components.

[0033] FIG. 1 is a schematic diagram illustrating an electronic device according to one embodiment of the present invention.

[0034] An electronic device (100) according to one embodiment of the present invention is a device that generates a defective product image and performs vision inspection using the same, and can be implemented as a computer, server, laptop, smart phone, tablet PC, smart pad, etc.

[0035] An electronic device (100) can construct an image generation model (20) using defect information (10) utilized to generate a defective product image, and obtain a defective product image (30) from the constructed image generation model (20). Through this, a sufficient number of defective product images (30) and normal product images (40) can be obtained, and a vision inspection model (50) can be constructed with balanced proportions.

[0036] Subsequently, the electronic device (100) can input a product image (60) obtained from the process line into a vision inspection model (50) to inspect good or defective products and obtain inspection information (70).

[0037] As illustrated in FIG. 1, the electronic device (100) of the present invention can perform all of the following: constructing an image generation model (20) for generating a defective product image (30), generating a defective product image (30) using the image generation model (20), constructing a vision inspection model (50) using the defective product image (30) and a normal product image (40), and performing product inspection using the vision inspection model (50). However, a plurality of electronic devices (100) may be separately provided to individually or selectively perform the operations listed above, and are not limited to any one of them.

[0038] As described above, the currently built vision inspection model has a problem in that its performance is poor due to the imbalance in learning data between normal and defective product images.

[0039] In the present invention, a method for generating various defective product images to improve the biased performance of a vision inspection model is proposed, and subsequently, a method for constructing and utilizing a vision inspection model by utilizing the defective product images is proposed.

[0040] Additionally, the present invention proposes to implement a defective product image that includes a defect that occurred in the internal structure of the product, as well as a defective product image that includes a defect that occurred in the appearance of the product.

[0041] Through this, the present invention can expand the defect range of a defective product image and further increase the applicability of a vision inspection model using the same.

[0042] Hereinafter, the configuration and operation of an electronic device (100) according to one embodiment of the present invention will be specifically described with reference to the drawings.

[0043] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0044] An electronic device (100) according to one embodiment of the present invention may include an input unit (110), a communication unit (120), a display unit (130), a storage unit (140), and a processor (150).

[0045] The input unit (110) generates input data in response to user input of the electronic device (100). For example, the user input may be a user input that initiates the operation of the electronic device (100), a user input required to build an image generation model and a vision inspection model, a user input that inputs defect information, a user input that sets a target area, etc. In addition, if it is a user input required to generate a defective product image and perform vision inspection, it can be applied without limitation.

[0046] The input unit (110) includes at least one input means. The input unit (110) may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a mouse, a menu button, etc.

[0047] The communication unit (120) can perform communication with an external device such as a server to transmit and receive defect information, an external structure image of a normal product, an internal structure image of a normal product, a defective product test image, prompt information corresponding to each image, inspection information, etc.

[0048] To this end, the communication unit (120) can perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Wi-Fi (wireless fidelity), Bluetooth, or wired communication such as LAN (local area network), WAN (Wide Area Network), and power line communication.

[0049] The display unit (130) displays display data according to the operation of the electronic device (100). The display unit (130) can display a screen that displays the process of creating a defective product image, a screen that displays the created defective product image, a screen that performs vision inspection, a screen that receives user input, etc.

[0050] The display unit (130) includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (130) may be implemented as a touch screen by being combined with the input unit (110).

[0051] The storage unit (140) stores the operation programs of the electronic device (100). The storage unit (140) includes non-volatile storage that can preserve data (information) regardless of whether power is supplied, and volatile memory that cannot preserve data if power is not supplied and into which data to be processed by the processor (150) is loaded. The storage includes flash memory, hard-disc drive (HDD), solid-state drive (SSD), read-only memory (ROM), etc., and the memory includes buffer, random access memory (RAM), etc.

[0052] The storage unit (140) can store defect information (10), an image generation model (20), a defective product image (30), a normal product image (40), a vision inspection model (50), a product image (60), inspection information (70), etc. The storage unit (140) can store operation programs, etc. required in the process of constructing an image generation model (20), creating a defective product image (30) using the image generation model (20), constructing a vision inspection model (50), and performing product inspection using the vision inspection model (50).

[0053] The processor (150) can control at least one other component (e.g., hardware or software component) of the electronic device (100) by executing software such as a program, and can perform various data processing or operations.

[0054] According to one embodiment of the present invention, a processor (150) obtains a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image, applies a feature model generated based on the learning data set to a base model trained to generate a defective product image, constructs an image generation model, inputs defect information in the form of at least one of text and an image representing a defect that may occur in the internal structure of a product into the image generation model, and obtains a defective product image including a defect corresponding to the defect information through the image generation model.

[0055] A processor (150) according to one embodiment of the present invention can generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information regarding defects included in the defective product images, and normal product images.

[0056] At this time, the processor (150) may construct at least one of an image generation model and a vision inspection model, or may receive and store a previously constructed image generation model and a vision inspection model from the outside and use them, but is not limited to either one.

[0057] Meanwhile, the processor (150) may perform at least a portion of the data analysis, processing, and result information generation for performing the above operations using at least one of a machine learning, neural network, or deep learning algorithm as a rule-based or artificial intelligence (AI) algorithm. Examples of the neural network may include models such as a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), and an RNN (Recurrent Neural Network).

[0058] FIG. 3 is a diagram illustrating an operation flow diagram of an electronic device according to one embodiment of the present invention.

[0059] A processor (150) according to one embodiment of the present invention can obtain a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image (S10).

[0060] An external structural image of a normal product refers to an image of the product's exterior, while an internal structural image refers to an image that allows for the internal structure of the product to be seen. For example, an internal structural image could be an image of a normal product with its cover removed. Furthermore, an internal structural image can be an image taken during the manufacturing process, or it can be sufficient if it allows for at least a portion of the product's interior to be seen.

[0061] Prompt information represents each image and can be constructed by matching each image. Each image and prompt information can be acquired externally or generated through user input that inputs prompt information corresponding to each image. The acquisition path or method is not limited to any one.

[0062] For example, in the case of an image of the external structure of a normal product, it can be matched with first prompt information such as "Circuit breaker" (product name), "Plastic texture," "Outer structure," etc. In addition, in the case of an image of the internal structure of a normal product, it can be matched with second prompt information such as "Circuit breaker" (product name), "Plastic texture," "Inner structure," etc. In the case of a test image of a defective product, it can be matched with third prompt information such as "Circuit breaker" (product name), "Plastic texture," "stained," "crack," etc.

[0063] A processor (150) according to one embodiment of the present invention can build an image generation model by applying a feature model generated based on a learning data set to a base model trained to generate a defective product image (S20).

[0064] Defective product images refer to product images created to include external defects such as scratches, breaks, and dents on the product.

[0065] The image generation model of the present invention is a generative artificial intelligence model, and can be constructed by applying a feature model implemented to fine-tune the features of a base model learned to generate defective product images.

[0066] The base model can be selected based on a model that best reflects the characteristics of the target product. This model varies depending on field and product conditions, so it's recommended that field experts conduct testing to select the model. For example, the base model could be Stable Diffusion, a denoising technique-based model. Among the Stable Diffusion models, the base model could include the v1.5 pruned-emaonly.safetensor model.

[0067] However, generative AI models are primarily trained on people, so fine-tuning is necessary to ensure the output images correspond to the product. To achieve this, a feature model can be added to the base model to build an image generation model. This feature model can be, for example, a Low-Rank Adaptation (LoRA) model. Even if the number of defective product test images is small when creating a LoRA model, this is not a major issue because the base model contains a wealth of defect information about the defect types.

[0068] There are several ways to create an image generation model, including using the Dreambooth Extension module of Stable Diffusion and using the Kohya-ss module.

[0069] According to one embodiment of the present invention, a feature model can learn about not only the external structure of a normal product but also its internal structure using a previously constructed learning data set. Therefore, an image generation model built using this feature model can easily generate images of defective products even when the internal structure of the product is visible during the defect generation process.

[0070] A processor (150) according to one embodiment of the present invention can input defect information in the form of at least one of text and image representing a defect, which is related to a location where a defect occurs in a product, into an image generation model (S30).

[0071] Defect information is the information required to create a defective product image. Defect information may include text describing potential defects in a product, particularly text related to the location of the defect and text describing the defect itself (hereinafter referred to as "first text"), text describing features to be excluded from the defective product image (hereinafter referred to as "second text"), a normal product image, a masking image displaying potential defects in the product, and more.

[0072] At this time, the first text and the second text may be set differently depending on the type of defective product image.

[0073] For example, text related to the location of a defect may be, for example, "Plastic Texture," "Inner Structure," "Under Cover," "Cover," etc. Text describing the defect itself may be, for example, "stained," "crack," "failure," "dent," "leakage," "damaged," "broken," "open," etc. In this case, the text related to the location of a product defect and the text indicating the defect itself may be combined and entered, for example, "Open Inner Structure." In addition, various other applications may be applied, and the content or expression format included in the first text is not limited to any one.

[0074] The second text is text that indicates features to be excluded from the defective product image, and may include, for example, “clean”, “clear”, “pass”, “normal”, etc.

[0075] At this time, if you want to create a defective product image where only the external structure is defective and the internal structure is free of defects, you can input "Inner structure", "Under cover", etc. as the second text. A more specific example is described with reference to Fig. 6.

[0076] The first text and the second text can be input in the form of a prompt, and the processor (150) can apply the defect information with an appropriate weight.

[0077] Text-based defect information can include additional product information. For example, if the target product is an MCCB, defect information can include terms such as "circuit braker" or "MCCB."

[0078] Normal product images can be base information for the image generation model to reference when generating defective product images.

[0079] A processor (150) according to one embodiment of the present invention can obtain a defective product image including a defect corresponding to defect information through an image generation model (S40).

[0080] The processor (150) can input a normal product image—i.e., an external structure image or an internal structure image—into an image generation model and acquire a defective product image based on the input (IMG2IMG function). At this time, the normal product image can utilize one or more of the normal product test images used to form the feature model.

[0081] The processor (150) can acquire a defective product image by targeting the entire area of ​​a normal product image, or can acquire a defective product image by using an inpaint function that masks only a specific area of ​​a normal product image so that a defect is generated. In particular, a masking image is an image that indicates a defect in a specific area, and the processor (150) can change only a specific area of ​​a normal product image into a masking image by using the inpaint function.

[0082] When utilizing the Inpaint function, it is more efficient when creating images to focus on improving types that were not detected by the existing image model or were over-detected as defects when they were not.

[0083] Unlike conventional generative models, generative models based on denoising techniques are advantageous for generating defects from normal product images, demonstrating superior image quality and image generation efficiency. Therefore, while the present invention is described based on examples generated using a generative model based on denoising techniques, the same invention structure can be utilized even if more advanced solutions emerge in the future.

[0084] According to one embodiment of the present invention, the biased performance of existing vision inspection models can be dramatically improved by generating a variety of defective product images. Furthermore, by utilizing a generative model, defective product images can be extracted at a very high productivity rate, with quality nearly identical to actual images. Ultimately, this can shorten the development time of vision inspection models and enhance their performance.

[0085] According to one embodiment of the present invention, the scope of defective product images can be expanded by generating defective product images that are related to the internal structure of the product and include defects.

[0086] According to one embodiment of the present invention, defect inspection can be applied even when the inside of a product is visible while the outside of the product is damaged, thereby further improving the performance of a vision inspection model.

[0087] FIG. 4 is a drawing illustrating a construction of a feature model according to one embodiment of the present invention.

[0088] FIG. 4 is a diagram illustrating the construction of an image generation model according to one embodiment of the present invention. FIG. 4 illustrates the construction of an image generation model, as described in relation to S10 of FIG. 3.

[0089] An electronic device (100) according to one embodiment of the present invention can build an image generation model (430) by applying a feature model (420) generated based on a learning data set (421) to a previously built base model (410).

[0090] At this time, when constructing an image generation model (430), the processor (150) can set defect information and hyperparameters. Hyperparameters are variables that directly affect the image generation quality, and a process of finding optimal values ​​through repeated execution is required.

[0091] FIG. 5 is a drawing showing an external structural image and an internal structural image of a normal product according to one embodiment of the present invention.

[0092] The left image of Fig. 5 is an external structural image (510) of a normal product, and the right image is an internal structural image (520) of a normal product with the cover removed. If a request is made to generate a defective product image with a visible defect in the internal structure, a feature model trained solely on the external structural image (510) will have difficulty generating a suitable defective product image because it cannot properly represent the internal structure of the product.

[0093] On the other hand, in the present invention, such internal structure images (520) are also learned, so that various types of defective product images including the internal structure can be generated.

[0094] FIG. 6 is a diagram illustrating a defective product image generation process of an electronic device according to a first embodiment of the present invention. FIG. 6 illustrates a process of generating a defective product image, as described in relation to S30 of FIG. 3.

[0095] Referring to FIG. 6, the processor (150) can input defect information including a normal product image (610), a first text (620), and a second text (630) into an image generation model (430) that applies a Laura model (420) to a base model (410) to generate a defective product image (640).

[0096] Meanwhile, the present invention enables the generation of various types of defective product images by adding a set of internal structure images. For example, defective product images can be generated for types where defects occur only in the external structure without revealing the internal structure, types where defects occur only in the external structure with the internal structure revealed, and types where defects occur in both the internal and external structures with the internal structure revealed.

[0097] In order to generate each type of defective product image, it is necessary to appropriately select a first text (620) indicating a defect and a second text (630) describing a feature to be excluded from the defective product image and input them into the image generation model (430).

[0098] For example, to create a defective product image of a type in which only the outer structure is defective and the inner structure is not exposed, a first text (620) including "Circuit breaker, Plastic Texture, Broken, Damaged" and a second text (630) including "Open Inner Structure, heavy damaged cover" can be input.

[0099] In order to create a defective product image of a type in which only the external structure is defective while the internal structure is exposed, a first text (620) including "Circuit breaker, Plastic Texture, Broken, Damaged, heavy damaged cover, Open Inner Structure, Clean Inner Structure" and a second text (630) including "Damaged Inner Structure" can be input.

[0100] At this time, if the first text (620) and the second text (630) are entered so that a defect occurs only in the external structure while the internal structure is exposed, the product may include a defect in which the surface paint of the product is partially peeled off and the internal structure is visible, as in the defective product image (640) shown in FIG. 6.

[0101] In order to create a defective product image of a type in which defects have occurred in the internal structure and the external structure with the internal structure exposed, a first text (620) including "Circuit breaker, Plastic Texture, Broken, Damaged, heavy damaged cover, Open Inner Structure, Damaged Inner Structure" and a second text (630) including "Clean Inner Structure" can be input.

[0102] The processor (150) can likewise set defect information and hyper parameters, and generate a defective product image (640) through inpaint settings.

[0103] Meanwhile, the processor (150) may input a normal product image (610) corresponding to the defective product image to be generated. For example, if the defective product image to be generated includes a defect occurring in the external structure, the processor (150) may input defect information including the external structure image into the image generation model. Conversely, if the defective product image to be generated includes a defect while showing the internal structure, the processor (150) may input defect information including the internal structure image into the image generation model.

[0104] FIG. 7 is a drawing illustrating a defective product image generation operation of an electronic device according to a second embodiment of the present invention.

[0105] The defective product image generation method described above with reference to FIG. 6 generates a defective product image by inputting the entire normal product image (610). However, due to the high resolution of the vision inspection image, an issue may arise where the image generation model cannot be trained and the image cannot be generated due to insufficient computing resources. FIG. 7 proposes a method for conserving computing resources by adjusting the normal product image.

[0106] The processor (150) can adjust the learning data set (710) including the external structural image of a normal product, the internal structural image, and the defective product test image so that each includes a target area. The target area refers to an area in which defects are primarily determined to occur in a product, and can be set by coordinates within the image, etc.

[0107] The adjusting technique can be, for example, cropping the test image of the learning data set (710) so that only the target area remains, and any other technique used to extract the target area can be applied without limitation.

[0108] The processor (150) can generate a feature model using an adjusted learning data set (720) including an external structure image / internal structure image of an adjusted normal product and an adjusted defective product test image.

[0109] The processor (150) can build an image generation model (430) by using the adjusted learning data set (720) as learning images. In this case, the learning speed can be increased by reducing the resources required for learning, such as GPU memory, by half.

[0110] Even in the process of using the image generation model (430) thereafter, the processor (150) can adjust the normal product image (730) to be included in the defect information to include the same target area as the adjusted learning data set (720).

[0111] The processor (150) can input defect information including an adjusted normal product image (730) into the image generation model (430) to obtain an adjusted defective product image (770). At this time, as described above, the processor (150) can input defect information including a first text (740) and a second text (750) into the image generation model (430) in addition to the adjusted normal product image (730) to generate an adjusted defective product image (770). At this time, the processor (150) can similarly set defect information and hyper parameters, and can generate an adjusted defective product image (770) through inpaint settings.

[0112] The processor (150) can generate a final defective product image (790) by combining the remaining area (780) excluding the target area of ​​the normal product image with the adjusted defective product image (770). At this time, the processor (150) can store the remaining area (780) when generating the adjusted normal product image (730).

[0113] FIG. 8 is a drawing illustrating a defective product image generation operation of an electronic device according to a third embodiment of the present invention.

[0114] In Fig. 8, a method is proposed for selecting meaningful images from among generated defective product images and generating labeling information for the selected images to be used in building a vision inspection model.

[0115] First, the processor (150) can create a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information about defects included in the defective product images, and normal product images.

[0116] Referring to FIG. 8, the processor (150) can adjust a normal product image (810) into a target area (811) and a remaining area (812), and obtain a defective product image (820) using the target area (811) as an input image. At this time, since the defective product image (820) is of an adjusted size, the processor (150) can generate a final defective product image (840) by combining the defective product image (820) with the remaining area (812). At this time, the processor (150) can generate the defective product image (820) through an inpaint function using a masking image (830).

[0117] Thereafter, the processor (150) can filter the acquired defective product image (840) by comparing the normal product image (810) with the acquired defective product image (840) depending on whether the degree of defect exceeds a threshold. That is, if there is no significant difference in the defective generated portion compared to the original, the defective product image can be excluded from the results.

[0118] Furthermore, generative AI-based image generation tools, including Stable Diffusion, allow for the creation of creative and diverse images in a free form, even with the same prompt information. Therefore, even after three rounds of prompt engineering, the internal structures generated from defective product images may differ from the internal structures of the actual product. Therefore, the following features can be included to preemptively filter out such patterns.

[0119] Specifically, the processor (150) can identify the type of defect from the prompt information of the generated defective product image.

[0120] The processor (150) can obtain a masking image (830) from the inpaint information of the generated defective product image, and calculate similarity by comparing the masking coordinate area of ​​the generated defective product image with the same coordinate area of ​​the original product internal structure image.

[0121] At this time, the processor (150) uses Python's OpenCV library to determine similarity based on the CORREL, CHISQR, INTERSECT, and BHATTACHARYYA values ​​as reference values, and can identify an appropriate threshold through comparison of at least 10 samples for the same Inpaint Case.

[0122] For example, in the case of a defective product image in which the internal structure is visible, if the calculated similarity is less than a threshold value, the processor (150) may determine that the internal structure is not properly implemented and filter out the image (exclude it from the results).

[0123] On the other hand, if the calculated similarity is greater than the threshold, it is determined that the internal structure has been properly implemented and the image can be adopted.

[0124] At this time, the processor (150) can separately check whether the degree of defect in the defective product image exceeds a threshold before and after checking whether the internal structure of the product included in the defective product image has been properly implemented.

[0125] Meanwhile, the processor (150) may store defect information, such as a masking image (830), as labeling information (860), and match and store the labeling information (860) and the filtered defective product image (840). The labeling information (860) may also include location and defect type information for the defective portion.

[0126] The processor (150) can generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information about defects included in the defective product images, and normal product images.

[0127] According to one embodiment of the present invention, the quality of generated defective product images can be improved to enhance the performance of a vision inspection model.

Claims

1. In electronic devices, Obtain a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image, An image generation model is built by applying a feature model generated based on the above learning data set to a base model trained to generate defective product images, Inputting defect information in the form of at least one of text and image representing the defect, which is related to the location of the defect in the product, into the image generation model, An electronic device including a processor that obtains a defective product image including a defect corresponding to the defect information through the image generation model.

2. In paragraph 1, An electronic device characterized in that the above feature model is a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

3. In paragraph 1, An electronic device wherein the defect information further includes text describing a feature to be excluded from the defective product image.

4. In paragraph 1, The above processor, An electronic device that receives defect information including a masking image showing a possible defect in a product.

5. In paragraph 1, The above processor, The above external structure image, the above internal structure image, and the above defective product test image are each adjusted to include the target area, An electronic device that generates the feature model using the adjusted images.

6. In paragraph 5, The above processor, By inputting defect information including the adjusted internal structure image into the image generation model, an adjusted defective product image is obtained, An electronic device that generates a defective product image by combining the above-described adjusted defective product image and the remaining area excluding the target area of ​​the internal structure image.

7. In paragraph 1, The above processor, An electronic device that matches the defect information with the acquired defective product image and stores it.

8. In paragraph 1, The above processor, An electronic device that compares the external structure image or the internal structure image with the acquired defective product image and filters the acquired defective product image based on whether the degree of defect exceeds a threshold.

9. A method for generating a defective product image performed by an electronic device, A step of obtaining a learning data set including an external structural image of a normal product and first prompt information corresponding to the external structural image, an internal structural image of a normal product and second prompt information corresponding to the internal structural image, and a defective product test image and third prompt information corresponding to the defective product test image; A step of building an image generation model by applying a feature model generated based on the above learning data set to a base model trained to generate a defective product image; A step of inputting defect information related to a location of a defect in a product and in the form of at least one of text and an image representing the defect into the image generation model; A method comprising the step of obtaining a defective product image including a defect corresponding to the defect information through the image generation model.

10. In paragraph 9, A method characterized in that the above feature model is a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

11. In paragraph 9, A method wherein the defect information further includes text describing a feature to be excluded from the defective product image.

12. In paragraph 9, The step of inputting the above defect information into the image generation model is: A method comprising the step of receiving defect information including a masking image indicating a possible defect in a product.

13. In paragraph 9, The steps of building the above image generation model are: A step of adjusting the external structure image, the internal structure image, and the defective product test image to each include a target area; A method comprising the step of generating the feature model using the adjusted images.

14. In paragraph 13, The step of obtaining the above defective product image is: A step of obtaining an adjusted defective product image by inputting defect information including the adjusted internal structure image into the image generation model; A method comprising the step of generating a defective product image by combining the adjusted defective product image and the remaining area excluding the target area of ​​the internal structure image.

15. In paragraph 9, After the step of obtaining the above defective product image, A method comprising a step of comparing the external structure image or the internal structure image with the acquired defective product image and filtering the acquired defective product image based on whether the degree of defect exceeds a threshold.

Images