Electronic device and method for generating defective product image and performing vision inspection labeling

Abstract

An electronic device according to one embodiment of the present invention may comprise a processor, which: inputs, into an image generation model constructed to generate a defective product image, defect information composed of text and an image indicating a defect that can occur in a product; acquires, through the image generation model, a defective product image that includes a defect corresponding to the defect information; identifies, on the basis of the defect information, labeling information that includes a defective location and a defective type of the acquired defective product image; and matches the defective product image to the labeling information and stores same.

Description

Electronic device and method for generating defective product images and labeling them for vision inspection

[0001] The present invention relates to an electronic device and method for generating an image of a defective product and labeling it for vision inspection.

[0002] With the recent advancements in related technologies like the Internet of Things (IoT) and Artificial Intelligence (AI), smart factories are becoming a hot topic. A smart factory is a manufacturing system that operates by applying Information and Communications Technology (ICT) combined with digital automation solutions throughout the production process, from design and development to manufacturing.

[0003] One technology related to smart factories is vision inspection, which detects product defects during the manufacturing process. To perform vision inspection, a vision inspection model trained to determine product defects using images of both normal and defective products must first be developed. The vision inspection model then determines whether a product is pass or fail based on images captured by cameras installed on the manufacturing line.

[0004] However, considering the ratio of good to defective products shipped from manufacturing production lines, the number of good products is overwhelmingly high. This biases the training data for vision inspection models toward good products, inevitably reducing the model's ability to detect defective products.

[0005] Currently, the model is created based on a limited number of defective cases, and the model is retrained and improved whenever an undetected defective product occurs.

[0006] This is inefficient and diminishes the value of introducing an automatic vision inspection solution, so a method is needed to secure defective product images for training vision inspection models.

[0007] Meanwhile, even if we succeed in generating creative and diverse defective product images, in order to utilize them in a vision inspection model, we need specific information about where the defect occurred and what type of defect it is in each defective product image. The process of creating this information is called labeling.

[0008] To develop models for vision inspection, developers must manually label each image. Despite the availability of many techniques to automate and streamline this process, labeling still consumes a significant portion of the resources required for model development.

[0009] An object of the present invention is to provide an electronic device and method for labeling defective product images for more efficiently learning a vision inspection model.

[0010] An object of the present invention is to provide an electronic device and method for generating defective product images and performing vision inspection using the images in order to train a higher-performance vision inspection model.

[0011] An object of the present invention is to provide an electronic device and method for generating defective product images more diversely and randomly.

[0012] An object of the present invention is to provide an electronic device and method for generating a defective product image not only for the appearance of the product but also for a failure type in which the product is damaged and its internal structure is exposed.

[0013] In an electronic device according to one embodiment of the present invention, a processor may be included that inputs defect information consisting of text and images indicating defects that may occur in a product into an image generation model constructed to generate a defective product image, obtains a defective product image including a defect corresponding to the defect information through the image generation model, identifies labeling information including a defect location and a defect type of the obtained defective product image based on the defect information, and stores the defective product image by matching it with the labeling information.

[0014] The above processor can generate the defect information using a plurality of predefined defect types and text corresponding to each of the defect types.

[0015] The above processor can quantify the characteristics of each of a plurality of defective product images generated based on the same defect information through an image analysis library, determine a normal distribution according to the quantification, and label the generated defective product image if the numerical value for the characteristic of the defective product image generated based on the same defect information as the defect information is included in the normal distribution.

[0016] The above processor can store defective product images that deviate from the normal distribution by matching them with new labeling information.

[0017] The processor may acquire a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image, and may build the image generation model by applying a feature model generated based on the learning data set to a base model trained to generate a defective product image.

[0018] The above feature model may be a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

[0019] The above defect information may further include text describing features to be excluded from the defective product image.

[0020] The above processor can receive defect information including a masking image indicating a defect that may occur in the product.

[0021] The processor can adjust the external structure image, the internal structure image, and the defective product test image to each include a target area, and generate the feature model using the adjusted images.

[0022] The processor can input defect information including the adjusted internal structure image into the image generation model to obtain an adjusted defective product image, and can generate a defective product image by combining the adjusted defective product image and the remaining area of ​​the internal structure image excluding the target area.

[0023] The above processor can filter the acquired defective product image by comparing the external structure image or internal structure image of the product with the acquired defective product image and whether the degree of defect exceeds a threshold.

[0024] A method for generating labeling information of a defective product image performed by an electronic device according to one embodiment of the present invention may include the steps of: inputting defect information consisting of text and images indicating defects that may occur in a product into an image generation model constructed to generate a defective product image; obtaining a defective product image including a defect corresponding to the defect information through the image generation model; identifying labeling information including a defect location and a defect type of the obtained defective product image based on the defect information; and storing the defective product image by matching it with the labeling information.

[0025] According to one embodiment of the present invention, by automatically performing labeling of defective product images, resources for building a vision inspection model can be minimized.

[0026] According to one embodiment of the present invention, not only can the quality of generated defective product images be improved, but also the consistency of labeling information of defective product images can be improved, thereby improving the performance of a vision inspection model.

[0027] According to one embodiment of the present invention, the biased performance of existing vision inspection models can be dramatically improved by generating a variety of defective product images. Furthermore, by utilizing a generative model, defective product images can be extracted at a very high productivity rate, with quality nearly identical to actual images. Ultimately, this can shorten the development time of vision inspection models and enhance their performance.

[0028] FIG. 1 is a schematic diagram illustrating an electronic device according to one embodiment of the present invention.

[0029] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0030] FIG. 3 is a diagram illustrating an operation flow diagram of an electronic device according to one embodiment of the present invention.

[0031] FIG. 4 is a drawing illustrating a construction of a feature model according to one embodiment of the present invention.

[0032] FIG. 5 is a drawing showing an external structural image and an internal structural image of a normal product according to one embodiment of the present invention.

[0033] FIG. 6 is a drawing illustrating a defective product image generation operation of an electronic device according to a first embodiment of the present invention.

[0034] FIG. 7 is a drawing illustrating a defective product image generation operation of an electronic device according to a second embodiment of the present invention.

[0035] FIG. 8 is a drawing illustrating a defective product image generation operation of an electronic device according to a third embodiment of the present invention.

[0036] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. In the drawings, portions irrelevant to the description may be omitted for clarity in describing the present invention, and the same reference numerals may be used throughout the specification for identical or similar components.

[0037] FIG. 1 is a schematic diagram illustrating an electronic device according to one embodiment of the present invention.

[0038] An electronic device (100) according to one embodiment of the present invention is a device that generates an image of a defective product and labels it for vision inspection, and can be implemented as a computer, server, laptop, smart phone, tablet PC, smart pad, etc.

[0039] An electronic device (100) can build an image generation model (20) using defect information (10) used to generate a defective product image, and obtain a defective product image (31) from the built image generation model (20).

[0040] At this time, the electronic device (100) can match labeling information corresponding to each generated defective product image (31). Labeling information means specific information about where the defect occurred and what type of defect it is in each generated defective product image (31), and the included content and format are not limited to any one.

[0041] Through this, the electronic device (100) can sufficiently obtain defective product images (31) and normal product images (40) and build a vision inspection model (50) with balanced proportions.

[0042] Subsequently, the electronic device (100) can input the product's appearance image (60) obtained from the process line into a vision inspection model (50) to inspect good or defective products and obtain inspection information (70).

[0043] As illustrated in FIG. 1, the electronic device (100) of the present invention can perform all of the following: constructing an image generation model (20) for generating a defective product image (31), generating a defective product image (31) using the image generation model (20), matching labeling information to the defective product image (31), constructing a vision inspection model (50) using the defective product image (31) and a normal product image (40), and performing product inspection using the vision inspection model (50). However, a plurality of electronic devices (100) may be separately provided to individually or selectively perform the operations listed above, and are not limited to any one of them.

[0044] As previously described, the current vision inspection model suffers from performance issues due to the imbalance in training data between normal and defective product images. Furthermore, the labeling process for defective product images to be used in the vision inspection model is cumbersome, requiring manual labeling.

[0045] In the present invention, a method for generating various defective product images is proposed to improve the biased performance of a vision inspection model, and at this time, a defective product image including a defect occurring in the product's external appearance as well as a defective product image including a defect occurring in the product's internal structure is proposed to be implemented.

[0046] Furthermore, the present invention proposes a method for automatically labeling generated defective product images so that they can be easily used as training data for a vision inspection model. Subsequently, a method for building and utilizing a vision inspection model using defective product images is proposed.

[0047] Through this, the present invention can expand the defect range of a defective product image and further increase the applicability of a vision inspection model using the same.

[0048] Hereinafter, the configuration and operation of an electronic device (100) according to one embodiment of the present invention will be specifically described with reference to the drawings.

[0049] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0050] An electronic device (100) according to one embodiment of the present invention may include an input unit (110), a communication unit (120), a display unit (130), a storage unit (140), and a processor (150).

[0051] The input unit (110) generates input data in response to user input of the electronic device (100). For example, the user input may be a user input that initiates the operation of the electronic device (100), a user input required to build an image generation model and a vision inspection model, a user input that inputs defect information, a user input that sets a target area, etc. In addition, the user input may be applied without limitation if it is required to generate a defective product image, match label information, or perform vision inspection.

[0052] The input unit (110) includes at least one input means. The input unit (110) may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a mouse, a menu button, etc.

[0053] The communication unit (120) can perform communication with an external device such as a server to transmit and receive defect information, an external structure image of a normal product, an internal structure image of a normal product, a test image of a defective product, prompt information corresponding to each image, labeling information, inspection information, etc.

[0054] To this end, the communication unit (120) can perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Wi-Fi (wireless fidelity), Bluetooth, or wired communication such as LAN (local area network), WAN (Wide Area Network), and power line communication.

[0055] The display unit (130) displays display data according to the operation of the electronic device (100). The display unit (130) may display a screen that displays the process of generating a defective product image, a screen that displays the generated defective product image, a screen that matches a label to the generated defective product image, a screen that performs a vision inspection, a screen that receives user input, etc.

[0056] The display unit (130) includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (130) may be implemented as a touch screen by being combined with the input unit (110).

[0057] The storage unit (140) stores the operation programs of the electronic device (100). The storage unit (140) includes non-volatile storage that can preserve data (information) regardless of whether power is supplied, and volatile memory that cannot preserve data if power is not supplied and into which data to be processed by the processor (150) is loaded. The storage includes flash memory, hard-disc drive (HDD), solid-state drive (SSD), read-only memory (ROM), etc., and the memory includes buffer, random access memory (RAM), etc.

[0058] The storage unit (140) can store defect information (10), an image generation model (20), a defective product image (31), labeling information (32), a normal product image (40), a vision inspection model (50), a product image (60), inspection information (70), etc. The storage unit (140) can store operation programs, etc. required in the process of constructing an image generation model (20), creating a defective product image (31) using the image generation model (20), matching labeling information (32) to the defective product image (31), constructing a vision inspection model (50), and inspecting a product using the vision inspection model (50).

[0059] The processor (150) can control at least one other component (e.g., hardware or software component) of the electronic device (100) by executing software such as a program, and can perform various data processing or operations.

[0060] A processor (150) according to one embodiment of the present invention inputs defect information consisting of text and images indicating defects that may occur in a product into an image generation model constructed to generate a defective product image, obtains a defective product image including a defect corresponding to the defect information through the image generation model, identifies labeling information including a defect location and a defect type of the obtained defective product image based on the defect information, and stores the defective product image by matching it with the labeling information.

[0061] A processor (150) according to one embodiment of the present invention can generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information regarding defects included in the defective product images, and normal product images.

[0062] At this time, the processor (150) may construct at least one of an image generation model and a vision inspection model, or may receive and store a previously constructed image generation model and a vision inspection model from the outside and use them, but is not limited to either one.

[0063] Meanwhile, the processor (150) may perform at least a portion of the data analysis, processing, and result information generation for performing the above operations using at least one of a machine learning, neural network, or deep learning algorithm as a rule-based or artificial intelligence (AI) algorithm. Examples of the neural network may include models such as a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), and an RNN (Recurrent Neural Network).

[0064] FIG. 3 is a diagram illustrating an operation flow diagram of an electronic device according to one embodiment of the present invention.

[0065] A processor (150) according to one embodiment of the present invention can input defect information consisting of text and images indicating defects that may occur in a product into an image generation model constructed to generate a defective product image (S10).

[0066] Defective product images refer to product images created to include external defects such as scratches, breaks, and dents on the product.

[0067] The image generation model of the present invention is a generative artificial intelligence model. It can be constructed by applying a feature model implemented to fine-tune the features of a base model trained to generate defective product images to the base model. The process of constructing the image generation model is described with reference to FIG. 4.

[0068] Defect information is the information required to create a defective product image. Defect information may include text describing potential defects in a product, particularly text related to the location of the defect and text describing the defect itself (hereinafter referred to as "first text"), text describing features to be excluded from the defective product image (hereinafter referred to as "second text"), a normal product image, a masking image displaying potential defects in the product, and more.

[0069] For example, text related to the location of a defect may be, for example, "Plastic Texture," "Inner Structure," "Under Cover," "Cover," etc. Text describing the defect itself may be, for example, "stained," "crack," "failure," "dent," "leakage," "damaged," "broken," "open," etc. In this case, the text related to the location of a product defect and the text indicating the defect itself may be combined and entered, for example, "Open Inner Structure." In addition, various other applications may be applied, and the content or expression format included in the first text is not limited to any one.

[0070] The second text is text that indicates features to be excluded from the defective product image, and may include, for example, “clean”, “clear”, “pass”, “normal”, etc.

[0071] At this time, if you want to create a defective product image where only the external structure is defective and the internal structure is free of defects, you can input "Inner structure", "Under cover", etc. as the second text. A more specific example is described with reference to Fig. 6.

[0072] The first text and the second text can be input in the form of a prompt, and the processor (150) can apply the defect information with an appropriate weight.

[0073] Meanwhile, the processor (150) can generate defect information using a plurality of predefined defect types and text corresponding to each defect type. This can be achieved by pre-defining the defect types and corresponding texts through Prompt Engineering development, thereby establishing the relationship between the defect types and text. This prevents the problem of unfamiliar images intermingling with the overall direction of defective product image generation, which is characteristic of generative AI technology.

[0074] For example, the types of defects may include surface scratches, surface marks, surface contamination, cover damage (medium), cover damage (small), foreign substances, etc., and the types or number of defects are not limited to any one.

[0075] The text corresponding to a surface scratch might be "(abnormal surface, Damaged):1.0, Scratched:1.2, stained:0.1, broken:0.8". The text corresponding to a surface dent might be "(abnormal surface, Damaged):1.0, Scratched:0.3, punched:1.5, stained:0.1, broken:0.8".

[0076] The text corresponding to surface contamination might be "stained:1.1, dirty:1.0". The text corresponding to cover damage (medium) might be "(abnormal surface, Damaged):1.3, Scratched:1.5, punched:0.6, broken:1.3, cutting off cover:1.5". The text corresponding to cover damage (small) might be "(abnormal surface, Damaged):1, Scratched:1.0, punched:0.3, broken:1.3, cutting off cover:1.0". The text corresponding to foreign matter might be "(abnormal surface):1.0, Scratched:0.1, punched:0.1, foreign matter contamination:1.0".

[0077] Here, the number following each text represents the weighting factor applied when inputting defect information into the image generation model, and is typically expressed as a decimal number between 0 and 2. The actual application of these weighting factors varies depending on the conditions under which the base model and feature model are trained, so it's important to develop appropriate numbers and the desired generation level in advance.

[0078] Text-based defect information can include additional product information. For example, if the target product is an MCCB, defect information can include terms such as "circuit braker" or "MCCB."

[0079] A normal product image can serve as base information for the image generation model to reference when generating a defective product image. The processor (150) can input a normal product image—i.e., an external structural image or an internal structural image—into the image generation model and acquire a defective product image based on the input (IMG2IMG function). In this case, the normal product image can utilize one or more of the normal product test images used to form the feature model.

[0080] The processor (150) can receive defect information including a masking image indicating a defect that may occur in a product. The processor (150) can obtain a defective product image by using an inpaint function that masks a normal product image so that a defect is generated only in a specific area. In particular, a masking image is an image that indicates a defect in a specific area, and the processor (150) can use the inpaint function to change only a specific area of ​​a normal product image into a masking image. When the inpaint function is utilized, it is more efficient when generating an image to intensively improve types that were not detected by an existing image model or were over-detected as defects when they were not.

[0081] A processor (150) according to one embodiment of the present invention can obtain a defective product image including a defect corresponding to defect information through an image generation model (S20).

[0082] Unlike conventional generative models, generative models based on denoising techniques are advantageous for generating defects from normal product images, demonstrating superior image quality and image generation efficiency. Therefore, while the present invention is described based on examples generated using a generative model based on denoising techniques, the same invention structure can be utilized even if more advanced solutions emerge in the future.

[0083] A processor (150) according to one embodiment of the present invention can identify labeling information including a defective location and defective type of a defective product image acquired based on defect information (S30).

[0084] As described above, defect information may include text (prompt) indicating a defect type, a normal product image, a masking image, etc. The processor (150) can identify the defect type of the defective product image generated from the text (prompt) among the defect information. The processor (150) can identify the defect location of the defective product image from the masking image among the defect information. At this time, the defect location can be obtained as coordinate information.

[0085] At this time, multiple defects may exist within a single defective product image generated based on defect information. The multiple defects may include multiple occurrences of the same defect type or at least one occurrence of multiple defect types. The processor (150) may identify multiple labeling information pieces including multiple defect locations and at least one defect type based on the defect information. When multiple labeling information pieces are identified, the processor (150) may assign weights to each defect type based on the size of the defect, the importance of the defect type, etc.

[0086] In addition, the processor (150) can identify the file name of the labeling information based on the hyper parameters required for generating a defective product image, or the storage path and file name of the defective product image to be generated.

[0087] At this time, various methods for obtaining labeling information can be implemented. For example, as described above, the processor (150) can directly obtain information regarding the type and location of defects included in the labeling information from masking images or prompt information.

[0088] As another example, the processor (150) can obtain labeling information by defining the file name of the defective product image to contain information about the type and location of the defect (e.g., location_type.jpg) and then reading the file name.

[0089] A processor (150) according to one embodiment of the present invention can store a defective product image by matching it with labeling information (S40).

[0090] For example, the processor (150) can insert labeling information into the metadata of a defective product image and match it. In addition, various matching methods for defective product images and labeling information can be employed, and are not limited to any one.

[0091] According to one embodiment of the present invention, by automatically performing labeling of defective product images (training data), resources for building a vision inspection model can be minimized.

[0092] According to one embodiment of the present invention, the biased performance of existing vision inspection models can be dramatically improved by generating a variety of defective product images. Furthermore, by utilizing a generative model, defective product images can be extracted at a very high productivity rate, with quality nearly identical to actual images. Ultimately, this can shorten the development time of vision inspection models and enhance their performance.

[0093] FIG. 4 is a drawing illustrating a construction of a feature model according to one embodiment of the present invention.

[0094] FIG. 4 is a diagram illustrating the construction of an image generation model according to one embodiment of the present invention. FIG. 4 illustrates the construction of an image generation model, as described in relation to S10 of FIG. 3.

[0095] The processor (150) can obtain a learning data set (421) including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image.

[0096] An external structural image of a normal product refers to an image of the product's exterior, while an internal structural image refers to an image that allows for the internal structure of the product to be seen. For example, an internal structural image could be an image of a normal product with its cover removed. Furthermore, an internal structural image can be an image taken during the manufacturing process, or it can be sufficient if it allows for at least a portion of the product's interior to be seen.

[0097] Prompt information represents each image and can be constructed by matching each image. Each image and prompt information can be acquired externally or generated through user input that inputs prompt information corresponding to each image. The acquisition path or method is not limited to any one.

[0098] For example, in the case of an image of the external structure of a normal product, it can be matched with first prompt information such as "Circuit breaker" (product name), "Plastic texture," "Outer structure," etc. In addition, in the case of an image of the internal structure of a normal product, it can be matched with second prompt information such as "Circuit breaker" (product name), "Plastic texture," "Inner structure," etc. In the case of a test image of a defective product, it can be matched with third prompt information such as "Circuit breaker" (product name), "Plastic texture," "stained," "crack," etc.

[0099] A processor (150) according to one embodiment of the present invention can build an image generation model (430) by applying a feature model (420) generated based on a learning data set (421) to a previously built base model (410).

[0100] The base model (410) can be selected as a model that well reflects the characteristics of the target product. Since this model varies depending on the field and product conditions, it is recommended that a field expert conduct testing to select the appropriate model. For example, the base model (410) may be Stable Diffusion, a denoising technique-based model. Among the Stable Diffusion models, the base model may include the v1.5 pruned-emaonly.safetensor model.

[0101] However, since generative artificial intelligence models are mainly trained based on people, fine-tuning is required so that the output image corresponds to the product. To this end, an image generation model (430) can be built by adding a feature model (420) to a base model (410). The feature model (420) can be, for example, a Low-Rank Adaptation (LoRA) model. Even if the number of defective product test images is not large when generating a LoRA model, it is not a major problem because the base model (410) has a lot of defect information about the defect type.

[0102] There are several methods for generating an image generation model (430), including a method utilizing the Dreambooth Extension module of stable diffusion and a method utilizing the Kohya-ss module.

[0103] At this time, when constructing an image generation model (430), the processor (150) can set defect information and hyperparameters. Hyperparameters are variables that directly affect the image generation quality, and a process of finding optimal values ​​through repeated execution is required.

[0104] The feature model (420) according to one embodiment of the present invention can learn about not only the external structure of a normal product but also its internal structure using the previously constructed learning data set (421). Therefore, the image generation model (430) constructed using this feature model (420) can easily generate images of defective products even when the internal structure of the product is visible during the defect generation process.

[0105] FIG. 5 is a drawing showing an external structural image and an internal structural image of a normal product according to one embodiment of the present invention.

[0106] The left image of Fig. 5 is an external structural image (510) of a normal product, and the right image is an internal structural image (520) of a normal product with the cover removed. If a request is made to generate a defective product image with a visible defect in the internal structure, a feature model trained solely on the external structural image (510) will have difficulty generating a suitable defective product image because it cannot properly represent the internal structure of the product.

[0107] On the other hand, in the present invention, such internal structure images (520) are also learned, so that various types of defective product images including the internal structure can be generated.

[0108] FIG. 6 is a drawing illustrating a defective product image generation operation of an electronic device according to a first embodiment of the present invention.

[0109] FIG. 6 illustrates a process for generating a defective product image, as described in relation to S10 and S20 of FIG. 3. FIG. 6 also illustrates a process for identifying labeling information, as described in relation to S30 of FIG. 3.

[0110] Referring to FIG. 6, the processor (150) can input defect information including a normal product image (611), a masking image (612), a first text (620), and a second text (630) into an image generation model (430) that applies a feature model (420) to a base model (410) to generate a defective product image (640). In addition, the processor (150) can identify defect location information from the masking image (612) and identify defect type information from the first text (620) and the second text (630). Accordingly, the processor (150) can obtain labeling information (650) including defect location information and defect type information corresponding to the generated defective product image (640).

[0111] Meanwhile, the present invention enables the generation of various types of defective product images by adding a set of internal structure images. For example, defective product images can be generated for types where defects occur only in the external structure without revealing the internal structure, types where defects occur only in the external structure with the internal structure revealed, and types where defects occur in both the internal and external structures with the internal structure revealed.

[0112] In order to generate a defective product image corresponding to each type, it is necessary to appropriately select a first text (620) indicating a defect and a second text (630) describing a feature to be excluded from the defective product image and input them into the image generation model (430).

[0113] For example, to create a defective product image of a type in which only the outer structure is defective and the inner structure is not exposed, a first text (620) including "Circuit breaker, Plastic Texture, Broken, Damaged" and a second text (630) including "Open Inner Structure, heavy damaged cover" can be input.

[0114] In order to create a defective product image in which only the outer structure is defective while the inner structure is exposed, a first text (620) including "Circuit breaker, Plastic Texture, Broken, Damaged, heavy damaged cover, Open Inner Structure, Clean Inner Structure" and a second text (630) including "Damaged Inner Structure" may be input. In this case, when the first text (620) and the second text (630) that cause only the outer structure to be defective while the inner structure is exposed are input, the defective product image (640) illustrated in FIG. 6 may include a defect in which a portion of the surface paint of the product is peeled off and the inner structure is visible.

[0115] In order to create a defective product image of a type in which defects have occurred in the internal structure and the external structure with the internal structure exposed, a first text (620) including "Circuit breaker, Plastic Texture, Broken, Damaged, heavy damaged cover, Open Inner Structure, Damaged Inner Structure" and a second text (630) including "Clean Inner Structure" can be input.

[0116] The processor (150) can likewise set defect information and hyper parameters, and generate a defective product image (640) through inpaint settings.

[0117] Meanwhile, the processor (150) may input a normal product image (611) corresponding to the defective product image to be generated. For example, if the defective product image to be generated includes a defect occurring in the external structure, the processor (150) may input defect information including the external structure image into the image generation model. Conversely, if the defective product image to be generated includes a defect while showing the internal structure, the processor (150) may input defect information including the internal structure image into the image generation model.

[0118] The processor (150) can input a masking image (612) showing defects that may occur in the product along with a normal product image (611).

[0119] According to one embodiment of the present invention, the scope of defective product images can be expanded by generating defective product images that are related to the internal structure of the product and include defects.

[0120] According to one embodiment of the present invention, defect inspection can be applied even when the inside of a product is visible while the outside of the product is damaged, thereby further improving the performance of a vision inspection model.

[0121] FIG. 7 is a drawing illustrating a defective product image generation operation of an electronic device according to a second embodiment of the present invention.

[0122] The defective product image generation method described above with reference to Fig. 6 generates a defective product image by inputting the entire normal product image (611). However, due to the high resolution of the vision inspection image, an issue may arise where the image generation model cannot be trained and the image cannot be generated due to insufficient computing resources. Fig. 7 proposes a method for conserving computing resources by adjusting the normal product image.

[0123] The processor (150) can adjust the learning data set (710) to include a target area. The target area refers to an area where defects are primarily determined to occur in a product, and can be set as coordinates within an image, etc.

[0124] The adjusting technique can be, for example, cropping the test image of the learning data set (710) so that only the target area remains, and any other technique used to extract the target area can be applied without limitation.

[0125] The processor (150) can generate a feature model using an adjusted learning data set (720) including an external structure image / internal structure image of an adjusted normal product and an adjusted defective product test image.

[0126] The processor (150) can build an image generation model (430) by using the adjusted learning data set (720) as learning images. In this case, the learning speed can be increased by reducing the resources required for learning, such as GPU memory, by half.

[0127] Even in the process of using the image generation model (430) thereafter, the processor (150) can adjust the normal product image (730) to be included in the defect information to include the same target area as the adjusted learning data set (720).

[0128] The processor (150) can input defect information including an adjusted normal product image (730) into the image generation model (430) to obtain an adjusted defective product image (770). At this time, as described above, the processor (150) can input defect information including a first text (740) and a second text (750) into the image generation model (430) in addition to the adjusted normal product image (730) to generate an adjusted defective product image (770). At this time, the processor (150) can similarly set defect information and hyper parameters, and can generate an adjusted defective product image (770) through inpaint settings.

[0129] The processor (150) can generate a final defective product image (790) by combining the remaining area (780) excluding the target area of ​​the normal product image with the adjusted defective product image (770). At this time, the processor (150) can store the remaining area (780) when generating the adjusted normal product image (730).

[0130] FIG. 8 is a drawing illustrating a defective product image generation operation of an electronic device according to a third embodiment of the present invention.

[0131] In Fig. 8, a method is proposed for selecting meaningful images from among generated defective product images and generating labeling information for the selected images to be used in building a vision inspection model.

[0132] First, the processor (150) can create a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information about defects included in the defective product images, and normal product images.

[0133] Referring to FIG. 8, the processor (150) can adjust a normal product image (810) into a target area (811) and a remaining area (812), and obtain a defective product image (820) using the target area (811) as an input image. At this time, since the defective product image (820) is of an adjusted size, the processor (150) can generate a final defective product image (840) by combining the defective product image (820) with the remaining area (812). At this time, the processor (150) can generate the defective product image (820) through an inpaint function using a masking image (830).

[0134] Thereafter, the processor (150) can filter the acquired defective product image (840) by comparing the normal product image (810) with the acquired defective product image (840) depending on whether the degree of defect exceeds a threshold. That is, if there is no significant difference in the defective generated portion compared to the original, the defective product image can be excluded from the results.

[0135] At this time, the processor (150) may store defect information, such as a masking image (830), as labeling information (860), and match and store the labeling information (860) and the filtered defective product image (840). The labeling information (860) may include defect location information and defect type information.

[0136] Meanwhile, as previously described, when generating images of defective products based on text developed for predefined defect types, labeling can be done using the predefined text. However, even when images are generated using predefined text, the nature of generative AI technology can result in images being randomly generated in various shapes and degrees of detail, potentially interfering with the overall generation direction and unfamiliar images. Therefore, a method to improve the overall labeling consistency is needed.

[0137] The processor (150) can digitize a defective product image generated based on defect information through an image analysis library.

[0138] The image analysis library used to quantify the image characteristics of defective product images may be the HOG (Histogram of Oriented Gradient) library in Python OpenCV. However, this library is not limited to this library, and other image analysis libraries such as SIFT (Scale Invariant Feature Transform), Haar feature, and Ferns can also be utilized.

[0139] The processor (150) labels defective images within a normal distribution using the corresponding defect information based on numerical information of a plurality of defective product images generated based on the same defect information, and stores defective product images that deviate from the normal distribution (have outliers) by matching them with new labeling information.

[0140] That is, the processor (150) can numerically analyze multiple defective product images generated based on the same defect information and, if they deviate from this, label a new label as another type.

[0141] The processor (150) can generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information about defects included in the defective product images, and normal product images.

[0142] According to one embodiment of the present invention, not only can the quality of generated defective product images be improved, but also the consistency of labeling information of defective product images can be improved, thereby improving the performance of a vision inspection model.

Claims

1. In electronic devices, Input the defect information consisting of text and images representing possible defects in the product into the image generation model built to generate defective product images, Obtaining a defective product image including a defect corresponding to the defect information through the image generation model, Identifying labeling information including the defect location and defect type of the acquired defective product image based on the defect information; An electronic device including a processor that matches the defective product image with the labeling information and stores the same.

2. In paragraph 1, The above processor, An electronic device that generates defect information using a plurality of predefined defect types and text corresponding to each of the defect types.

3. In paragraph 2, The above processor, The characteristics of each of the multiple defective product images generated based on the same defect information are quantified through an image analysis library, and a normal distribution according to the quantification is determined. An electronic device that labels a generated defective product image when a numerical value of a characteristic of the generated defective product image based on the same defect information as the above defect information is included in the normal distribution.

4. In paragraph 3, The above processor, An electronic device that stores defective product images that deviate from a normal distribution by matching them with new labeling information.

5. In paragraph 1, The above processor, Obtain a learning data set including an external structure image of a normal product and first prompt information corresponding to the external structure image, an internal structure image of a normal product and second prompt information corresponding to the internal structure image, and a defective product test image and third prompt information corresponding to the defective product test image, An electronic device that builds an image generation model by applying a feature model generated based on the learning data set to a base model trained to generate a defective product image.

6. In paragraph 5, An electronic device characterized in that the above feature model is a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

7. In paragraph 1, An electronic device wherein the defect information further includes text describing a feature to be excluded from the defective product image.

8. In paragraph 1, The above processor, An electronic device that receives defect information including a masking image showing a possible defect in a product.

9. In paragraph 5, The above processor, The above external structure image, the above internal structure image, and the above defective product test image are each adjusted to include the target area, An electronic device that generates the feature model using the adjusted images.

10. In paragraph 9, The above processor, By inputting defect information including the adjusted internal structure image into the image generation model, an adjusted defective product image is obtained, An electronic device that generates a defective product image by combining the above-described adjusted defective product image and the remaining area excluding the target area of ​​the internal structure image.

11. In paragraph 1, The above processor, An electronic device that compares the external or internal structural image of the product with the acquired defective product image and filters the acquired defective product image based on whether the degree of defect exceeds a threshold.

12. A method for generating labeling information of a defective product image performed by an electronic device, A step of inputting defect information consisting of text and images representing possible defects in a product into an image generation model built to generate defective product images; A step of obtaining a defective product image including a defect corresponding to the defect information through the image generation model; A step of identifying labeling information including a defect location and defect type of the acquired defective product image based on the defect information; A method comprising the step of storing the defective product image by matching it with the labeling information.

Images