Electronic device for generating images of product having defects and performing vision inspection by using same

Abstract

An electronic device according to an embodiment of the present invention may comprise a processor that constructs an image generation model by applying, to a base model trained to generate a defective product image, a feature model generated on the basis of a normal product inspection image and a defective product inspection image, inputs defect information in the form of at least one of text and an image to the image generation model, and obtains a defective product image including a defect corresponding to the defect information through the image generation model.

Description

An electronic device that generates images of defective products and performs vision inspection using them.

[0001] The present invention relates to an electronic device that generates an image of a defective product and performs vision inspection using the image.

[0002] With the recent advancements in related technologies like the Internet of Things (IoT) and Artificial Intelligence (AI), smart factories are becoming a hot topic. A smart factory is a manufacturing system that operates by applying Information and Communications Technology (ICT) combined with digital automation solutions throughout the production process, from design and development to manufacturing.

[0003] One technology related to smart factories is vision inspection, which detects product defects during the manufacturing process. To perform vision inspection, a vision inspection model trained to determine product defects using images of both normal and defective products must first be developed. The vision inspection model then determines whether a product is pass or fail based on images captured by cameras installed on the manufacturing line.

[0004] However, considering the ratio of good to defective products shipped from manufacturing production lines, the number of good products is overwhelmingly high. This biases the training data for vision inspection models toward good products, inevitably reducing the model's ability to detect defective products.

[0005] Currently, the model is created based on a limited number of defective cases, and the model is retrained and improved whenever an undetected defective product occurs.

[0006] This is inefficient and diminishes the value of introducing an automatic vision inspection solution, so there is a need to secure images of defective products for training vision inspection models.

[0007] An object of the present invention is to provide an electronic device that generates a defective product image and performs vision inspection using the image to train a higher-performance vision inspection model.

[0008] An object of the present invention is to provide an electronic device that generates defective product images more diversely and randomly.

[0009] In an electronic device according to one embodiment of the present invention, an image generation model may be constructed by applying a feature model generated based on a normal product test image and a defective product test image to a base model learned to generate a defective product image, and a processor may be included that inputs defect information in the form of at least one of text and image to the image generation model, and obtains a defective product image including a defect corresponding to the defect information through the image generation model.

[0010] The above feature model may be a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

[0011] The processor may receive defect information including a first text describing a possible defect in the product.

[0012] The processor may further receive defect information including second text describing features to be excluded from the defective product image.

[0013] The above processor can receive defect information including a masking image indicating a defect that may occur in the product.

[0014] The processor can adjust the normal product test image and the defective product test image to include a target area, respectively, and generate the feature model using the adjusted normal product test image and the adjusted defective product test image.

[0015] The above processor can input defect information including the adjusted normal product test image into the image generation model to obtain an adjusted defective product image, and generate a defective product image by combining the remaining areas excluding the target area of ​​the normal product test image.

[0016] The above processor can store the defect information by matching it with the acquired defective product image.

[0017] The above processor can compare the normal product test image and the acquired defective product image and filter the acquired defective product image based on whether the degree of defect exceeds a threshold.

[0018] In an electronic device according to one embodiment of the present invention, a processor is included for generating a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information regarding defects included in the defective product images, and normal product images, wherein the image generation model can be constructed by applying a feature model generated based on a normal product test image and a defective product test image to a base model trained to generate a defective product image.

[0019] The above defective product images may include defects corresponding to defect information input to the image generation model, wherein the defect information is in the form of at least one of text and image.

[0020] The above text may include a first text describing a defect that may occur in the product.

[0021] The above text may further include a second text describing features to be excluded from the defective product image.

[0022] The above image may include a masking image showing possible defects that may occur in the product.

[0023] The above feature model can be generated using the normal product test image and the defective product test image adjusted to include the target area.

[0024] Each of the above defective product images can be obtained by combining a defective product image generated by inputting defect information including the adjusted normal product test image into the image generation model and a remaining area excluding the target area of ​​the normal product test image.

[0025] The above processor can acquire an external image of the product and input the external image into the vision inspection model to inspect the product for defects.

[0026] A method for generating a defective product image performed by an electronic device according to one embodiment of the present invention may include: a step of constructing an image generation model by applying a feature model generated based on a normal product test image and a defective product test image to a base model learned to generate a defective product image; a step of inputting defect information in the form of at least one of text and an image to the image generation model; and a step of obtaining a defective product image including a defect corresponding to the defect information through the image generation model.

[0027] According to one embodiment of the present invention, the biased performance of an existing vision inspection model can be dramatically improved by generating various defective product images.

[0028] According to one embodiment of the present invention, by utilizing a generative model, the quality of a generated defective product image can be extracted at a very high productivity speed with a quality almost identical to that of an actual image.

[0029] According to one embodiment of the present invention, high quality random and diverse defective product images can be generated.

[0030] FIG. 1 is a schematic diagram illustrating an electronic device according to one embodiment of the present invention.

[0031] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0032] FIG. 3 is a diagram illustrating an operation flow chart of an electronic device according to one embodiment of the present invention.

[0033] FIG. 4 is a drawing illustrating the construction of an image generation model according to one embodiment of the present invention.

[0034] FIG. 5 is a drawing illustrating a defective product image generation operation of an electronic device according to a first embodiment of the present invention.

[0035] FIG. 6 is a drawing illustrating a defective product image generation operation of an electronic device according to a second embodiment of the present invention.

[0036] FIG. 7 is a drawing illustrating a defective product image generation operation of an electronic device according to a third embodiment of the present invention.

[0037] FIG. 8 is a drawing illustrating examples of defective product images according to one embodiment of the present invention.

[0038] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. In the drawings, portions irrelevant to the description may be omitted for clarity in describing the present invention, and the same reference numerals may be used throughout the specification for identical or similar components.

[0039] FIG. 1 is a schematic diagram illustrating an electronic device according to one embodiment of the present invention.

[0040] An electronic device (100) according to one embodiment of the present invention is a device that generates a defective product image and performs vision inspection using the same, and can be implemented as a computer, server, laptop, smart phone, tablet PC, smart pad, etc.

[0041] An electronic device (100) can construct an image generation model (20) using defect information (10) utilized to generate a defective product image, and obtain a defective product image (30) from the constructed image generation model (20). Through this, a sufficient number of defective product images (30) and normal product images (40) can be obtained, and a vision inspection model (50) can be constructed with balanced proportions.

[0042] Subsequently, the electronic device (100) can input an external image (60) of a product obtained from a process line into a vision inspection model (50) to inspect good or defective products and obtain inspection information (70). The vision inspection model (50) will be described with reference to FIG. 7.

[0043] As illustrated in FIG. 1, the electronic device (100) of the present invention can perform all of the following: constructing an image generation model (20) for generating a defective product image (30), generating a defective product image (30) using the image generation model (20), constructing a vision inspection model (50) using the defective product image (30) and a normal product image (40), and performing product inspection using the vision inspection model (50). However, a plurality of electronic devices (100) may be separately provided to individually or selectively perform the operations listed above, and are not limited to any one of them.

[0044] In the present invention, a method for generating defective product images to improve the biased performance of a vision inspection model is proposed, and subsequently, a method for constructing and utilizing a vision inspection model by utilizing the defective product images is proposed.

[0045] Hereinafter, the configuration and operation of an electronic device (100) according to one embodiment of the present invention will be specifically described with reference to the drawings.

[0046] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present invention.

[0047] An electronic device (100) according to one embodiment of the present invention may include an input unit (110), a communication unit (120), a display unit (130), a storage unit (140), and a processor (150).

[0048] The input unit (110) generates input data in response to user input of the electronic device (100). For example, the user input may be a user input that initiates the operation of the electronic device (100), a user input required to build an image generation model and a vision inspection model, a user input that inputs defect information, a user input that sets a target area, etc. In addition, if it is a user input required to generate a defective product image and perform vision inspection, it can be applied without limitation.

[0049] The input unit (110) includes at least one input means. The input unit (110) may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a mouse, a menu button, etc.

[0050] The communication unit (120) can perform communication with an external device such as a server to transmit and receive defect information, normal product test images, defective product test images, appearance images, inspection information, etc.

[0051] To this end, the communication unit (120) can perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Wi-Fi (wireless fidelity), Bluetooth, or wired communication such as LAN (local area network), WAN (Wide Area Network), and power line communication.

[0052] The display unit (130) displays display data according to the operation of the electronic device (100). The display unit (130) can display a screen for generating a defective product image, a screen for displaying the generated defective product image, a screen for performing a vision inspection, a screen for receiving user input, etc.

[0053] The display unit (130) includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, and an electronic paper display. The display unit (130) may be implemented as a touch screen by being combined with the input unit (110).

[0054] The storage unit (140) stores the operation programs of the electronic device (100). The storage unit (140) includes non-volatile storage that can preserve data (information) regardless of whether power is supplied, and volatile memory that cannot preserve data if power is not supplied and into which data to be processed by the processor (150) is loaded. The storage includes flash memory, hard-disc drive (HDD), solid-state drive (SSD), read-only memory (ROM), etc., and the memory includes buffer, random access memory (RAM), etc.

[0055] The storage unit (140) can store defect information (10), an image generation model (20), a defective product image (30), a normal product image (40), a vision inspection model (50), an appearance image (60), inspection information (70), etc. The storage unit (140) can store operation programs, etc. required in the process of constructing an image generation model (20), creating a defective product image (30) using the image generation model (20), constructing a vision inspection model (50), and inspecting a product using the vision inspection model (50).

[0056] The processor (150) can control at least one other component (e.g., hardware or software component) of the electronic device (100) by executing software such as a program, and can perform various data processing or operations.

[0057] A processor (150) according to one embodiment of the present invention builds an image generation model by applying a feature model generated based on a normal product test image and a defective product test image to a base model learned to generate a defective product image, inputs defect information in the form of at least one of text and image to the image generation model, and obtains a defective product image including a defect corresponding to the defect information through the image generation model.

[0058] A processor (150) according to one embodiment of the present invention can generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information regarding defects included in the defective product images, and normal product images.

[0059] At this time, the processor (150) may construct at least one of an image generation model and a vision inspection model, or may receive and store a previously constructed image generation model and a vision inspection model from the outside and use them, but is not limited to either one.

[0060] Meanwhile, the processor (150) may perform at least a portion of the data analysis, processing, and result information generation for performing the above operations using at least one of a machine learning, neural network, or deep learning algorithm as a rule-based or artificial intelligence (AI) algorithm. Examples of the neural network may include models such as a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), and an RNN (Recurrent Neural Network).

[0061] FIG. 3 is a diagram illustrating an operation flow chart of an electronic device according to one embodiment of the present invention.

[0062] A processor (150) according to one embodiment of the present invention can build an image generation model by applying a feature model generated based on a normal product test image and a defective product test image to a base model learned to generate a defective product image (S10).

[0063] Defective product images refer to product images that have external defects such as scratches, breaks, or dents.

[0064] The image generation model of the present invention is a generative artificial intelligence model, and can be constructed by applying a feature model implemented to fine-tune the features of a base model learned to generate defective product images.

[0065] The base model can be selected based on a model that best reflects the characteristics of the target product. This model varies depending on field and product conditions, so it's recommended that field experts conduct testing to select the model. For example, the base model could be Stable Diffusion, a denoising technique-based model. Among the Stable Diffusion models, the base model could include the v1.5 pruned-emaonly.safetensor model.

[0066] However, generative AI models are primarily trained on people, requiring fine-tuning to ensure the output images correspond to the product. To achieve this, a feature model can be added to the base model to build an image generation model. This feature model can be, for example, a Low-Rank Adaptation (LoRA) model. Even if the number of defective product images is small when creating a LoRA model, this is not a major issue because the base model already contains a wealth of defect information about the defect types.

[0067] There are several ways to create an image generation model, including using the Dreambooth Extension module of Stable Diffusion and using the Kohya-ss module.

[0068] A processor (150) according to one embodiment of the present invention can input defect information in the form of at least one of text and image into an image generation model (S20).

[0069] Defect information is information required to generate a defective product image. The defect information may include a first text describing a possible defect in the product, a second text describing features to be excluded from the defective product image, a normal product image, and a masking image indicating the possible defect in the product. For example, the first text may include "stained," "crack," "failure," "marked," "incontinence," etc., and the second text may include "clean," "clear," "pass," "normal," etc. The first and second texts may be input in the form of prompts, and the processor (150) may apply the defect information with an appropriate weight.

[0070] Text-based defect information can include additional product information. For example, if the target product is an MCCB, defect information can include terms such as "circuit braker" or "MCCB."

[0071] A processor (150) according to one embodiment of the present invention can obtain a defective product image including a defect corresponding to defect information through an image generation model (S30).

[0072] The processor (150) can input a normal product image into an image generation model and acquire a defective product image based on the input image (IMG2IMG function). At this time, the normal product image may utilize one or more of the normal product test images used to form the feature model.

[0073] The processor (150) can acquire a defective product image by targeting the entire area of ​​a normal product image, or can acquire a defective product image by using an inpaint function that masks only a specific area of ​​a normal product image so that a defect is generated. In particular, a masking image is an image that indicates a defect in a specific area, and the processor (150) can change only a specific area of ​​a normal product image into a masking image by using the inpaint function.

[0074] When utilizing the Inpaint function, it is more efficient when creating images to focus on improving types that were not detected by the existing image model or were over-detected as defects when they were not.

[0075] Unlike conventional generative models, generative models based on denoising techniques are advantageous for generating defects from normal product images, demonstrating superior image quality and image generation efficiency. Therefore, while the present invention is described based on examples generated using a generative model based on denoising techniques, the same invention structure can be utilized even if more advanced solutions emerge in the future.

[0076] According to one embodiment of the present invention, the biased performance of existing vision inspection models can be dramatically improved by generating a variety of defective product images. Furthermore, by utilizing a generative model, the quality of the generated defective product images can be nearly identical to that of actual images, at a very high productivity rate. Ultimately, this can shorten the development time for vision inspection models and enhance their performance.

[0077] FIG. 4 is a diagram illustrating the construction of an image generation model according to one embodiment of the present invention. FIG. 4 illustrates the construction of an image generation model, as described in relation to S10 of FIG. 3.

[0078] An electronic device (100) according to one embodiment of the present invention can build an image generation model (430) by applying a feature model generated based on a test image (420) including a normal product test image and a defective product test image to a previously built base model (410).

[0079] At this time, when constructing an image generation model (430), the processor (150) can set defect information and hyperparameters. Hyperparameters are variables that directly affect the image generation quality, and a process of finding optimal values ​​through repeated execution is required.

[0080] FIG. 5 is a diagram illustrating a defective product image generation operation of an electronic device according to a first embodiment of the present invention. FIG. 5 illustrates a defective product image generation operation, as described in relation to S20 of FIG. 3.

[0081] Referring to FIG. 5, the processor (150) can input defect information including a normal product image (510), a first text (520), and a second text (530) into the image generation model (430) to generate a defective product image (540).

[0082] At this time, the processor (150) can similarly set defect information and hyper parameters, and generate a defective product image (540) through inpaint settings.

[0083] FIG. 6 is a drawing illustrating a defective product image generation operation of an electronic device according to a second embodiment of the present invention.

[0084] The defective product image generation method described above with reference to FIG. 5 generates a defective product image using the entire normal product image (510) as input. However, due to the high resolution of the vision inspection image, an issue may arise where the image generation model cannot be trained and the image cannot be generated due to insufficient computing resources. FIG. 6 proposes a method for conserving computing resources by adjusting the normal product image.

[0085] The processor (150) can adjust the test images (610) including a normal product test image and a defective product test image so that each includes a target area. The target area refers to an area where defects are primarily determined to occur in the product, and can be set by coordinates within the image, etc.

[0086] The adjusting technique can be, for example, cropping the test image (610) so that only the target area remains, and any other technique used to extract the target area can be applied without limitation.

[0087] The processor (150) can generate a feature model using an adjusted test image (620) including an adjusted normal product test image and an adjusted defective product test image.

[0088] The processor (150) can build an image generation model (430) by using the adjusted test image (620) as a learning image. In this case, the resources required for learning, such as GPU memory, can be reduced by half, thereby increasing the learning speed.

[0089] Even in the process of using the image generation model (430) thereafter, the processor (150) can adjust the normal product test image to include the same target area as the test image (610).

[0090] The processor (150) can input defect information including an adjusted normal product test image (630) into the image generation model (430) to obtain an adjusted defective product image (670). At this time, as described above, the processor (150) can input defect information including a first text (520) and a second text (530) into the image generation model (430) in addition to the adjusted normal product test image (630) to generate an adjusted defective product image (670). At this time, the processor (150) can similarly set defect information and hyper parameters, and can generate an adjusted defective product image (670) through inpaint settings.

[0091] The processor (150) can generate a final defective product image (690) by combining the remaining area (680) excluding the target area of ​​the normal product test image with the adjusted defective product image (670). At this time, the remaining area (680) can be stored when generating the adjusted normal product test image (630).

[0092] FIG. 7 is a drawing illustrating a defective product image generation operation of an electronic device according to one embodiment of the present invention.

[0093] In Fig. 7, a method is proposed for selecting meaningful images from among generated defective product images and generating labeling information for the selected images to be used in building a vision inspection model.

[0094] First, the processor (150) can adjust the normal product test image (710) into a target area (711) and a remaining area (712), and obtain a defective product image (720) using the target area (711) as an input image. At this time, since the defective product image (720) is of an adjusted size, the processor (150) can generate a final defective product image (740) by combining the defective product image (720) with the remaining area (712). At this time, the processor (150) can generate the defective product image (720) through an inpaint function using a masking image (730).

[0095] Thereafter, the processor (150) can filter the acquired defective product image (740) by comparing the normal product test image (710) with the acquired defective product image based on whether the degree of defect exceeds a threshold. That is, if there is no significant difference in the defective generated portion compared to the original, the defective product image can be excluded from the results.

[0096] At this time, the processor (150) may store defect information, such as a masking image (730), as labeling information (760), and match and store the labeling information (760) and the filtered defective product image (740). The labeling information (760) may also include location and defect type information for the defective portion.

[0097] The processor (150) can generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information about defects included in the defective product images, and normal product images.

[0098] According to one embodiment of the present invention, the quality of generated defective product images can be improved to enhance the performance of a vision inspection model.

[0099] FIG. 8 is a drawing illustrating examples of defective product images according to one embodiment of the present invention.

[0100] The upper left image in Figure 8 is an image of a normal product, and the remaining images are images of defective products.

Claims

1. In electronic devices, An image generation model is built by applying a feature model generated based on normal product test images and defective product test images to a base model trained to generate defective product images. Input defect information in the form of at least one of text and image into the image generation model, An electronic device including a processor that obtains a defective product image including a defect corresponding to the defect information through the image generation model.

2. In paragraph 1, An electronic device characterized in that the above feature model is a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

3. In paragraph 1, The above processor, An electronic device receiving defect information including a first text describing a possible defect in a product.

4. In paragraph 3, The above processor, An electronic device further receiving defect information including a second text describing a feature to be excluded from the defective product image.

5. In paragraph 1, The above processor, An electronic device that receives defect information including a masking image showing a possible defect in a product.

6. In paragraph 1, The above processor, The above normal product test image and the above defective product test image are each adjusted to include the target area, An electronic device for generating the feature model using the above-described adjusted normal product test image and the above-described adjusted defective product test image.

7. In paragraph 6, The above processor, By inputting defect information including the adjusted normal product test image into the image generation model, an adjusted defective product image is obtained, An electronic device that generates a defective product image by combining the remaining areas excluding the target area of ​​the above normal product test image.

8. In paragraph 1, The above processor, An electronic device that matches the above defect information with the acquired defective product image and stores it.

9. In paragraph 1, The above processor, An electronic device that compares the above normal product test image with the above acquired defective product image and filters the above acquired defective product image based on whether the degree of defect exceeds a threshold.

10. In electronic devices, A processor is included to generate a vision inspection model that inspects whether a product is defective by using defective product images obtained from an image generation model, labeling information about defects included in the defective product images, and normal product images. An electronic device characterized in that the image generation model is constructed by applying a feature model generated based on a normal product test image and a defective product test image to a base model learned to generate a defective product image.

11. In paragraph 10, An electronic device characterized in that the above feature model is a Low-Rank Adaptation (LoRA) model that fine-tunes the base model so that the defective product image corresponds to the product.

12. In paragraph 10, The images of the above defective products are, An electronic device characterized in that it includes a defect corresponding to defect information input to the image generation model, wherein the defect information is in the form of at least one of text and an image.

13. In paragraph 12, The above text, Contains a first text describing a possible defect in the product or a second text describing a feature to be excluded from the image of the defective product; The image above is, An electronic device characterized by including a masking image showing a possible defect in the product.

14. In paragraph 10, The above processor, Obtain an exterior image of the above product, An electronic device that inputs the appearance image into the vision inspection model to inspect the product for defects.

15. A method for generating a defective product image performed by an electronic device, A step of building an image generation model by applying a feature model generated based on a normal product test image and a defective product test image to a base model trained to generate a defective product image; A step of inputting defect information in the form of at least one of text and image to the image generation model; A method comprising the step of obtaining a defective product image including a defect corresponding to the defect information through the image generation model.

Images