Defect Image Generation System and Defect Image Generation Method

Abstract

A defect image generation system and method are disclosed. The defect image generation system includes a reference database, a defect database, and a processing device. The processing device runs a diffusion model and an application. The application combines a potential high-dimensional vector, a reference potential high-dimensional vector, and a defect potential high-dimensional vector to generate a synthetic high-dimensional vector. The diffusion model generates additional prediction noise based on the defect potential high-dimensional vector and a cue word high-dimensional vector, and generates main prediction noise based on a noisy synthetic high-dimensional vector generated by the application along different time steps. The noisy synthetic high-dimensional vector is then progressively denoised using the main prediction noise and the additional prediction noise to generate a generative defect image.

Description

[Technical Field]

[0001] This invention relates to an image generation system and an image generation method, and more particularly to a defect image generation system and a defect image generation method. [Previous Technology]

[0002] In recent years, manufacturers have widely used image recognition to determine whether their finished or semi-finished products have defects. However, most manufacturers do not have a sufficient number of defect samples, which can easily lead to the inaccuracy of image recognition of finished or semi-finished products. Therefore, there is an urgent need for researchers in related fields to propose corresponding solutions. [Summary of the Invention]

[0003] Therefore, the object of the present invention is to provide a defect image generation system and a defect image generation method for generating defect images according to requirements.

[0004] Therefore, the defect image generation system of the present invention is applicable to a display device and an input device. The defect image generation system includes a reference database storing multiple reference images, a defect database storing multiple defect images, and a processing device. The processing device is electrically connected to the reference database and the defect database and is used to electrically connect the display device and the input device. The processing device is used to run a diffusion model and an application. The diffusion model has a prompt word encoder, a first image encoder, a second image encoder, a noise prediction module, a primary attention layer, an interactive attention layer, a feature memory, and a decoder. The application is used to display a graphical user interface on the display device and to generate a prompt word, a reference area marking instruction, and a defect masking instruction via the input device. The application is used to generate a prompt word, a reference area marking instruction, and a defect masking instruction based on the reference area... The domain labeling instruction selects one of the reference images and obtains a portion to be replaced and an unreplaced portion from the reference image. The application is used to select one of the defective images according to the defect masking instruction and obtain a portion to be extracted and an unreplaced portion from the defective image. The cue word encoder is used to receive the cue word and generate a cue word high-dimensional vector indicating the semantics of the cue word. The first image encoder is used to receive one of the defective images and generate a latent high-dimensional vector indicating the features of the one defective image. The second image encoder is used to receive the cue word. The unreplaced portion of a reference image and the portion to be extracted in a defective image are used to generate a reference latent high-dimensional vector indicating features of the unreplaced portion and a defect latent high-dimensional vector indicating features of the portion to be extracted. The noise prediction module and the original attention layer are used to generate additional prediction noise based on the corresponding high-dimensional vectors of the prompt words and the defect latent high-dimensional vectors. The application combines the latent high-dimensional vectors, the reference latent high-dimensional vector, and the defect latent high-dimensional vector to generate a synthesized high-dimensional vector. The application progressively processes the synthesized high-dimensional vector along a complex number of time steps. Noise addition; the application is used to extract a noisy synthetic high-dimensional vector generated after adding noise to the synthetic high-dimensional vector at each time step, and store it in the feature memory; the noise prediction module, the original attention layer and the interactive attention layer are used together to generate a main prediction noise corresponding to the noisy synthetic high-dimensional vector generated at each time step; the application is used to progressively denoise the noisy synthetic high-dimensional vector according to the main prediction noise and the additional prediction noise, so as to generate a denoised synthetic high-dimensional vector; the decoder is used to receive and restore the denoised synthetic high-dimensional vector to generate a generative defect image.

[0005] Therefore, the defect image generation method of the present invention is implemented by a display device, an input device, a reference database storing multiple reference images, a defect database storing multiple defect images, and a processing device. The processing device is used to run a diffusion model and an application. The diffusion model has a prompt word encoder, a first image encoder, a second image encoder, a noise prediction module, a primary attention layer, an interactive attention layer, a feature memory, and a decoder. The application is used to display a graphical user interface on the display device and to generate a prompt word, a reference area marking instruction, and a defect masking mark through operation via the input device. The defect image generation method comprises the following steps: (A) selecting one of the reference images by the application according to the reference region marking instruction, to obtain a portion to be replaced and an unreplaced portion from the reference image; (B) selecting one of the defect images by the application according to the defect mask marking instruction, to obtain a portion to be extracted and an unextracted portion from the defect image; (C) receiving the prompt word by the prompt word encoder, to generate a prompt word high-dimensional vector indicating the semantics of the prompt word; (D) receiving one of the defect images by the first image encoder, to generate a feature indicating the defect image. (E) The second image encoder receives the unperformed portion of one of the reference images and the portion to be extracted of one of the defective images to generate a reference latent high-dimensional vector indicating the features of the unperformed portion and a defect latent high-dimensional vector indicating the features of the portion to be extracted; (F) The noise prediction module and the original attention layer generate additional prediction noise based on the cue word high-dimensional vector and the defect latent high-dimensional vector; (G) The application combines the latent high-dimensional vector, the reference latent high-dimensional vector, and the defect latent high-dimensional vector to generate a synthesized high-dimensional vector; (H) The application executes the high-dimensional vector along multiple time intervals. The process involves: (I) progressively adding noise to the synthesized high-dimensional vector at each time step; (J) extracting a noisy synthesized high-dimensional vector generated at each time step using the application and storing it in the feature memory; (J) generating a subject prediction noise based on the noisy synthesized high-dimensional vector generated at each time step using the noise prediction module, the original attention layer, and the interactive attention layer; (K) progressively denoising the noisy synthesized high-dimensional vector using the application based on the subject prediction noise and the additional prediction noise to generate a denoised synthesized high-dimensional vector; and (L) restoring the denoised synthesized high-dimensional vector using the decoder to generate a generative defect image.

[0006] The advantage of the present invention is that by setting up the processing device to run the diffusion model and the application, the application combines the potential high-dimensional vector, the reference potential high-dimensional vector and the defect potential high-dimensional vector to generate the synthetic high-dimensional vector accordingly, and the diffusion model denoises the noisy synthetic high-dimensional vector according to the main prediction noise and the additional prediction noise, thereby generating the generative defect image that meets the current requirements.

Implementation Method

[0007] Referring to FIG1, an embodiment of the defect image generation system of the present invention is applicable to a display device 91 and an input device 92. In this embodiment, the display device 91 is a display panel, and the input device 92 is a keyboard and a mouse. In a variation of this embodiment, the display device 91 and the input device 92 may be implemented by a touch screen with touch capability.

[0008] This embodiment includes a reference database 1 for storing multiple reference images, a defect database 2 for storing multiple defect images, and a processing device 3. In this embodiment, the reference database 1, the defect database 2, and the processing device 3 can be implemented jointly by a computer or server with computing and storage capabilities.

[0009] The reference database 1 is used to store multiple reference images. In this embodiment, these reference images are images that have been actually captured and have not suffered damage or defects.

[0010] The defect database 2 is used to store multiple defect images. In this embodiment, these defect images are images that were actually captured and have caused damage or defects.

[0011] The processing device 3 is electrically connected to the input device 92, the display device 91, the reference database 1 and the defect database 2, and is used to run a diffusion model 31 and an application 32.

[0012] The diffusion model 31 has a prompt word encoder 310, a first image encoder 311, a second image encoder 312, a noise prediction module 313, a primary attention layer 314, an interactive attention layer 315, a feature memory 316, and a decoder 317.

[0013] In this embodiment, the diffusion model 31 adopts the existing Transformer stacking architecture and is a Denoising Diffusion Implicit Model (DDIM), thereby performing denoising in fewer steps.

[0014] The application 32 is used to display a graphical user interface 321 on the display device 91, and to generate a prompt, a reference area marker instruction, and a defect masking marker instruction via the input device 92. The graphical user interface 321 has a reference image adjustment sub-interface 322, a defect image adjustment sub-interface 323, and a defect image generation sub-interface 324. The defect masking marker instruction has a capture sub-instruction, a scaling sub-instruction, and a rotation sub-instruction. In this embodiment, the application 32 is software that executes algorithms and provides the graphical user interface 321.

[0015] Referring to Figures 1 to 3, a defect image generation method is performed by means of this embodiment, the display device 91, and the input device 92, comprising the following steps:

[0016] Step S1: The application 32 selects one of the reference images according to the reference area marking instruction to obtain a portion to be replaced and an unreplaced portion from the one of the reference images. In this embodiment, the display device 91 displays the reference image adjustment sub-interface 322 to display one of the reference images, and the input device 92 generates the reference area marking instruction to obtain the portion to be replaced and the unreplaced portion from the one of the reference images.

[0017] Step S2: The application 32 selects one of the defect images according to the defect masking mark instruction to obtain a portion to be extracted and a portion not extracted from the defect image. In this embodiment, the display device 91 displays the defect image adjustment sub-interface 323 to display one of the defect images, and the input device 92 generates the defect masking mark instruction to obtain the portion to be extracted and the portion not extracted from the defect image. Specifically, the defect masking mark instruction includes a capture sub-instruction, a scaling sub-instruction, and a rotation sub-instruction to capture, scale, and rotate the defect portion presented in the defect image to generate the portion to be extracted.

[0018] Step S3: The prompt word is received by the prompt word encoder 310 so as to generate a prompt word high-dimensional vector that indicates the semantics of the prompt word.

[0019] Step S4: The first image encoder 311 receives one of the defective images so that it generates a potential high-dimensional vector indicating the features of the one of the defective images.

[0020] Step S5: The second image encoder 312 receives the unreplaced portion of one of the reference images and the portion to be extracted of one of the defective images, so as to generate a reference potential high-dimensional vector indicating the features of the unreplaced portion and a defect potential high-dimensional vector indicating the features of the portion to be extracted.

[0021] Step S6: The noise prediction module 313 and the original attention layer 314 jointly generate an additional prediction noise based on the correspondence between the high-dimensional vector of the prompt word and the high-dimensional vector of the defect potential. In this embodiment, the original attention layer 314 is used to capture the features extracted by the prompt word encoder 310 and the second image encoder 312, and the multi-layer perceptron or fully connected layer contained in the noise prediction module 313 makes predictions based on the features captured by the original attention layer 314, thereby generating the additional prediction noise.

[0022] Step S7: The application 32 combines the potential high-dimensional vector, the reference potential high-dimensional vector, and the defect potential high-dimensional vector to generate a synthetic high-dimensional vector. In this embodiment, the application 32 aligns and superimposes the unperformed portion of one of the defect images indicated by the potential high-dimensional vector, the unperformed portion of one of the reference images indicated by the reference potential high-dimensional vector, and the portion to be extracted of one of the defect images indicated by the defect potential high-dimensional vector.

[0023] Step S8: The application 32 gradually adds noise to the synthesized high-dimensional vector along a complex time step.

[0024] Step S9: The application 32 extracts a noisy synthetic high-dimensional vector generated after adding noise to the synthetic high-dimensional vector at each time step, and stores it in the feature memory 316.

[0025] Step S10: Using the noise prediction module 313, the original attention layer 314, the interactive attention layer 315 and the feature memory 316, a main prediction noise is generated according to the noise-added synthesis high-dimensional vector generated at each time step.

[0026] Step S11: The application 32 progressively denoises the noisy synthesized high-dimensional vector based on the main prediction noise and the additional prediction noise, thereby generating a denoised synthesized high-dimensional vector. In this embodiment, the application 32 progressively denoises the noisy synthesized high-dimensional vector along a complex number of denoising time steps based on the main prediction noise and a certain proportion of the additional prediction noise, and adds random noise when denoising the synthesized high-dimensional vector at each denoising time step, thereby generating the denoised synthesized high-dimensional vector. It is worth mentioning that the proportion of the additional prediction noise is adjusted and set by the user.

[0027] Step S12: The decoder 317 receives and restores the denoised synthetic high-dimensional vector so as to generate a generative defect image.

[0028] Step S13: The defect image generation sub-interface 324 is displayed by the display device 91, thereby displaying the generated defect image on the defect image generation sub-interface 324 (as shown in Figure 4).

[0029] In summary, this embodiment has the following effects:

[0030] (a) By setting up the processing device 3 to run the diffusion model 31 and the application 32, the application 32 combines the potential high-dimensional vector, the reference potential high-dimensional vector and the defect potential high-dimensional vector to generate the synthetic high-dimensional vector accordingly. The diffusion model 31 and the application 32 denoise the noisy synthetic high-dimensional vector according to the main prediction noise and the additional prediction noise, thereby generating the generative defect image that meets the current requirements.

[0031] (ii) By setting the processing device 3 to run the diffusion model 31 and the application 32, the application 32 adds the random noise when denoising the synthesized high-dimensional vector at each denoising time step, thereby preventing the denoising process from over-converging and maintaining texture details.

[0032] Therefore, the defect image generation system and defect image generation method of the present invention can generate defect images according to requirements, and can indeed achieve the purpose of the present invention.

[0033] However, the above description is only an embodiment of the present invention and should not be construed as limiting the scope of the present invention. Any simple equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the patent specification of the present invention shall still fall within the scope of the patent of the present invention. [Simplified Explanation of the Diagram]

[0034] Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, wherein: FIG1 is a block diagram of an embodiment of the defect image generation system of the present invention; FIG2-3 is a flowchart illustrating the execution of a defect image generation method by the embodiment; and FIG4 is an appearance schematic diagram showing a display device displaying a defect image generation sub-interface of a graphical user interface in the embodiment.

Claims

1. A defect image generation system, applicable to a display device and an input device, the defect image generation system comprising: a reference database storing a plurality of reference images; a defect database storing a plurality of defect images; and a processing unit electrically connected to the reference database and the defect database and for electrically connecting the display device and the input device, the processing unit being used to run a diffusion model and an application, the diffusion model having a prompt word encoder, a first image encoder, a second image encoder, a noise prediction module, a primary attention layer, an interactive attention layer, a feature memory, and a decoder, the application being used to display a graphical user interface on the display device and to generate a prompt word, a reference region marker instruction, and a defect masking marker instruction via operation of the input device. The application is configured to select one of the reference images according to the reference region marking instruction, and obtain a portion to be replaced and an unreplaced portion from the reference image; the application is configured to select one of the defect images according to the defect mask marking instruction, and obtain a portion to be extracted and an unextracted portion from the defect image; the prompt word encoder is configured to receive the prompt word to generate a corresponding high-dimensional prompt word vector indicating the semantics of the prompt word; the first image encoder is configured to receive one of the defect images to generate a corresponding latent high-dimensional vector indicating the features of the one of the defect images; the first... The dual image encoder receives the unreplaced portion of a reference image and the portion to be extracted from a defective image to generate a reference latent high-dimensional vector indicating features of the unreplaced portion and a defect latent high-dimensional vector indicating features of the portion to be extracted. The noise prediction module generates additional prediction noise based on the prompt word high-dimensional vector and the defect latent high-dimensional vector. The application combines the latent high-dimensional vector, the reference latent high-dimensional vector, and the defect latent high-dimensional vector to generate a synthesized high-dimensional vector. The application iteratively processes the synthesized high-dimensional vector along a complex number of time steps. The application performs noise addition; it extracts a noisy synthetic high-dimensional vector generated after adding noise to the synthetic high-dimensional vector at each time step and stores it in the feature memory; the noise prediction module, the original attention layer, and the interactive attention layer are used together to generate a main prediction noise corresponding to the noisy synthetic high-dimensional vector generated at each time step; the application performs stepwise denoising on the noisy synthetic high-dimensional vector based on the main prediction noise and the additional prediction noise to generate a denoised synthetic high-dimensional vector; the decoder receives and restores the denoised synthetic high-dimensional vector to generate a generative defect image.

2. The defect image generation system as described in claim 1, wherein, The graphical user interface has a reference image adjustment sub-interface, a defect image adjustment sub-interface, and a defect image generation sub-interface. The reference image adjustment sub-interface is used to display one of the reference images and generate the reference area marking instruction through operation, thereby obtaining the part to be replaced and the part not replaced from the one of the reference images. The defect image adjustment sub-interface is used to display one of the defect images and generate the defect masking marking instruction through operation, thereby obtaining the part to be extracted and the part not extracted from the one of the defect images.

3. The defect image generation system as described in claim 2, wherein, The defect masking mark command generated by the defect image adjustment sub-interface has a capture sub-command, a scaling sub-command, and a rotation sub-command.

4. A defect image generation method, implemented by a display device, an input device, a reference database storing multiple reference images, a defect database storing multiple defect images, and a processing device, wherein the processing device is used to run a diffusion model and an application, the diffusion model having a prompt word encoder, a first image encoder, a second image encoder, a noise prediction module, a primary attention layer, an interactive attention layer, a feature memory, and a decoder, the application being used to display a graphical user interface on the display device and to generate a prompt word, a reference region marking instruction, and a defect masking instruction via the input device, the defect image generation method comprising the following steps: (A) selecting one of the reference images by the application according to the reference region marking instruction to obtain a portion to be replaced and an unreplaced portion from the one of the reference images; (B) selecting one of the defect images by the application according to the defect masking instruction to obtain an extracted portion and an unextracted portion from the one of the defect images; (C) The prompt word encoder receives the prompt word to generate a corresponding high-dimensional vector indicating the semantics of the prompt word; (D) The first image encoder receives one of the defective images to generate a corresponding latent high-dimensional vector indicating the features of the defective image; (E) The second image encoder receives the unreplaced portion of one of the reference images and the portion to be extracted of one of the defective images to generate a corresponding reference latent high-dimensional vector indicating the features of the unreplaced portion and a defect latent high-dimensional vector indicating the features of the portion to be extracted; (F) The noise prediction module and the original attention layer generate additional prediction noise based on the prompt word high-dimensional vector and the defect latent high-dimensional vector; (G) The application combines the latent high-dimensional vector, the reference latent high-dimensional vector, and the defect latent high-dimensional vector to generate a corresponding synthetic high-dimensional vector; (H) The application progressively adds noise to the synthetic high-dimensional vector along multiple time steps; (I) The application extracts a noisy synthetic high-dimensional vector generated after adding noise to the synthetic high-dimensional vector at each time step and stores it in the feature memory; (J) The noise prediction module, the original attention layer and the interactive attention layer generate a main prediction noise according to the noisy synthetic high-dimensional vector generated at each time step; (K) The application performs stepwise denoising on the noisy synthetic high-dimensional vector according to the main prediction noise and the additional prediction noise to generate a denoised synthetic high-dimensional vector; and (L) The decoder restores the denoised synthetic high-dimensional vector to generate a generative defect image.

5. The defect image generation method as described in claim 4 further includes a follow-up step (L) in step (M), wherein, The graphical user interface has a reference image adjustment sub-interface, a defect image adjustment sub-interface, and a defect image generation sub-interface. In step (A), the reference image adjustment sub-interface is displayed by the display device to display one of the reference images, and the input device generates the reference area marking instruction to obtain the part to be replaced and the part not to be replaced from one of the reference images. In step (B), the defect image adjustment sub-interface is displayed by the display device to display one of the defect images, and the input device generates the defect masking marking instruction to obtain the part to be extracted and the part not to be extracted from one of the defect images. In step (M), the defect image generation sub-interface is displayed by the display device, thereby displaying the generated defect image in the defect image generation sub-interface.

6. The defect image generation method as described in claim 5, wherein, In step (B), the display device displays the defect image adjustment sub-interface of the graphical user interface to display one of the defect images, and the input device generates the defect masking mark instruction to obtain the part to be extracted and the part not extracted from one of the defect images. The defect masking mark instruction has a capture sub-instruction, a scaling sub-instruction and a rotation sub-instruction.

Images