Workpiece defect image generation method, device, equipment and medium

Abstract

The embodiment of the invention provides a workpiece defect image generation method and device, equipment and a medium, and the method comprises the steps: firstly, cutting an input high-resolution defect image to obtain a slice containing a generated defect region, and cutting the slice to obtain a picture containing a defect region; after data enhancement operation is carried out, the data are sequentially input into a CLIP picture encoder and a multi-layer perceptron to obtain feature vectors, and mask shape enhancement processing is carried out on the feature vectors to generate an image with a defect mask; splicing the image with the defect mask and the Gaussian noise, inputting the spliced image and the feature vector into a diffusion model for training, diffusing to obtain a generated picture, pasting the generated picture to an input high-resolution defect image, inputting the high-resolution defect image into a variational auto-encoder, and decoding to obtain an original-resolution defect picture. Lora is added, so that the method is more suitable for generation of the defect field, low-dimensional feature vectors are recovered to generate high-quality and high-resolution defect images, and it is ensured that the generated images have high definition and details.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a method, device, equipment and medium for generating a workpiece defect image. Background Art

[0002] In traditional GAN-based methods, the defect images generated are low-resolution and unnatural. This is because the GAN model needs to process more pixels and more complex features when generating high-resolution, detail-rich images, which increases the difficulty of training. If the images in the training dataset are low-resolution or lack diversity, the generated images will also be affected. Traditional GAN ​​architectures may not be able to effectively capture subtle features in images, causing the generated images to look unrealistic.

[0003] In addition, in order to make the generated defects more accurately match the real situation, a finely defined mask is usually required to indicate which areas should be modified. Providing a fine mask usually requires a lot of manual annotation work, which increases the cost of data preparation.

[0004] Most methods cannot control the intensity of defect generation, because the intensity of defect generation is usually determined by the internal parameters of the model, which are often difficult to adjust directly. At the same time, the model design does not take into account the user's specific requirements for defect intensity.

[0005] Therefore, it is necessary to study a solution to solve the above technical problems. Summary of the invention

[0006] The present application provides a method, device, equipment and medium for generating workpiece defect images based on a diffusion model. By inputting the image into the diffusion model, the goal of generating high-quality, long-duration video from a single image is achieved, and the naturalness of face generation and the smoothness of video movements are improved.

[0007] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0008] In a first aspect, an embodiment of the present application provides a method for generating a workpiece defect image, the method being applied to a paint by example model, the paint by example model comprising a diffusion model with a lora module, a CLIP picture encoder, a multilayer perceptron, and a variational autoencoder, including:

[0009] The input high-resolution defect image is cropped to obtain a slice containing the generated defect area;

[0010] Cropping the slice containing the generated defect area to obtain a picture containing the defect area;

[0011] After data enhancement operation, the image containing the defect area is input into CLIP image encoder and multi-layer perceptron to obtain feature vector;

[0012] Perform mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask;

[0013] The image with the defect mask is concatenated with Gaussian noise and then input into the diffusion model together with the feature vector for training to obtain a trained diffusion model, and the trained diffusion model is used for diffusion to obtain a generated image;

[0014] The generated image is pasted onto the input high-resolution defect image and then input into the variational autoencoder for decoding to obtain the original resolution defect image.

[0015] In a possible implementation, the step of performing a data enhancement operation on a picture containing a defective area and then sequentially inputting the picture into a CLIP picture encoder and a multi-layer perceptron to obtain a feature vector includes:

[0016] Input the image containing the defect area into the CLIP image encoder for encoding to obtain the image feature representation;

[0017] The image feature vector is input into a multi-layer perceptron for nonlinear transformation to obtain the feature vector.

[0018] In a possible implementation, the step of performing mask shape enhancement processing on a picture containing a defect area to generate an image with a defect mask includes:

[0019] The image containing the defect area is multiplied by the mask to generate an image with the defect mask.

[0020] In a possible implementation, randomly selected images of non-defective regions or defective regions are used as true values ​​for training.

[0021] In a possible implementation, the method further includes:

[0022] If the slice containing the generated defect area does not meet the processing requirements, it is scaled.

[0023] In a second aspect, an embodiment of the present application further provides a workpiece defect image generating device, the device comprising:

[0024] A first cropping module, used for cropping the input high-resolution defect image to obtain a slice containing a generated defect area;

[0025] A second cropping module is used to crop the slice containing the generated defect area to obtain a picture containing the defect area;

[0026] A data enhancement module is used to perform data enhancement operations on the image containing the defect area and then input it into the CLIP image encoder and the multi-layer perceptron to obtain a feature vector;

[0027] The mask shape enhancement module is used to perform mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask;

[0028] A training and diffusion module, which is used to concatenate the image with the defect mask and the Gaussian noise and then input it into the diffusion model together with the feature vector for training to obtain a trained diffusion model, and use the trained diffusion model for diffusion to obtain a generated image;

[0029] The decoding module is used to paste the generated image onto the input high-resolution defect image and then input the image into the variational autoencoder for decoding to obtain the original resolution defect image.

[0030] In a possible implementation, the data enhancement module is further used to:

[0031] Input the image containing the defect area into the CLIP image encoder for encoding to obtain the image feature representation;

[0032] The image feature vector is input into a multi-layer perceptron for nonlinear transformation to obtain the feature vector.

[0033] In a possible implementation, the mask shape enhancement module is further used to:

[0034] The image containing the defect area is multiplied by the mask to generate an image with the defect mask.

[0035] In a third aspect, the present application further proposes a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the workpiece defect image generation method as described in any one of the first aspects.

[0036] In a fourth aspect, the present application further proposes a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the workpiece defect image generation method as described in any one of the first aspects.

[0037] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0038] Compared with the prior art, the embodiments of the present application provide a method, device, equipment and medium for generating a workpiece defect image. First, the input high-resolution defect image is cropped to obtain a slice containing the generated defect area, and then the slice is cropped to obtain a picture containing the defect area. After data enhancement operation, the slice is sequentially input into the CLIP picture encoder and the multi-layer perceptron to obtain a feature vector, and the mask shape enhancement processing is performed to generate an image with a defect mask. The image with the defect mask is spliced ​​with Gaussian noise and then input into the diffusion model together with the feature vector for training and then diffused to obtain a generated picture. The generated picture is pasted into the input high-resolution defect image and then input into the variational autoencoder for decoding to obtain the original resolution defect image. The addition of lora makes this method more suitable for the generation of defect areas, and restores the low-dimensional feature vector to generate a high-quality, high-resolution defect image, ensuring that the generated image has high clarity and details. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic flow chart of a workpiece defect image generation method proposed in an embodiment of the present application is shown.

[0041] Figure 2 A schematic diagram of the paint by example model proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0044] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0045] Please refer to Figure 1 and Figure 2 , Figure 1 A schematic flow chart of a workpiece defect image generation method proposed in an embodiment of the present application is shown. Figure 2 A schematic diagram of a paint by example model proposed in an embodiment of the present application is shown. The method is applied to the paint by example model. The paint by example model includes a diffusion model with a lora module, a CLIP picture encoder, a multilayer perceptron, and a variational autoencoder.

[0046] The diffusion model is a deep learning model for generating high-quality images. It restores the image by gradually removing the noise added to the original image to generate a new image.

[0047] The LORA (Low-Rank Adaptation) module can optimize model performance by adding a small number of trainable parameter layers without modifying the basic model parameters, so that the model can better adapt to specific tasks or data sets.

[0048] The CLIP (Contrastive Language–Image Pre-training) image encoder is a multimodal pre-training model that can encode images into feature vectors for extracting high-level features from the input image, which are then fed into a multi-layer perceptron for further processing.

[0049] Multilayer Perceptron (MLP), a feed-forward neural network, is used to further process the feature vector obtained from the CLIP encoder to help the model better understand and utilize these features, thereby improving the quality of the generated images.

[0050] Variational Autoencoders (VAEs) are generative models used for image reconstruction, which reconstruct images generated by the diffusion model into high-resolution defect images by learning a latent space distribution and mapping it back to the image space during the decoding process, thereby generating high-quality output.

[0051] The paint by example model in the embodiment of the present application executes a defect image generation method by combining a diffusion model with a LORA module, a CLIP picture encoder, a multilayer perceptron and a variational autoencoder. It can not only generate realistic defect images, but also significantly improve the adaptability and generation effect of the model in specific fields through the introduction of the LORA module.

[0052] The workpiece defect image generation method includes:

[0053] Step S1: cropping the input high-resolution defect image to obtain a slice containing a generated defect area.

[0054] Use existing detection algorithms or manual marking to identify defective areas in the image. Generate a bounding box for each defective area. Crop the image according to the bounding box to obtain a slice (512×512) containing the defective area. The size of the input original image is 512×512 pixels.

[0055] If the slice containing the generated defect area does not meet the processing requirements, it is scaled.

[0056] After cropping, if the cropped image size does not meet the requirements of subsequent processing, it needs to be scaled to meet the specific size requirements.

[0057] Step S2: cropping the slice containing the generated defect area to obtain a picture containing the defect area.

[0058] The cropped slices are further cropped to obtain small images containing a single defective area. Based on the initial cropping, each slice containing multiple defective areas needs to be cropped again to ensure that each small image contains only a single defective area, and finally a series of small images are obtained, each containing only one defective area (including images of defective areas).

[0059] Step S3: After performing data enhancement operation on the image containing the defect area, the image is sequentially input into the CLIP image encoder and the multi-layer perceptron to obtain a feature vector.

[0060] The image containing the defect area is input into the CLIP image encoder to extract the semantic features of the image. The feature vector output by the CLIP encoder is further processed by a multi-layer perceptron (MLP) to generate a more representative feature vector. The processed feature vector is input into the diffusion model as part of the conditional input.

[0061] Step S3 includes:

[0062] Input the image containing the defect area into the CLIP image encoder for encoding to obtain the image feature representation;

[0063] The image feature vector is input into a multi-layer perceptron for nonlinear transformation to obtain the feature vector.

[0064] The image containing the defect area is input into the CLIP image encoder, which encodes the image and generates a vector representing the image features (image feature representation). The image feature vector obtained from the CLIP image encoder is input into the multilayer perceptron, which performs nonlinear transformation on the feature vector through a multilayer neural network to generate the final feature vector.

[0065] Step S4: performing mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask.

[0066] An image containing defective areas refers to an image in which the areas where defects need to be generated (or contain defects) are replaced with irregularly shaped black layers on the original image, or in other words, an irregularly shaped black layer is used to cover the areas where defects need to be generated, define one or more defective areas, and generate masks m of different shapes through random distortion, magnification and other operations to specify the areas that need to be processed by the model to generate defects.

[0067] The image containing the defect area is multiplied by the mask to generate an image with the defect mask.

[0068] The mask m is a binary layer with the same size as the input image. In the mask, the positions with a value of 0 represent the places where defects need to be generated, and these positions will be replaced by a black layer in the final image. In the mask, the positions with a value of 1 represent the parts that do not need to be replaced, and these positions remain unchanged in the final image. Applying the mask m to the image containing the defective area generates an image with a black irregular shape occlusion, and obtains an image in which the defective area is occluded by the black layer. The processed image is spliced ​​with Gaussian noise to form the input data.

[0069] Step S5: splicing the image with the defect mask with Gaussian noise and inputting it into the diffusion model together with the feature vector for training to obtain a trained diffusion model, and using the trained diffusion model to perform diffusion to obtain a generated image.

[0070] During the training process, randomly selected images of non-defective areas or defective areas are used as true values ​​for training. During the generation process, the generation intensity of defects can be controlled by adjusting the intensity, thereby achieving more flexible image generation.

[0071] In the generation phase, given a highly noisy initial state, the model gradually performs denoising operations until it reaches the final state. The entire diffusion process is a recursive denoising cycle, and each iteration brings the output image closer to the target. First, starting with a highly noisy image, the model predicts and removes part of the noise to get a new image, and repeats the previous steps until the final state is reached. When a satisfactory result is obtained, it is placed back to the corresponding position in the original resolution image, and the output quality is further optimized through a variational autoencoder (VAE) to ensure that the generated defect image blends naturally into the background and the overall visual effect is natural and smooth.

[0072] The image with the defect area masked off is concatenated with the noise to form a new input. The concatenated image and feature vector are input into the diffusion model together. The LoRA module is added to the diffusion model to enable the model to better adapt to tasks in specific fields, such as defect generation, and the model generates repaired defect areas.

[0073] Step S6: Paste the generated image onto the input high-resolution defect image and then input the image into the variational autoencoder for decoding to obtain the original resolution defect image.

[0074] The generated defect area is placed back into the original high-resolution image. The pasted image is input into the VAE for decoding, and finally the defect image with the original resolution is obtained.

[0075] In summary, this application can crop defect areas from high-resolution defect images, generate diverse defect shapes, extract feature vectors using CLIP and MLP, and generate defect images in combination with the LoRA-adapted diffusion model. Finally, high-quality original-resolution defect images are obtained through VAE decoding.

[0076] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0077] First, by providing different types of masks, users can flexibly control the location and shape of defects, thereby generating more diverse defect images, enabling the model to generate more demanding results based on user guidance, thereby enhancing the diversity and controllability of the output.

[0078] Second, it can generate defect images at different image resolutions. The generated defect images maintain high quality and will not be significantly blurred or distorted due to the increase in resolution. Even at high resolutions, the generated images still maintain high details and clarity, and will not suffer quality degradation due to compression algorithms.

[0079] Third, the intensity of defect generation can be adjusted by adjusting the strength parameter.

[0080] A possible implementation of a workpiece defect image generating device is provided below, which is used to execute the various execution steps and corresponding technical effects of the workpiece defect image generating method shown in the above embodiment and possible implementation. The device includes:

[0081] A first cropping module, used for cropping the input high-resolution defect image to obtain a slice containing a generated defect area;

[0082] A second cropping module is used to crop the slice containing the generated defect area to obtain a picture containing the defect area;

[0083] A data enhancement module is used to perform data enhancement operations on the image containing the defect area and then input it into the CLIP image encoder and the multi-layer perceptron to obtain a feature vector;

[0084] The mask shape enhancement module is used to perform mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask;

[0085] A training and diffusion module, which is used to concatenate the image with the defect mask and the Gaussian noise and then input it into the diffusion model together with the feature vector for training to obtain a trained diffusion model, and use the trained diffusion model for diffusion to obtain a generated image;

[0086] The decoding module is used to paste the generated image onto the input high-resolution defect image and then input the image into the variational autoencoder for decoding to obtain the original resolution defect image.

[0087] In a possible implementation, the data enhancement module is further used to:

[0088] Input the image containing the defect area into the CLIP image encoder for encoding to obtain the image feature representation;

[0089] The image feature vector is input into a multi-layer perceptron for nonlinear transformation to obtain the feature vector.

[0090] In a possible implementation, the mask shape enhancement module is further used to:

[0091] The image containing the defect area is multiplied by the mask to generate an image with the defect mask.

[0092] This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the workpiece defect image generation method provided in the embodiments of the present application. Therefore, the beneficial effects of the workpiece defect image generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0093] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps of any embodiment of the workpiece defect image generation method provided in the embodiment of the present application.

[0094] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0095] Since the instructions stored in the storage medium can execute the steps in any workpiece defect image generation method embodiment provided in the embodiments of the present application, the beneficial effects that can be achieved by any workpiece defect image generation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0096] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a workpiece defect image, characterized in that: The method is applied to a paint by example model, which includes a diffusion model with a lora module, a CLIP picture encoder, a multilayer perceptron, and a variational autoencoder, including: The input high-resolution defect image is cropped to obtain a slice containing the generated defect area; Cropping the slice containing the generated defect area to obtain a picture containing the defect area; After data enhancement operation, the image containing the defect area is input into CLIP image encoder and multi-layer perceptron to obtain feature vector; Perform mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask; The image with the defect mask is concatenated with the Gaussian noise and then input into the diffusion model together with the feature vector for training to obtain a trained diffusion model, and the trained diffusion model is used for diffusion to obtain a generated image; The generated image is pasted onto the input high-resolution defect image and then input into the variational autoencoder for decoding to obtain the original resolution defect image.

2. The workpiece defect image generation method according to claim 1, characterized in that: The step of performing data enhancement on the image containing the defect area and then inputting it into the CLIP image encoder and the multi-layer perceptron to obtain the feature vector includes: Input the image containing the defect area into the CLIP image encoder for encoding to obtain the image feature representation; The image feature vector is input into a multi-layer perceptron for nonlinear transformation to obtain the feature vector.

3. The workpiece defect image generation method according to claim 1, characterized in that: The step of performing mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask includes: The image containing the defect area is multiplied by the mask to generate an image with the defect mask.

4. The workpiece defect image generation method according to claim 1, characterized in that: Randomly selected images of non-defective areas or defective areas are used as true values ​​for training.

5. The workpiece defect image generation method according to claim 1, characterized in that: The method further comprises: If the slice containing the generated defect area does not meet the processing requirements, it is scaled.

6. A workpiece defect image generating device, characterized in that: The device comprises: A first cropping module, used for cropping the input high-resolution defect image to obtain a slice containing a generated defect area; A second cropping module is used to crop the slice containing the generated defect area to obtain a picture containing the defect area; A data enhancement module is used to perform data enhancement operations on the image containing the defect area and then input it into the CLIP image encoder and the multi-layer perceptron to obtain a feature vector; The mask shape enhancement module is used to perform mask shape enhancement processing on the image containing the defect area to generate an image with a defect mask; A training and diffusion module, which is used to concatenate the image with the defect mask and the Gaussian noise and then input it into the diffusion model together with the feature vector for training to obtain a trained diffusion model, and use the trained diffusion model for diffusion to obtain a generated image; The decoding module is used to paste the generated image onto the input high-resolution defect image and then input the image into the variational autoencoder for decoding to obtain the original resolution defect image.

7. The workpiece defect image generating device according to claim 6, characterized in that: The data augmentation module is also used to: Input the image containing the defect area into the CLIP image encoder for encoding to obtain the image feature representation; The image feature vector is input into a multi-layer perceptron for nonlinear transformation to obtain the feature vector.

8. The workpiece defect image generating device according to claim 6, characterized in that: The mask shape enhancement module is also used to: The image containing the defect area is multiplied by the mask to generate an image with the defect mask.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the workpiece defect image generating method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the workpiece defect image generating method according to any one of claims 1 to 5.

Images