An industrial defect image generation method based on an adaptive hybrid latent diffusion model

Abstract

The application provides an industrial defect image generation method based on an adaptive hybrid latent diffusion model, and relates to the field of industrial image generation, and comprises the following steps: acquiring a defect-free image, a seed defect mask and language text prompts to construct cross-modal control information, wherein the cross-modal control information comprises a random three-channel graph and a learnable text prompt; performing three-stage reasoning including a free diffusion stage, a latent space mixing stage and a decoder adaptation stage according to the cross-modal control information to obtain a final defect image. By using the above method, the fidelity of the non-defect area is greatly improved. Various defect types, shapes and positions can be supported. The defect boundary transition is smooth and the boundary is more natural through super-Gaussian function fitting. The synthesized final defect image has a significant effect on improving the training of downstream detection performance.

Description

Technical Field

[0001] This invention relates to the field of industrial image generation, specifically to a method for generating industrial defect images based on an adaptive hybrid latent diffusion model. Background Technology

[0002] In industrial manufacturing processes, obtaining defective samples is far more difficult than obtaining defect-free samples. Therefore, most existing industrial defect detection algorithms employ an anomaly detection paradigm, modeling defective samples as "outliers." However, this paradigm sacrifices the balance of data distribution, limiting the performance of discriminative algorithms.

[0003] To alleviate the problem of imbalanced samples, researchers have proposed several defect generation methods: 1. Rule-based enhancement methods (such as CutPaste and DRAEM): These methods generate artificial defects by randomly cropping and pasting image patches or mixing external textures. While computationally efficient, the generated defect patterns often deviate significantly from the actual defect distribution, potentially leading to overfitting of the detection model.

[0004] 2. GAN-based methods (such as Defect-GAN and DFMGAN): These methods generate defect images through adversarial training. However, GAN methods are prone to mode collapse, unstable training, and cannot guarantee spatial alignment between the generated defects and the mask.

[0005] 3. Diffusion-based methods (such as AnomalyDiffusion and SeaS): These have recently become a research hotspot, capable of generating relatively realistic defect patterns. However, existing methods have an inherent contradiction: unconstrained generation can improve defect diversity, but the appearance of normal regions will deviate from reality; constrained generation can maintain the fidelity of normal regions, but the defect boundary regions will appear unnatural.

[0006] Therefore, there is an urgent need for an industrial defect image generation method based on an adaptive hybrid latent diffusion model. Summary of the Invention

[0007] To overcome the existing technical problems, this invention provides a method for generating industrial defect images based on an adaptive hybrid latent diffusion model.

[0008] The present invention adopts the following technical solution.

[0009] An industrial defect image generation method based on an adaptive hybrid latent diffusion model includes the following steps: S1. Obtain defect-free images, seed defect masks, and language text prompts to construct cross-modal control information, wherein the cross-modal control information includes random three-channel images and learnable text prompts; S2. Perform three-stage reasoning, including a free diffusion stage, a latent space mixing stage, and a decoder adaptation stage, based on the cross-modal control information to obtain the final defect image; The specific steps of the three-stage reasoning include: S21. Free diffusion stage: Based on the cross-modal control information, starting from standard Gaussian noise, standard conditional reverse diffusion steps are performed within a preset number of free diffusion steps to obtain intermediate potential variables characterizing the coarse defect structure. S22, Latent Space Mixing Stage: In the remaining back-diffusion steps, the intermediate latent variables are mixed with latent codes before each denoising step to obtain the edited latent vector; S23. Decoder adaptation stage: Based on the edited latent vector, the pre-trained VQ-VAE decoder is fine-tuned online. The fine-tuned VQ-VAE decoder generates the final defect image from the edited latent vector and resets the VQ-VAE decoder.

[0010] As a further improvement of the present invention, the specific steps for acquiring defect-free images, seed defect masks, and language text prompts to construct cross-modal control information include: acquiring defect-free images and seed defect masks to construct random three-channel images, and constructing learnable text prompts based on language text prompts and preset text prompt formats.

[0011] As a further improvement of the present invention, the specific steps for obtaining a defect-free image and a seed defect mask to construct a random three-channel image include: obtaining a defect-free image and a seed defect mask, extracting the target object region of the defect-free image through a saliency detection network, and obtaining the foreground channel; The seed defect mask is randomly cropped and affine transformed, and the defect response region is generated by fitting a super-Gaussian function to obtain the defect channel. The foreground channel and the defect channel are integrated to obtain a random three-channel image.

[0012] As a further improvement to the present invention, the expression for fitting the hypergaussian function is as follows: , in, The coordinates are based on the center of the fitted ellipse as the origin. and It is the length of the semi-axis of the ellipse. It is the first adjustment factor used to control the steepness of the boundary. It is the second adjustment factor used to control the response intensity.

[0013] As a further improvement of the present invention, the preset text prompt form of the expression is as follows: , in, It is a learnable token for object categories. It is a learnable placeholder token of the defect type.

[0014] As a further improvement of the present invention, S22 further includes: a preset mask prediction module generates a clean latent code based on the defect-free image, generates VQ-VAE multi-scale features from the clean latent code through a pre-trained VQ-VAE decoder, the mask prediction module generates U-Net deep features based on the noisy latent code from each denoising process, fuses the VQ-VAE multi-scale features and the U-Net deep features, and generates a predicted defect mask.

[0015] As a further improvement of the present invention, the expression for the training loss of the mask prediction module is as follows: , in, It is the Focal Loss function. It is a rough mask prediction. It is refined mask prediction. It is the sampled true mask after downsampling. It is a real mask based on manual annotation.

[0016] As a further improvement of the present invention, the expression for the latent code mixing is: , in, It is the latent code encoding of a defect-free image. It is the defect mask after downsampling. It is element-wise multiplication. It is a logical NOT operation.

[0017] As a further improvement to the present invention, the target expression for online fine-tuning is: , , in, These are the optimal decoder parameters obtained after fine-tuning. It is to find the parameter corresponding to the minimum value. It is the edited latent vector. This is the decoded image under the current decoder. It is a defect-free image. It is element-wise multiplication. It is a logical NOT operation. It is a binary defect mask. It is a defective channel in a random three-channel graph. It is the binarization threshold. This is the initial decoded image. It is a conservative weighting coefficient.

[0018] The beneficial effects of this invention are as follows: Through the latent space mixing stage and decoder adaptation stage, this invention achieves a high degree of consistency between the non-defect regions of the final defect image and the original image, significantly improving the fidelity of the non-defect regions. This invention supports diverse defect types, shapes, and locations by constructing random three-channel images and learnable text prompts. Furthermore, the random three-channel images of this invention, fitted with a super-Gaussian function, result in smooth transitions and more natural boundaries. The YFBLDM method of this invention, on the classic public datasets MVTec-AD and MVTec-3D-AD, shows that the synthesized final defect images significantly improve downstream detection performance during training, outperforming existing state-of-the-art generation methods. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is a flowchart of the overall workflow of the YFBLDM invention and a flowchart of the workflow of the prior art; Figure 3 This is a flowchart of the construction process of the random three-channel graph of the present invention; Figure 4 These are structural diagrams of the mask prediction module and SeaS of this invention. Figure 5 This is a comparison chart of the generation effects of the present invention and different methods. Detailed Implementation

[0021] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product.

[0022] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings. The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Reference Figure 1-5 As can be seen, an industrial defect image generation method based on an adaptive hybrid latent diffusion model includes the following steps: S1. Obtain defect-free images, seed defect masks, and language text prompts to construct cross-modal control information, wherein the cross-modal control information includes random three-channel images and learnable text prompts; S2. Perform three-stage inference, including a free diffusion stage, a latent space mixing stage, and a decoder adaptation stage, based on the cross-modal control information to obtain the final defect image.

[0024] This invention, through a latent space blending stage and a decoder adaptation stage, ensures that the non-defect regions of the final defect image are highly consistent with the original image, significantly improving the fidelity of the non-defect regions. This invention supports diverse defect types, shapes, and locations by constructing random three-channel images and learningable text prompts. Furthermore, the random three-channel images of this invention, fitted with a super-Gaussian function, achieve smooth transitions and more natural boundaries. The YFBLDM method of this invention, on the classic public datasets MVTec-AD and MVTec-3D-AD, shows that the synthesized final defect images significantly improve downstream detection performance during training, outperforming existing state-of-the-art generation methods.

[0025] It should be noted that a mask prediction module can also be added in this invention. The mask prediction module performs clean latent code + multi-scale feature fusion to generate a high-precision pixel-level mask. This is because after the algorithm obtains the final defect image, it should also be able to obtain the defect mask corresponding to the final defect image to facilitate subsequent training.

[0026] As a further improvement of the present invention, the specific steps for acquiring defect-free images, seed defect masks, and language text prompts to construct cross-modal control information include: acquiring defect-free images and seed defect masks to construct random three-channel images, and constructing learnable text prompts based on language text prompts and preset text prompt formats.

[0027] Reference Figure 3 As can be seen, the specific steps for constructing a random three-channel image from a defect-free image and a seed defect mask include: acquiring a defect-free image and a seed defect mask, extracting the target object region of the defect-free image through a saliency detection network, and obtaining the foreground channel;

[0028] The seed defect mask is randomly cropped and affine transformed, and the defect response region is generated by fitting a super-Gaussian function to obtain the defect channel. The foreground channel and the defect channel are integrated to obtain a random three-channel image.

[0029] The saliency detection network, U²-Net, uses an affine transformation with a rotation angle of [0°, 360°] and a scaling factor of [0.85, 1.15]. It is located in... Figure 3 The top line shows the process of building a defect channel, resulting in a defect channel. ,lie in Figure 3 The bottom line shows the process of building the foreground channel, resulting in the foreground channel. Integrate defect channels and forward channel Obtain a random three-channel image ,when , All values ​​are 0, representing the background. , A value of 1 or 0 indicates normal operation. , The value is 1, where 1 represents a defect.

[0030] Specifically, the value of each pixel in the defect channel is calculated using the following super-Gaussian fitting formula: , in, The coordinates are based on the center of the fitted ellipse as the origin. and It is the length of the semi-axis of the ellipse. It is the first adjustment factor used to control the steepness of the boundary. It is the second adjustment factor used to control the response intensity.

[0031] Specifically, The value is 3. The value is 0.8, which has shown the best performance in actual tests on the classic public dataset MVTec AD in this field.

[0032] As a further improvement of the present invention, the preset text prompt form of the expression is as follows: , in, It is a learnable token for object categories. It is a learnable placeholder token of the defect type.

[0033] The principle behind this expression is that by assigning four learnable placeholder tokens to each type of defect, it can effectively distinguish between different types of defects. It is a fixed text skeleton used to mimic the natural language description of "an object with a certain defect," with the aim of adapting to the input format of the CLIP text encoder and reusing the semantic understanding capabilities of the pre-trained model. The embedding vector of this learnable placeholder token is initialized in the embedding table of the CLIP text encoder and optimized during training, while the CLIP Transformer layer remains frozen.

[0034] The specific steps of the three-stage reasoning include: S21. Free diffusion stage: Based on the cross-modal control information, starting from standard Gaussian noise, standard conditional reverse diffusion steps are performed within a preset number of free diffusion steps to obtain intermediate potential variables characterizing the coarse defect structure. S22, Latent Space Mixing Stage: In the remaining back-diffusion steps, the intermediate latent variables are mixed with latent codes before each denoising step to obtain the edited latent vector; S23. Decoder adaptation stage: Based on the edited latent vector, the pre-trained VQ-VAE decoder is fine-tuned online. The fine-tuned VQ-VAE decoder generates the final defect image from the edited latent vector and resets the VQ-VAE decoder.

[0035] The specific steps of S21 include: from standard Gaussian noise Begin by performing the standard condition backdiffusion procedure: , This stage does not involve content editing; it only allows the model to establish a rough defect structure. It is a learnable text prompt. It is the encoding result of a random three-channel image. It is the noisy latent code at time step t. The preset number of free diffusion steps can be adjusted arbitrarily according to actual needs.

[0036] As a further improvement of the present invention, the expression for the latent code mixing is: , in, It is the latent code encoding of a defect-free image. It is the defect mask after downsampling. It is element-wise multiplication. It is a logical NOT operation.

[0037] This operation ensures that defective areas retain the generated result, while non-defective areas are completely restored to the latent code of a defect-free image, thus maintaining high background fidelity.

[0038] Due to the pre-trained VQ-VAE decoder This may lead to color shifts or texture inconsistencies in non-defective areas. This invention fine-tunes the decoder online for each generated sample. Initial decoding... Used to anchor the appearance of defects, while no-defect images This is used to constrain normal regions. A binarized defect mask is used. We update the decoder using the following formula. step: , , in, These are the optimal decoder parameters obtained after fine-tuning. It is to find the parameter corresponding to the minimum value. It is the edited latent vector. This is the decoded image under the current decoder. It is a defect-free image. It is element-wise multiplication. It is a logical NOT operation. It is a binary defect mask. It is a defective channel in a random three-channel graph. It is the binarization threshold. This is the initial decoded image. It is a conservative weighting coefficient.

[0039] To further explain, in calculating the optimal decoder parameters In the formula, the first term is used to restore the non-defective region to a defect-free image. The second step conservatively keeps the defective region close to its initial decoding result. After adaptation, This serves as the final image. After fine-tuning, the optimized decoder is used to generate the final defect image, and then the VQ-VAE decoder parameters are reset without being shared across samples.

[0040] Reference Figure 4 S22 further includes: a preset mask prediction module generates a clean latent code based on the defect-free image, generates VQ-VAE multi-scale features from the clean latent code through a pre-trained VQ-VAE decoder, the mask prediction module generates U-Net deep features based on the noisy latent code from each denoising process, fuses the VQ-VAE multi-scale features and U-Net deep features, and generates a predicted defect mask. Figure 4 The upper area also shows a flowchart of the existing technology SeaS algorithm for generating predicted defect masks.

[0041] Compared to traditional methods that use "one-step denoising estimation" Unlike other inventions, this invention uses clean latent codes. The input to the VQ-VAE decoder extracts multi-scale features for mask boundary refinement. This avoids feature degradation caused by estimation errors, generating more accurate defect boundaries. It should be noted that the VQ-VAE decoder used here is not a fine-tuned version.

[0042] As a further improvement of the present invention, the expression for the training loss of the mask prediction module is as follows: , in, It is the Focal Loss function. It is a rough mask prediction. It is refined mask prediction. It is the sampled true mask after downsampling. It is a real mask based on manual annotation.

[0043] The true mask is a manually annotated pixel-level defect mask. The sampled true mask is obtained by downsampling the true mask and is used to match the size of the coarse prediction. The coarse mask prediction is in the latent space and is used to quickly locate the defect region. The refined mask prediction is an accurate mask output after upsampling and high-dimensional feature optimization. It is also the output of the module and corresponds to the pixel-level mask required by the downstream anomaly detection model.

[0044] Furthermore, the training loss function of the model performing the back-diffusion step in this invention is formed by jointly using the standard diffusion loss and the attention alignment loss, expressed as: , in, It is the standard noise prediction diffusion loss. It is an attention alignment loss that aligns the cross-attention map with the random three-channel map. It is a balancing weight.

[0045] Since the standard noise prediction diffusion loss is a common training target for denoising diffusion, this invention will not elaborate further.

[0046] Specifically, the attention alignment loss is defined as: , in, It is the selected set of cross-attention layers. Attention graph (head average) of learnable placeholder tokens for defect types. Attention graph of learnable tokens for object categories Defect mask and foreground mask after downsampling and normalization.

[0047] Experiments have verified that, on the MVTec AD dataset, the DeSTSeg detection model, trained using defect samples generated in this invention, achieves the following results: The average detection metrics (pAUC / AP / PRO / iAUC) improved by +7.81 to +11.18 percentage points; The IAP90 index improved by 31.41 percentage points. The image fidelity index (KID) reached 1.732 × 10⁻², outperforming all comparison methods. (See reference...) Figure 5The comparison chart shows the results. The first row corresponds to the real defect image, the effect image of the comparison method, and the effect image of the present invention in the last row. The algorithm models of the comparison methods, from top to bottom, correspond to the real image, AnoDiff (AAAI2024), DualAnoDiff (CVPR2025), FAST (NeurIPS2025), SeaS (ICCV2025), MAGIC (CVPR2026), and the image of the present invention.

[0048] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for generating industrial defect images based on an adaptive hybrid latent diffusion model, characterized in that, Includes the following steps: S1. Obtain defect-free images, seed defect masks, and language text prompts to construct cross-modal control information, wherein the cross-modal control information includes random three-channel images and learnable text prompts; S2. Perform three-stage reasoning, including a free diffusion stage, a latent space mixing stage, and a decoder adaptation stage, based on the cross-modal control information to obtain the final defect image; The specific steps of the three-stage reasoning include: S21. Free diffusion stage: Based on the cross-modal control information, starting from standard Gaussian noise, standard conditional reverse diffusion steps are performed within a preset number of free diffusion steps to obtain intermediate potential variables characterizing the coarse defect structure. S22, Latent Space Mixing Stage: In the remaining back-diffusion steps, the intermediate latent variables are mixed with latent codes before each denoising step to obtain the edited latent vector; S23. Decoder adaptation stage: Based on the edited latent vector, the pre-trained VQ-VAE decoder is fine-tuned online. The fine-tuned VQ-VAE decoder generates the final defect image from the edited latent vector and resets the VQ-VAE decoder.

2. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 1, characterized in that, The specific steps for acquiring defect-free images, seed defect masks, and language text prompts to construct cross-modal control information include: acquiring defect-free images and seed defect masks to construct random three-channel images, and constructing learnable text prompts based on language text prompts and preset text prompt formats.

3. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 2, characterized in that, The specific steps for constructing a random three-channel image from a defect-free image and a seed defect mask include: acquiring a defect-free image and a seed defect mask, extracting the target object region of the defect-free image through a saliency detection network, and obtaining the foreground channel; The seed defect mask is randomly cropped and affine transformed, and the defect response region is generated by fitting a super-Gaussian function to obtain the defect channel. The foreground channel and the defect channel are integrated to obtain a random three-channel image.

4. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 3, characterized in that, The expression for fitting the superGaussian function: ， in, The coordinates are based on the center of the fitted ellipse as the origin. and It is the length of the semi-axis of the ellipse. It is the first adjustment factor used to control the steepness of the boundary. It is the second adjustment factor used to control the response intensity.

5. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 2, characterized in that, Preset text prompt expressions: ， in, It is a learnable token for object categories. It is a learnable placeholder token of the defect type.

6. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 1, characterized in that, S22 further includes: a preset mask prediction module generates a clean latent code based on the defect-free image, generates VQ-VAE multi-scale features from the clean latent code through a pre-trained VQ-VAE decoder, the mask prediction module generates U-Net deep features based on the noisy latent code from each denoising process, fuses the VQ-VAE multi-scale features and U-Net deep features, and generates a predicted defect mask.

7. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 6, characterized in that, The expression for the training loss of the mask prediction module is as follows: ， in, It is the Focal Loss function. It is a rough mask prediction. It is refined mask prediction. It is the sampled true mask after downsampling. It is a real mask based on manual annotation.

8. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 1, characterized in that, The expression for the latent code mixing: ， in, It is the latent code encoding of a defect-free image. It is the defect mask after downsampling. It is element-wise multiplication. It is a logical NOT operation.

9. The method for generating industrial defect images based on an adaptive hybrid latent diffusion model according to claim 1, characterized in that, The target expression for online fine-tuning: ， ， in, These are the optimal decoder parameters obtained after fine-tuning. It is to find the parameter corresponding to the minimum value. It is the edited latent vector. This is the decoded image under the current decoder. It is a defect-free image. It is element-wise multiplication. It is a logical NOT operation. It is a binary defect mask. It is a defective channel in a random three-channel graph. It is the binarization threshold. This is the initial decoded image. It is a conservative weighting coefficient.

Images