Industrial defect detection and data enhancement method based on lightweight diffusion model adapter

Abstract

The invention relates to an industrial defect detection and data enhancement method based on a lightweight diffusion model adapter. The method comprises the following steps: collecting and preprocessing industrial scene data; generating a high-fidelity image by using a pre-trained denoising diffusion model, initializing the weight of the model, and keeping freezing in the reasoning process; the global scene features are embedded into an encoder of a denoising diffusion model, the overall global structure of the generated image is controlled, local control joint features fused by local defects and position information are embedded into a decoder of the denoising diffusion model, and the global structure and local details of the image are generated; task perception loss is combined with basic loss of a denoising diffusion model to serve as a comprehensive loss function, and a generation result is optimized; according to the method, the hierarchical prediction network is adopted, the significant features in the industrial scene are firstly positioned, then refined data enhancement is performed, the reasoning speed is increased through lightweight design, efficient generation and enhancement of industrial defect data are realized, and powerful data support is provided for downstream detection tasks.

Description

Technical Field

[0001] The present invention relates to the technical fields of machine learning and computer vision, and particularly to an industrial defect detection and data augmentation method based on a lightweight diffusion model adapter. Background Art

[0002] In industrial inspection scenarios, the performance of deep learning models depends to a large extent on the diversity and quality of data. However, due to the complex and diverse industrial scenarios, defective samples are scarce and difficult to obtain, and traditional data augmentation methods cannot meet the requirements of high-quality, domain-specific data generation.

[0003] Existing industrial image generation methods are mainly based on generative adversarial networks (GANs) or variational autoencoders (VAEs), but these methods have the following problems:

[0004] 1. Mode collapse: GANs are prone to generating samples lacking diversity;

[0005] 2. Insufficient visual fidelity: Industrial images generated by existing methods are difficult to fully simulate real scenarios.

[0006] In recent years, diffusion models have demonstrated superior performance in image generation tasks, and they can gradually denoise to generate high-fidelity images. However, directly applying diffusion models to industrial scenarios requires a large amount of computing resources and training data, and lacks domain-specific generation control capabilities. Therefore, there is an urgent need for a lightweight and adaptable detection system to efficiently handle defect detection tasks from different industrial fields. Summary of the Invention

[0007] Aiming at the deficiencies of the prior art, the present invention provides an industrial defect detection and data augmentation method based on a lightweight diffusion model adapter, which solves the problems that directly applying diffusion models to industrial scenarios by traditional methods requires a large amount of computing resources and training data and lacks domain-specific generation control capabilities. On the premise that industrial defect samples are difficult to collect and there are many types of defects, compared with traditional methods based on generative adversarial networks (GANs) or variational autoencoders (VAEs), the present invention can better maintain the high quality and diversity of generated images, avoid the mode collapse problem in the training process, and at the same time improve the inference speed through lightweight design, realizing the efficient generation and augmentation of industrial defect data and providing strong data support for downstream detection tasks.

[0008] To solve the above technical problems, the present invention provides the following technical solution: An industrial defect detection and data augmentation method based on a lightweight diffusion model adapter, comprising the following steps:

[0009] S1. Collection and preprocessing of industrial scene data;

[0010] S2. Generate high-fidelity images using a pre-trained denoising diffusion model, initialize the model weights and keep them frozen during inference;

[0011] S3. Design a lightweight perception information branch SIA to generate global scene features F through the scene control adapter combined with the self-attention mechanism sc ;

[0012] S4. Design a lightweight local and position information branch LLIA and use the positioning Dino to extract the local features of industrial defects H 4,lc , use CLIP encoder to generate information H describing the defect location t Finally, the local defect characteristics and position information are integrated to generate the local control signal F lc ;

[0013] S5. The global scene feature F sc Embedded in the encoder of the denoising diffusion model, it controls the overall global structure of the generated image and transforms the local defects F lc With location information H t Fusion of local control joint features F control Embedded in the decoder of the denoising diffusion model, it generates the global structure and local details of the image;

[0014] S6. For the trained lightweight diffusion model, the task-aware loss is combined with the basic loss of the denoising diffusion model as a comprehensive loss function to optimize the generated results and enhance the performance of downstream detection and segmentation tasks.

[0015] Furthermore, S11 collects images from various industrial scenes, including defect images of steel plate surfaces, plastic parts, and electronic components of different materials. The images cover a variety of industrial defects such as scratches, cracks, bumps, pits, and corrosion.

[0016] S12, performing denoising, normalization, and size adjustment standardization processing on the collected original image to improve the consistency and adaptability of the data, adjusting the image to a size suitable for the diffusion model input, and performing color standardization processing to reduce the deviation of different image sources;

[0017] S13. Label the defect image and generate a true label mask for calculation and verification of perception loss in subsequent tasks.

[0018] Furthermore, the key to the S21,denoising diffusion model is the U-Net structure, which consists of an encoder and a decoder, and can gradually restore a high-fidelity image from a noisy image through convolution and deconvolution operations;

[0019] S22. The forward diffusion process transforms the original image x0 into an image with gradually increasing noise x, t and finally becomes a pure noise image x T at time step T;

[0020] S23. The reverse denoising process starts from the pure noise image x T , and the diffusion model gradually removes the noise to recover the denoised image x t-1 ;

[0021] S24. During the inference process, to improve the inference efficiency and save computing resources, the weights of the denoising diffusion model are frozen, that is, no longer performing backpropagation and gradient update.

[0022] Further, step S22 specifically includes:

[0023] At each time step t, noise is added through the conditional probability distribution q(x t |x t-1 ), and the formula is:

[0024]

[0025] where x0 represents the original image, x t represents the image at time step t, represents the cumulative amount of image information retained at time step t, α t represents the intensity of the noise added at each time step, α t = 1 - β t , β t represents the intensity of the noise added at each time step, and I is the identity matrix;

[0026] For any time step t, the noise image x t can be directly generated from the original image x0:

[0027]

[0028] where represents the noise sampled from the standard normal distribution.

[0029] Further, step S23 specifically includes:

[0030] At time step t, the calculation formula for the conditional probability distribution of the reverse process is:

[0031]

[0032] where Σ θ (x t , t) represents the variance of the noise, usually a constant in the model or calculated according to the time step t, μθ (x t ,t) represents the mean value predicted by the denoising network U-Net, which is predicted by the following formula:

[0033]

[0034] Among them, ∈ θ (x t ,t) represents the model’s predicted value for the current noise;

[0035] At each time step t, the noise predicted by the model is extracted from the current image x t Remove it to get the denoised image x t-1 , this process gradually removes the noise in the image, thereby restoring the original content of the image. The model uses the following formula to denoise and restore the image:

[0036]

[0037] Among them, x t Represents the image at the current time step, ∈ θ (x t ,t) represents the noise predicted by the model, σ t Represents the predefined noise standard deviation, usually taken as represents noise sampled from a standard Gaussian distribution.

[0038] Furthermore, in step S3, the specific process includes the following steps:

[0039] S31. Select 4 feature extraction blocks and 4 double downsampling blocks to build a scene control adapter. The feature extraction block contains a feature enhancement block and a residual block to extract the global features of the industrial scene layer by layer, and use the Vision Mamba mechanism to extract the global hidden scene features H. 4,sc , enhance the ability to capture the global characteristics of industrial scenarios;

[0040] S32, extracted global hidden scene feature H 4,sc It is further processed by a 1×1 convolutional layer to reduce the dimension and adjust the features to generate the final global feature F sc .

[0041] Furthermore, in step S4, the specific process includes the following steps:

[0042] S41. Use the localization Dino module to extract local features of scratches and pits from the public industrial defect dataset. The defect features are further refined through a specific Attention mechanism, and the local hidden defect features H are extracted through the pre-trained model. 4,lc ;

[0043] S42. Define the text position description and type of the defect. Tip: There is a scratch on the upper left corner of the workpiece, or there is a bulge in the center area of the workpiece, or there is a scratch on the surface of the cylinder. Use the CLIP text encoder to process the position text description and extract the defect position information H. t ;

[0044] S42, locally hide the defect feature H 4,lc and defect location information H t The final LLIA branch local and position control signal F is generated by concatenation and 1×1 convolution. lc .

[0045] Furthermore, in step S5, the specific process includes the following steps:

[0046] S51, global scene feature F extracted from lightweight perception information branch SIA sc Embedded into each layer encoder of the diffusion model, it is used to control the overall layout and background characteristics of the generated image. The cross-attention mechanism calculates the attention weight according to the dot product of the query matrix Q and the key matrix K to convert the final global scene feature F sc and the feature F of the current layer of the encoder en,i Fusion is performed to obtain the output feature F' en,i , the specific calculation formula is as follows:

[0047] F′ en,i =Atten(Q,K,V)

[0048] Where Q = F en,i W q : The query matrix Q is input by the encoder layer i feature F en,i and the weight matrix W q Generate, K = F sc W k :The key matrix K is composed of the global scene features F sc and the weight matrix W k Generate, V = F sc W v :The value matrix V is composed of the global scene features F sc and the weight matrix W v Generate, F' en,i Represents the output features after fusion, which is used in the subsequent decoding stage;

[0049] S52, the final local defect feature F extracted by the lightweight local and position information branch LLIA lc and location information H t is fused into the local control joint feature F control :

[0050] Fcontrol = K 1×1 *(F lc ⊕ H t )

[0051] where ⊕ represents the concatenation operation, and K 1×1 represents the convolutional kernel for dimensionality compression.

[0052] At each layer of the denoising diffusion model decoder, the local control joint feature F control is embedded into the decoded feature F de,i of the current layer, and the calculation formula is:

[0053] F' de,i = Atten(Q, K s , V s ) ⊕ Atten(Q, K l , V l )

[0054] where Q = F de,i · W q means that the query matrix Q is generated by the input feature F de,i of the i-th layer of the decoder, represents the key matrix of the global scene feature, represents the value matrix of the global scene feature, represents the key matrix of the local control joint feature, represents the value matrix of the local control joint feature; ⊕ represents the element-wise addition operation, and F' de,i represents the fused feature output by the decoder.

[0055] Furthermore, step S6 specifically includes:

[0056] Combining the base loss L θ of the denoising diffusion model and the task-aware loss L task as the comprehensive loss function L, and the comprehensive loss function can be expressed as:

[0057] L = L θ + λL task ;

[0058] where λ represents the weight of the task-aware loss, controlling the relative importance of the two loss functions, and L θ represents the diffusion loss function, which is optimized by calculating the mean squared error MSE between the noise ∈ θ (x t , t) predicted by the model and the true noise ∈. The training objective of the diffusion model is to enable the model to accurately predict the noise added to the image at each time step, which can be specifically expressed as:

[0059]

[0060] Among them, x0 represents the original image, and x t represents the image after adding noise at the t-th step, ∈ represents the actually added noise, ∈ θ (x t , t) represents the noise predicted by the model, represents the expected value, which calculates the average error of all time steps and noise samples;

[0061] L task represents the task-aware loss, which specifically combines multiple task-related loss functions and is dedicated to ensuring that the generated images can effectively support downstream tasks such as defect detection and image segmentation. The task-aware loss L task mainly uses the weighted sum method to fuse the Dice loss L D and the boundary loss L B , and can be specifically expressed as:

[0062] L task = αL D +(1 - α)L B

[0063] Among them, α is a hyperparameter representing the weight allocation between the Dice loss and the boundary loss, which is usually dynamically adjusted as the training progresses; L D represents the Dice loss, which measures the similarity of the target region. This loss function calculates the Dice coefficient between the generated image g i and the ground truth mask p i , and optimizes the model according to the value of the coefficient; The Dice coefficient is calculated by comparing the overlap degree of the predicted mask and the ground truth mask:

[0064]

[0065] Among them, p i represents the predicted value of the i-th pixel in the generated image, and g i represents the ground truth value of the i-th pixel in the target image, and Σ i represents the summation over all pixels;

[0066] L B is the boundary loss used to optimize the boundary details of the defect targets in the generated image, especially the contour or edge part of the targets in the image. This loss emphasizes the boundary region of the targets rather than the overall region; The boundary loss L B can be calculated by the following formula:

[0067]

[0068] Among them, C and N represent the number of classes and pixels respectively, pi represents the predicted value of the i-th pixel in the generated image, g i represents the true value of the i-th pixel in the target image, δ(g i ) represents a weight function that assigns larger weights to the boundary pixels of the target, ensuring stronger optimization of the boundary part.

[0069] By means of the above technical solution, the present invention provides an industrial defect detection and data enhancement method based on a lightweight diffusion model adapter, which has at least the following beneficial effects:

[0070] Compared with traditional methods based on generative adversarial networks (GANs) or variational autoencoders (VAEs), the present invention can better maintain the high quality and diversity of the generated images, avoid the mode collapse problem during training, and at the same time improve the inference speed through lightweight design, realizing the efficient generation and enhancement of industrial defect data and providing strong data support for downstream detection tasks.

[0071] The present invention improves the traditional industrial defect detection model. By decoupling the classification and localization tasks and adopting a hierarchical method, it can accurately detect defects with high similarity and low pixels. The Real-Adapter adapter uses the SIA and LLIA branches to extract key features and combines them with the denoising U-Net to effectively enhance the data and improve the detection accuracy. This method is applicable to the defect detection of various low-pixel targets and has good generalization and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0073] Figure 1 is a schematic flowchart of the industrial defect detection proposed by the present invention;

[0074] Figure 2 is a schematic overall structure diagram of the lightweight diffusion model adapter Real-Adapter proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0075] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments. Thereby, a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects can be obtained and implemented accordingly.

[0076] Those of ordinary skill in the art can understand that all or part of the steps in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable program code.

[0077] Please refer to Figure 1 - Figure 2 , which shows a specific implementation manner of this embodiment. In this embodiment, by improving the traditional industrial defect detection model, through decoupling the classification and localization tasks, and adopting a hierarchical method, defects with high similarity and low pixels can be accurately detected. The Real-Adapter adapter uses the SIA and LLIA branches to extract key features and combines them with the denoising U-Net to effectively enhance the data and improve the detection accuracy. This method is applicable to the defect detection of various low-pixel targets and has good generalization and efficiency.

[0078] Please refer to Figure 1 , this embodiment proposes an industrial defect detection and data enhancement method based on a lightweight diffusion model adapter, and this method includes the following steps:

[0079] S1. Collection and preprocessing of industrial scenario data;

[0080] As a preferred implementation manner of step S1, the specific process includes the following steps:

[0081] S11. Collect images from various industrial scenarios, including defect images of different materials such as the surface of steel plates, plastic parts, and electronic components, and this image covers various industrial defects such as scratches, cracks, protrusions, pits, and corrosion;

[0082] S12. Perform standardization processing such as denoising, normalization, and size adjustment on the collected original images to improve the consistency and adaptability of the data, adjust the images to a size suitable for input to the diffusion model, and perform color standardization processing to reduce the deviation of different image sources;

[0083] S13. Label the defect images to generate a true label mask for the calculation and verification of the subsequent task perception loss.

[0084] S2. Use a pre-trained denoising diffusion model to generate high-fidelity images, initialize the model weights and keep them frozen during the inference process; as Figure 2 shown, this patent takes the industrial scenario defect detection as an example to demonstrate the method proposed by this patent. Among them, industrial defect samples are difficult to collect and there are many types of defects. Therefore, it conforms to the application scenario of this patent;

[0085] As a preferred embodiment of step S2, the specific process includes the following steps:

[0086] S21. The key of the denoising diffusion model is the U-Net structure, which includes an encoder and a decoder, and can gradually recover a high-fidelity image from a noisy image through convolutional and transposed convolutional operations;

[0087] S22. The forward diffusion process transforms the original image x0 into an image with gradually increasing noise x t , and finally becomes a pure noise image x T at time step T;

[0088] Step S22 specifically includes:

[0089] At each time step t, noise is added through the conditional probability distribution q(x t |x t-1 ), and the formula is:

[0090]

[0091] where x0 represents the original image, x t represents the image at time step t, represents the cumulative amount of image information retained at time step t, α t represents the intensity of the noise added at each time step, ɑ t = 1 - β t , β t represents the intensity of the noise added at each time step, and I is the identity matrix;

[0092] For any time step t, the noise image x t can be directly generated from the original image x0:

[0093]

[0094] where, represents the noise sampled from the standard normal distribution.

[0095] S23. The reverse denoising process starts from the pure noise image x T , and the diffusion model gradually removes the noise to obtain the denoised and restored image x t-1 , and this process gradually performs image restoration by reverse predicting the noise and using the denoising estimate;

[0096] Step S23 specifically includes:

[0097] At time step t, the conditional probability distribution calculation formula for the reverse process is:

[0098]

[0099] Among them, Σ θ (x t , t) represents the variance of the noise, which is usually a constant in the model or calculated according to the time step t. μ θ (x t , t) represents the mean predicted by the denoising network U-Net and is predicted by the following formula:

[0100]

[0101] Among them, ∈ θ (x t , t) represents the predicted value of the model for the current noise;

[0102] At each time step t, the noise predicted by the model is removed from the current image x t to obtain the denoised image x t-1 . This process gradually removes the noise in the image, thereby restoring the original content of the image. The model denoises and restores the image through the following formula:

[0103]

[0104] Among them, x t represents the image at the current time step, ∈ θ (x t , t) represents the noise predicted by the model, σ t represents the predefined standard deviation of the noise, usually taking the value of represents the noise sampled from the standard Gaussian distribution.

[0105] S24. During the inference process, in order to improve the inference efficiency and save computing resources, the weights of the denoising diffusion model are frozen, that is, no more backpropagation and gradient update are performed; freezing the weights enables the model to directly generate images efficiently without retraining. During this process, the weights of the model remain unchanged, and the inference process is simply to generate images by gradually denoising. The advantage of freezing the model weights is to reduce the computational burden and be able to quickly generate high-quality images with limited computing resources.

[0106] S3. Design a lightweight perception information branch SIA to generate the global scene feature F through the scene control adapter combined with the self-attention mechanism sc ;

[0107] As a preferred implementation manner of step S3, the specific process includes the following steps:

[0108] S31. Select 4 feature extraction blocks and 4 double downsampling blocks to construct a scene control adapter. Four blocks are a reasonable choice that can achieve a good balance between feature extraction ability and computational efficiency. The feature extraction block contains a feature enhancement block and a residual block, which are used to extract the global features of the industrial scene layer by layer, and the VisionMamba mechanism is used to extract the global hidden scene feature H 4,sc , enhancing the ability to capture the global characteristics of the industrial scene; each feature extraction block further strengthens the feature expression ability through a residual module, and the extracted features can be recursively expressed as:

[0109]

[0110] Among them, represents the hidden scene feature map of the i-th layer, and its size is where w and h represent the width and height of the input image respectively, and φ res represents the residual module, and φ ds represents the double downsampling operation. The VisionMamba mechanism further captures the global characteristics through multi-scale feature extraction;

[0111] S32. The extracted global hidden scene feature H 4,sc is further processed by a 1×1 convolutional layer to reduce the dimension and adjust the features, generating the final global feature f sc , making it suitable for subsequent fusion with other branches (lightweight local and location information branch LLIA); generating the global feature f of the final SIA branch sc The formula is as follows:

[0112] F sc = K 1×1 * H 4,sc

[0113] Among them, K 1×1 represents the convolutional kernel, and * represents the convolutional operation.

[0114] S4. Design a lightweight local and location information branch LLIA, use the localization Dino to extract the local feature H of the industrial defect 4,lc , use the CLIP encoder to generate the description of the defect location information H t , and finally fuse the local defect feature and the location information to generate the local control signal F lc ;

[0115] As a preferred implementation manner of step S4, the specific process includes the following steps:

[0116] S41. Use the Grounding Dino module to extract local features of defect areas such as scratches and pits from publicly available industrial defect datasets (such as NEU-DET and GC10-DET). The defect features are further refined through a specific Attention mechanism, and local hidden defect features H are extracted through a pre-trained model. 4,lc ; Among them, to extract the detailed information of local industrial defects, the calculation formula is as follows:

[0117]

[0118] Among them, represents the hidden local defect feature of the i-th layer, and φ atten represents the attention mechanism, which is used to optimize the local feature expression, and φ res represents the residual module, and φ ds represents the double downsampling operation;

[0119] S42. Define text location prompts that describe the defect location and type: (such as "There is a scratch in the upper left corner of the workpiece", "There is a protrusion in the central area of the workpiece", or "There is a scratch on the surface of the cylinder"). Use the CLIP text encoder to process the location text description and extract the defect location information H t , and the formula is as follows:

[0120] H t =CLIP(text)

[0121] Among them, H t represents the extracted location information;

[0122] S42. Fuse the local hidden defect feature H 4,lc and the defect location information H t through concatenation and 1×1 convolution to generate the final local and position control signal F of the LLIA branch lc :

[0123] F lc =K 1×1 *(H 4,lc ⊕H t )

[0124] Among them, ⊕ represents the feature concatenation operation.

[0125] S5. Embed the global scene feature F sc into the encoder of the denoising diffusion model to control the overall global structure of the generated image. Embed the local control joint feature F lc obtained by fusing the local defect F t and the position information H control into the decoder of the denoising diffusion model to generate the global structure and local details of the image;

[0126] As a preferred embodiment of step S5, the specific process includes the following steps:

[0127] S51. The global scene feature F extracted from the lightweight perception information branch SIA sc is embedded into each layer of the encoder of the diffusion model to control the overall layout and background characteristics of the generated image. The cross-attention mechanism calculates the attention weights according to the dot product of the query matrix Q and the key matrix K, and thus combines the final global scene feature F sc with the feature F en,i of the current layer of the encoder to obtain the output feature F'. en,i The specific calculation formula is as follows:

[0128] F' en,i = Atten(Q, K, V) +

[0129] where Q = F en,i ·W q : The query matrix Q is generated by the input feature F en,i of the i-th layer of the encoder and the weight matrix W q ; K = F sc ·W k : The key matrix K is generated by the global scene feature F sc and the weight matrix W k ; V = F sc ·W v : The value matrix V is generated by the global scene feature F sc and the weight matrix W v ; F' en,i represents the output feature after fusion and is used as the input for the subsequent decoding stage;

[0130] The attention weights are calculated by the dot product of the query and the key, and the calculation formula is as follows:

[0131]

[0132] where d k represents the dimension of the key and is used to normalize the dot product result. α represents the weight, which reflects the influence degree of the global scene feature on different positions of the encoder. By embedding the global feature into each layer of the encoder, it is ensured that the overall layout and background of the generated image are consistent with the target industrial scene;

[0133] S52. The final local defect feature F lc extracted from the lightweight local and position information branch LLIA and the position information H t are fused into the local control joint feature F control :

[0134] Fcontrol = K 1×1 *(F lc ⊕ H t )

[0135] where ⊕ represents the concatenation operation, and K 1×1 represents the convolutional kernel for dimensionality compression.

[0136] At each layer of the denoising diffusion model decoder, the local control joint feature F control is embedded into the decoded feature F de,i of the current layer, and the calculation formula is:

[0137] F' de,i = Atten(Q, K s , V s ) ⊕ Atten(Q, K l , V l )

[0138] where Q = F de,i · W q means that the query matrix Q is generated by the input feature F de,i of the i-th layer of the decoder. The input feature F de,i of the i-th layer of the decoder is the output feature F' en,i after fusion by the encoder. represents the key matrix of the global scene feature, represents the value matrix of the global scene feature, represents the key matrix of the local control joint feature, represents the value matrix of the local control joint feature; ⊕ represents the element-wise addition operation, and F' de,i represents the fused feature output by the decoder.

[0139] In this embodiment, the encoding stage corresponds to the forward diffusion process that transforms the original image x0 into an image x t with gradually increasing noise, and finally becomes a pure noise image x T at time step T; the decoding stage corresponds to the reverse denoising process that starts from the pure noise image x T and the diffusion model gradually removes the noise to recover the denoised and restored image x t-1; To avoid the mutual interference between global features and local features, a decoupling strategy is adopted in the cross-attention mechanism to process global and local control signals separately. The local defect features and location information are accurately embedded into the decoder through the attention mechanism, ensuring that the generated defect images are consistent with the target features in terms of local details and locations. Through the effective fusion of global scene features and local detail features, the consistency of the generated images in terms of global structure and local details is ensured, and finally, high-quality images that meet the requirements of industrial defect detection are generated.

[0140] S6. For the trained Real-Adapter lightweight diffusion model, the task-aware loss is combined with the basic loss of the denoising diffusion model as the comprehensive loss function to optimize the generation result and enhance the performance of downstream detection and segmentation tasks.

[0141] The basic loss L of the denoising diffusion model θ and the task-aware loss L are combined task as the comprehensive loss function L, and the comprehensive loss function can be expressed as:

[0142] L = L θ + λL task ;

[0143] where λ represents the weight of the task-aware loss, controlling the relative importance of the two loss functions, and L θ represents the diffusion loss function, which is optimized by calculating the mean squared error MSE between the noise ∈ θ (x t , t) predicted by the model and the real noise ∈. The training objective of the diffusion model is to enable the model to accurately predict the noise added to the image at each time step, which can be specifically expressed as:

[0144]

[0145] where x0 represents the original image, x t represents the image after adding noise at the t-th step, ∈ represents the actually added noise, and ∈ θ (x t , t) represents the noise predicted by the model, represents the expected value, and the calculated value is the average error of all time steps and noise samples;

[0146] L task represents the task-aware loss, which specifically combines multiple task-related loss functions and is dedicated to ensuring that the generated images can effectively support downstream tasks such as defect detection and image segmentation. The task-aware loss L task mainly uses the weighted sum method to fuse the Dice loss L D and the boundary loss L B, which can be specifically expressed as:

[0147] L task = αL D +(1 - α)L B

[0148] where α is a hyperparameter representing the weight distribution between the Dice loss and the boundary loss, which is usually dynamically adjusted as the training progresses; L D represents the Dice loss, which measures the similarity of the target region. This loss function calculates the Dice coefficient between the generated image g i and the ground truth mask p i , and optimizes the model according to the value of the coefficient; The Dice coefficient is calculated by comparing the overlap degree of the predicted mask and the ground truth mask:

[0149]

[0150] where p i represents the predicted value of the i-th pixel in the generated image, g i represents the ground truth value of the i-th pixel in the target image, and ∑ i represents the summation over all pixels;

[0151] L B is the boundary loss used to optimize the boundary details of the defective targets in the generated image, especially the contour or edge part of the targets in the image. This loss emphasizes the boundary region of the targets rather than the overall region; The boundary loss L B can be calculated by the following formula:

[0152]

[0153] where C and N represent the number of classes and pixels respectively, p i represents the predicted value of the i-th pixel in the generated image, g i represents the ground truth value of the i-th pixel in the target image, and δ(g i ) represents a weight function that assigns a larger weight to the boundary pixels of the target, ensuring stronger optimization of the boundary part.

[0154] In this embodiment, for the Real-Adapter lightweight diffusion model, the basic loss of the diffusion model and the task-aware loss are combined as the comprehensive loss function to achieve accurate identification and data augmentation of defects at different scales. The task-aware loss module, which combines the Dice loss and the boundary loss, is used to optimize the generated images and enhance the performance of downstream detection and segmentation tasks. The loss function design of the present invention combines the basic loss and the task-aware loss to ensure that the generated images can reach a high level in terms of both quality and task adaptability. The basic loss optimizes the visual quality of the generated images, while the task-aware loss optimizes the actual application effect of the images, especially in tasks such as industrial defect detection.

[0155] In summary, the present invention improves the traditional industrial defect detection method, adopts a hierarchical data augmentation scheme, and completes the defect detection of low-pixel targets, high similarity, and small samples with high precision. This data augmentation scheme is applicable to the similarity defect detection of various types of industrial scenarios. By introducing the Sense of Information Awareness (SIA) branch and the Local and Location Information Awareness (LLIA) branch, Real-Adapter can accurately extract and fuse industrial scene features, as well as local defect features and location information, thereby improving the quality and applicability of data augmentation. Combined with the task-aware loss function, this scheme can generate more realistic and diverse industrial data, significantly improving the performance of downstream detection tasks. Thus, the construction and training of the lightweight diffusion model adapter Real-Adapter are realized. By working in cooperation with the industrial defect detection data augmentation task, a hierarchical prediction network is adopted to first locate the significant features in the industrial scene and then perform refined data augmentation to complete the defect detection in low-sample industrial scenarios.

[0156] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0157] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices.

[0158] The above embodiments have introduced the present invention in detail. Specific examples are used herein to elaborate on the principles and embodiments of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An industrial defect detection and data augmentation method based on a lightweight diffusion model adapter, characterized in that Including the following steps: S1. Acquisition and preprocessing of industrial scenario data; S2. Use a pre-trained denoising diffusion model to generate high-fidelity images, initialize the model weights and keep them frozen during the inference process; S3. Design a lightweight perception information branch SIA, and generate global scene features F through the scene control adapter combined with the self-attention mechanism sc ; S4. Design a lightweight local and location information branch LLIA, and use the localization Dino to extract the local features H of industrial defects 4,lc , and use the CLIP encoder to generate the description of the defect location information H t , and finally fuse the local defect features and location information to generate the local control signal F lc ; S5. Embed the global scene feature F sc into the encoder of the denoising diffusion model to control the overall global structure of the generated image, and fuse the local defect F lc with the position information H t to obtain the local control joint feature F control and embed it into the decoder of the denoising diffusion model to generate the global structure and local details of the image; S6. For the trained lightweight diffusion model, use the task-aware loss combined with the basic loss of the denoising diffusion model as the comprehensive loss function to optimize the generation results and enhance the performance of downstream detection and segmentation tasks.

2. The industrial defect detection and data enhancement method based on a lightweight diffusion model adapter according to claim 1, wherein: In step S1, for the acquisition and preprocessing of industrial scenario data, the specific process includes the following steps: S11. Collect images from various industrial scenarios, including defect images of different materials such as steel plate surfaces, plastic parts, and electronic components. These images cover various industrial defects such as scratches, cracks, protrusions, pits, and corrosion; S12. Perform denoising, normalization, and size adjustment standardization on the collected original images to improve data consistency and adaptability, adjust the images to a size suitable for input to the diffusion model, and perform color standardization to reduce the deviation of different image sources; S13. Label the defect images to generate true label masks for subsequent calculation and verification of the task-aware loss.

3. A method for industrial defect detection and data augmentation based on a lightweight diffusion model adapter according to claim 1, characterized in that: In step S2, for using a pre-trained denoising diffusion model to generate high-fidelity images, initialize the model weights and keep them frozen during the inference process, the specific process includes the following steps: S21. The key of the denoising diffusion model is the U-Net structure, which includes an encoder and a decoder, and can gradually recover high-fidelity images from the noisy images through convolutional and transposed convolutional operations; S22. The forward diffusion process transforms the original image x0 into an image with gradually increasing noise x t , and finally becomes a pure noise image x T at time step T; S23. The reverse denoising process starts with a pure noise image x T and the diffusion model gradually removes the noise to recover the denoised image x t-1 ; S24. During the inference process, to improve the inference efficiency and save computing resources, freeze the weights of the denoising diffusion model, that is, no longer perform backpropagation and gradient update.

4. An industrial defect detection and data augmentation method based on a lightweight diffusion model adapter according to claim 3, characterized in that: Step S22 specifically includes: At each time step t, noise is added through the conditional probability distribution q(x t |x t-1 ), with the formula: Among them, \(x_0\) represents the original image, and \(x\) t represents the image at time step \(t\), \(\alpha\) i represents the cumulative amount of image information retained at time step \(t\), and \(\alpha\) t represents the intensity of the noise added at each time step, and \(\alpha\) t \( = 1 - \beta\) t where \(\beta\) t represents the intensity of the noise added at each time step, and \(I\) is the identity matrix; For any time step t, the noisy image x can be directly generated from the original image x0 t : Among them, denotes the noise sampled from the standard normal distribution.

5. A method for industrial defect detection and data augmentation based on a lightweight diffusion model adapter according to claim 3, characterized in that: Step S23 specifically includes: At time step t, the conditional probability distribution calculation formula for the reverse process is: Among them, Σ θ (x t , t) represents the variance of the noise, which is usually a constant in the model or calculated according to the time step t. μ θ (x t , t) represents the mean predicted by the denoising network U-Net and is predicted by the following formula: Among them, ∈ θ (x t , t) represents the predicted value of the model for the current noise; At each time step t, the noise predicted by the model is removed from the current image x t to obtain the denoised and restored image x t-1 . This process gradually removes the noise in the image, thereby restoring the original content of the image. The model denoises and restores the image through the following formula: where, x t represents the image at the current time step, ∈ θ (x t , t) represents the noise predicted by the model, σ t represents the predefined standard deviation of the noise, usually taking a value of represents the noise sampled from the standard Gaussian distribution.

6. A method for industrial defect detection and data augmentation based on a lightweight diffusion model adapter according to claim 1, characterized in that: The specific process of step S3 includes the following steps: S31. Select 4 feature extraction blocks and 4 double downsampling blocks to construct a scene control adapter. The feature extraction block contains a feature enhancement block and a residual block, which are used to extract the global features of the industrial scene layer by layer. This design can not only effectively extract global features, but also will not significantly increase the computational burden due to too many blocks, making it suitable for implementing efficient object detection and feature extraction tasks under limited computational resources; and use the Vision Mamba mechanism to extract the global hidden scene feature H 4,sc , enhancing the ability to capture the global characteristics of the industrial scene; S32. Extracted global hidden scene feature H 4,sc It is further processed by a 1×1 convolutional layer to reduce the dimension and adjust the features, generating the final global feature F sc .

7. A method for industrial defect detection and data augmentation based on a lightweight diffusion model adapter according to claim 1, characterized in that: The specific process of step S4 includes the following steps: S41. Use the positioning Dino module to extract the local features of the scratch and pit defect areas from the publicly available industrial defect dataset. The defect features are further refined through a specific Attention mechanism, and the local hidden defect features H are extracted through a pre-trained model 4,lc ; S42. Define a text position prompt for describing the defect location and type: There is a scratch in the upper left corner of the workpiece, or there is a protrusion in the central area of the workpiece, or there is a scratch on the surface of the cylinder. Use the CLIP text encoder to process the position text description and extract the defect location information H t ; S42. Fuse the local hidden defect feature H 4,lc with the defect position information H t through splicing and 1×1 convolution to generate the final local and position control signal F of the LLIA branch lc .

8. A method for industrial defect detection and data augmentation based on a lightweight diffusion model adapter according to claim 1, characterized in that: The specific process of step S5 includes the following steps: S51. The global scene feature F extracted from the lightweight perception information branch SIA sc is embedded into each layer of the encoder of the diffusion model to control the overall layout and background characteristics of the generated image. The cross-attention mechanism calculates the attention weights based on the dot product of the query matrix Q and the key matrix K, thereby applying the final global scene feature F sc to the feature F of the current layer of the encoder en,i to obtain the output feature F' through fusion en,i . The specific calculation formula is as follows: F′ en,i = Atten(Q, K, V) + where Q = F en,i · W q : The query matrix Q is generated by the input feature F of the i-th layer of the encoder en,i and the weight matrix W q ; K = F sc · W k : The key matrix K is generated by the global scene feature F sc and the weight matrix W k ; V = F sc · W v : The value matrix V is generated by the global scene feature F sc and the weight matrix W v ; F' en,i represents the output feature after fusion and is used for the subsequent decoding stage; S52. The final local defect feature F extracted by the lightweight local and location information branch LLIA lc and the location information H t are fused into the local control joint feature F control : Among them, represents a splicing operation, and K 1×1 represents a convolution kernel for dimensional compression; At each layer of the denoising diffusion model decoder, the local control joint feature F control is embedded in the decoded feature F of the current layer de,i and the calculation formula is as follows: where Q = F de,i ·W q means that the query matrix Q is generated by the input feature F of the i-th layer of the decoder de,i and denotes the key matrix of the global scene feature, denotes the value matrix of the global scene feature, denotes the key matrix of the local control joint feature, denotes the value matrix of the local control joint feature; denotes the element-wise addition operation, and F' de,i denotes the fused feature output by the decoder.

9. A method for industrial defect detection and data augmentation based on a lightweight diffusion model adapter according to claim 1, characterized in that: Step S6 specifically includes: Combine the base loss L of the denoising diffusion model θ with the task-aware loss L task as the comprehensive loss function L, and the comprehensive loss function can be expressed as: L = L θ + λL task ; where λ represents the weight of the task perception loss, controlling the relative importance of the two loss functions, and L θ represents the diffusion loss function, which is optimized by calculating the mean square error MSE between the noise ∈ θ (x t , t) predicted by the model and the true noise ∈. The training objective of the diffusion model is to enable the model to accurately predict the noise added to the image at each time step, which can be specifically expressed as: where \(x_0\) represents the original image, \(x\) t represents the image after adding noise at the \(t\)-th step, \(\epsilon\) represents the actually added noise, \(\epsilon\) θ (x t , t) represents the noise predicted by the model, represents the expected value, which calculates the average error over all time steps and noise samples; L task represents the task-aware loss, which specifically combines multiple task-related loss functions and is dedicated to ensuring that the generated images can effectively support downstream tasks such as defect detection and image segmentation. The task-aware loss L task mainly uses the weighted sum method to fuse the Dice loss L D and the boundary loss L B , and can be specifically expressed as: L task = αL D + (1 - α)L B Among them, α is a hyperparameter representing the weight allocation between the Dice loss and the boundary loss, which is usually dynamically adjusted as the training progresses; L D represents the Dice loss, which measures the similarity of the target region. This loss function calculates the Dice coefficient between the generated image g i and the ground truth mask p i and optimizes the model according to the value of the coefficient. The Dice coefficient is calculated by comparing the overlap degree of the predicted mask and the ground truth mask: Among them, p i represents the predicted value of the i-th pixel in the generated image, and g i represents the true value of the i-th pixel in the target image, and ∑ i represents the summation over all pixels; L B is the boundary loss used to optimize the boundary details of defective targets in the generated image, especially the contour or edge part of the targets in the image. This loss emphasizes the boundary region of the targets rather than the overall region; the boundary loss L B can be calculated by the following formula: where C and N represent the number of classes and pixels respectively, and p i represents the predicted value of the i-th pixel in the generated image, and g i represents the true value of the i-th pixel in the target image, and δ(g i ) represents a weight function that assigns larger weights to the boundary pixels of the target, ensuring stronger optimization of the boundary part.

Images