Zero-shot industrial defect generation method and device based on decoupling and recoupling

Abstract

The present application relates to the technical field of computer vision and industrial defect detection, and particularly relates to a zero-shot industrial defect generation method and device based on decoupling and recoupling. The method comprises the following steps: training a diffusion model injected with a low-rank adapter by using a normal image set of a target specific product to obtain a target product low-rank adapter P-LoRA; training the diffusion model injected with the low-rank adapter by using a joint text prompt word, a defect image set of a source reference product, and a defect mask, adjusting parameters of the low-rank adapter through a double routing flow mechanism and a parallel product independent flow PI-Flow constraint to obtain a source defect low-rank adapter A-LoRA; and mixing the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA into the diffusion model to obtain an inference model, and generating an industrial defect image by using the inference model. The present application can generate high-quality industrial defect images.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial defect detection technology, and in particular to a method and apparatus for generating zero-sample industrial defects based on decoupling and recoupling. Background Technology

[0002] In modern manufacturing, automated quality control is crucial for ensuring product yield. Industrial defect detection aims to identify defective areas on industrial products. However, defect samples are extremely scarce in real-world industrial scenarios, leading to data hunger for model training. To alleviate this data hunger problem, zero-shot defect generation technology has become a research hotspot. Zero-shot defect generation technology can generate defect samples without requiring images of actual product defects. Existing zero-shot methods are mainly divided into two categories: text-guided methods and reference image-guided methods. However, in text-guided methods, due to the semantic gap between fixed prompts and complex industrial defects, unrealistic patterns are often generated. While reference image-guided methods can introduce visual features, they struggle to effectively separate the defects of the source reference product from the product background, causing irrelevant textures from the source reference product to easily leak into the normal image of the target product, resulting in severe feature entanglement. Furthermore, existing methods lack explicit defect-product fusion mechanisms, making the generated defects appear abrupt on the target product.

[0003] To address feature entanglement in image generation, LoRA ensemble decoupling techniques (such as B-LoRA and UnzipLoRA) have been widely explored in general domains. These techniques separate image attributes through decoupling and coupling mechanisms and apply them to new targets. However, directly transferring LoRA ensemble decoupling techniques to industrial defect generation tasks yields limited results. This is because existing LoRA ensemble decoupling techniques heavily rely on strong semantic priors (such as explicit concepts like "dog" and "watercolor") inherent in pre-trained models for feature binding. In contrast, industrial defects (such as scratches and cracks) are highly abstract and lack strong semantic priors. This semantic vacuum renders existing binding strategies ineffective, and the model cannot accurately extract the pure defect structure. Furthermore, during the inference phase, existing methods rely solely on simple text concatenation (such as a tear in a carpet) to generate defects. This approach ignores potential texture conflicts that may occur during cross-product transfer, resulting in poor compatibility between the extracted defect structure and the surface texture of the target product. The generated defects are either submerged in the background texture or appear unnatural due to abrupt blending.

[0004] In summary, although existing zero-sample defect generation technology and LoRA combined decoupling technology provide preliminary ideas for generating industrial defect samples, they still cannot effectively resolve the contradiction between accurate defect structure stripping and cross-product compatibility when dealing with highly abstract industrial defects.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem that this invention aims to solve is that, in the scenario of generating industrial defect samples, existing technologies cannot effectively resolve the contradiction between accurate stripping of defect structures and cross-product compatibility.

[0007] The embodiments of the present invention adopt the following technical solutions: Firstly, this invention provides a zero-sample industrial defect generation method based on decoupling and recoupling, specifically: using a target specific product... Normal image set The diffusion model injected with the low-rank adapter was trained to obtain the target product low-rank adapter P-LoRA; Use combined text prompts Source reference product Defect image set Defect Image Set Defect mask The diffusion model injected with the low-rank adapter was trained, and the parameters of the low-rank adapter were adjusted by the dual-route flow mechanism and the parallel product-independent flow PI-Flow constraint to obtain the source defect low-rank adapter A-LoRA. The source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA are mixed and injected into the diffusion model to obtain the inference model. The inference model is then used to analyze the target specific product. The surface texture is seamlessly integrated from the original reference product. To identify defects and generate high-fidelity images of industrial defects.

[0008] Preferably, during the training process, the method further includes: Inject a trainable source-product low-rank adapter P-LoRA into the diffusion model, using normal product prompt words. With source reference product Normal image set Train the diffusion model, and freeze the source product low-rank adapter P-LoRA after training is complete; Based on the diffusion model with the source product low-rank adapter P-LoRA, a trainable source defect low-rank adapter A-LoRA is injected, and a dual-route flow mechanism is used to process the joint text prompts. Source reference product Defect image set By performing dual-structure decoupling, a joint predicted velocity field is obtained. and in the defect mask The flow matching loss based on the velocity field is constrained to obtain the joint loss value. ; Compare with normal product prompts Compared to pure defect warning words The difference in flow matching loss is calculated, and parallel product-independent flows are used as constraints to calculate the product-independent loss value. ; Combined loss value Product-independent loss value Weighted fusion is performed to obtain the total loss value. The parameters of the trainable source defect low-rank adapter A-LoRA are updated based on the total loss value to complete the training of the source defect low-rank adapter A-LoRA.

[0009] Preferably, the method further includes: AP-Routing on the text side for combined text prompts Text tokens are segmented, and the segmented text tokens are processed using the frozen source-product low-rank adapter P-LoRA and the trainable source-defect low-rank adapter A-LoRA to obtain the query features of the text modality. Key features of text modality and text modality value features ; QK-Routing on the image side for source reference products Defect image set Spatial geometric structure extraction is performed to obtain query features of image modalities. Key features of image modalities and image modality value features ; Combine text modalities and image modalities into a unified vector set. The system performs joint self-attention calculations within the multimodal transformer, and then outputs the jointly predicted velocity field through subsequent network layers. .

[0010] Preferably, the frozen source product low-rank adapter P-LoRA includes the frozen text product LoRA, and the trainable source defect low-rank adapter A-LoRA includes the trainable text defect LoRA. The method further includes: The combined text prompt words Word segmentation yields text tags. ; The text mark Classified as defect text markup Product text markings and other background fill marks ; Using trainable text defect LoRA to process defective text tags separately , obtain query features Key features Sum value characteristics ;Utilize frozen text products LoRA to process product text tags separately , obtain query features Key features Sum value characteristics The remaining background-filled markers are processed jointly using trainable text defect LoRA and frozen text product LoRA. To obtain the query features of the text modality. Key features Sum value characteristics ; The query features of the text modality are obtained by concatenating the three sets of query features, key features, and value features respectively. Key features of text modality and text modality value features .

[0011] Preferably, the frozen source product low-rank adapter P-LoRA further includes the frozen image product LoRA, and the trainable source defect low-rank adapter A-LoRA further includes the trainable image defect LoRA. The method further includes: Source reference product Defect image set Feature extraction is performed to obtain image labels. ; Image tagging using trainable image defect LoRA Get query features ; Use the frozen image product LoRA to process image tags Obtain key features Sum value characteristics .

[0012] Preferably, the source reference product is used. Normal product labeling Perform constraint calculations to obtain the product-irrelevant loss value. ,include: Activate only the frozen source product low-rank adapter P-LoRA, and enter the normal product prompt. To predict the velocity field of pure products and the pure product speed field From constant target speed Subtract from the middle to construct the residual target flow representing the pure defect. ; Activate only the trainable source defect low-rank adapter A-LoRA and input pure defect cue words. Predicting product-irrelevant flows ; In defect mask Internal residual-based target flow Unrelated to the product Parallel constraint calculations are performed on the flow matching loss to obtain the product-independent loss value. .

[0013] Preferably, during the reasoning process, the method further includes: Source reference product Defect name and target specific product By combining the category names, we can obtain mixed reasoning clues. ; Only activate the target product's low-rank adapter P-LoRA, and display the target product prompt. As input to the reasoning model, calculate the target-specific product. Predicted velocity field ; Simultaneously activate the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA, and use the hybrid inference prompt. As input to the inference model, the joint predicted velocity field is calculated. ; Jointly predict the velocity field Along the direction away from the predicted velocity field Extrapolate the direction to calculate the corrected velocity field and using the corrected velocity field For noisy normal images Denoising is performed to generate images of industrial defects.

[0014] Preferably, the method further includes: Target specific products Normal image Mapped to latent space features by variational autoencoder ; For the latent spatial features Perform forward denoising to obtain the denoised reference latent features for the corresponding time step. ; Using the corrected velocity field Driven inverse denoising to obtain denoising latent variables And within each inverse denoising iteration time step, the noisy reference latent features are... and the denoising latent variables Perform a mask-based region-specific hybrid diffusion stitching to obtain an image of industrial defects.

[0015] In a second aspect, the present invention provides a zero-sample industrial defect generation apparatus based on decoupling and recoupling, the apparatus comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to perform the zero-sample industrial defect generation method based on decoupling and recoupling in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the zero-sample industrial defect generation method based on decoupling and recoupling described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: During the training phase, a dual-route flow mechanism precisely decouples the textual concepts and image attention dimensions. Furthermore, it innovatively utilizes parallel product-independent flow (PI-Flow) constraints to completely block the leakage path from the source reference product texture to the target specific product, ensuring the purity of the captured defect features. In the inference phase, this invention abandons the traditional joint cross-attention generation method, injecting a hybrid source defect low-rank adapter and a target product low-rank adapter into the diffusion model. During inference, these adapters are used in stages, overcoming the feature rejection problem of independent training adapters in cross-domain coupling and effectively mitigating the technical defect of defect features being easily swallowed by the background. Through the collaborative cooperation of adapters in the training and inference phases of this invention, high-quality, seamlessly integrated, and high-fidelity industrial defect images can be generated. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is a flowchart illustrating a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention. Figure 2 This is a flowchart of a method for training a source defect low-rank adapter A-LoRA in a zero-sample industrial defect generation method based on decoupling and recoupling, provided by an embodiment of the present invention. Figure 3This is a flowchart of a method for obtaining a joint predicted velocity field in a zero-sample industrial defect generation method based on decoupling and recoupling, provided by an embodiment of the present invention. Figure 4 This is a flowchart of a method for obtaining text features in a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention; Figure 5 This is a flowchart of a method for obtaining image features in a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention; Figure 6 This is a flowchart of a method for calculating product-irrelevant loss values ​​in a zero-sample industrial defect generation method based on decoupling and recoupling, provided by an embodiment of the present invention. Figure 7 This is a flowchart of the inference stage in a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention; Figure 8 This is a flowchart of a method for obtaining industrial defect images in a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention; Figure 9 This is a flowchart of the training phase in a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention. Figure 10 This is a flowchart of the inference stage in a zero-sample industrial defect generation method based on decoupling and recoupling provided in an embodiment of the present invention. Figure 11 This is a schematic diagram of a zero-sample industrial defect generation device based on decoupling and recoupling provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0022] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0023] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0024] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0025] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Example 1: like Figure 1As shown, Embodiment 1 of the present invention provides a zero-sample industrial defect generation method based on decoupling and recoupling. The method includes a training phase and an inference phase, wherein steps 101 and 102 are the training phase, and the training phase does not use a target-specific product. For any real defect image, step 103 is the inference stage. In the inference stage, the trained source defect low-rank adapter A-LoRA and the trained target product low-rank adapter P-LoRA are mixed and injected into the diffusion model to obtain the inference model. Then, the inference model is used to target specific products without real defect samples. To generate industrial defect images using zero-sample generation, the method specifically includes the following steps: Step 101: Use the target-specific product Normal image set The diffusion model injected with the low-rank adapter was trained to obtain the target product low-rank adapter P-LoRA.

[0027] During training, this invention uses a diffusion model as the main body and injects a low-rank adapter into the diffusion model to target specific products. Normal image set As a training set, only the parameters of the low-rank adapter are updated, without retraining the full set of parameters of the diffusion model.

[0028] Step 102: Use combined text prompts Source reference product Defect image set Defect Image Set Defect mask The diffusion model of the low-rank adapter was trained, and the parameters of the low-rank adapter were adjusted by the dual-route flow mechanism and the parallel product-independent flow PI-Flow constraint to obtain the source defect low-rank adapter A-LoRA.

[0029] In one embodiment, during the training of the source defect low-rank adapter (Anomaly LoRA, abbreviated as A-LoRA), a dual-routing flow (DR-Flow) is introduced to train the source defect low-rank adapter A-LoRA. The dual-routing flow DR-Flow includes text-side AP-Routing and image-side QK-Routing. The text-side AP-Routing performs text tag splitting and classification on the joint text cue words, and uses different low-rank adapters to process different categories of text tags; the image-side QK-Routing processes the source reference product... Defect image set Spatial geometric structure extraction is performed, and query features are obtained using different low-rank adapters. Key features Sum value characteristics .

[0030] This invention also sets up a parallel product-invariant flow (PI-Flow), which is used as a constraint in the training process of the source-defect low-rank adapter A-LoRA to completely prevent the source-defect low-rank adapter A-LoRA from directly fitting the source reference product during training. The tendency to capture surface texture defects to reduce training loss further ensures the purity of capturing defective features.

[0031] Step 103: Inject the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA into the diffusion model to obtain the inference model. Use the inference model to target specific products. The surface texture is seamlessly integrated from the original reference product. To identify defects and generate high-fidelity images of industrial defects.

[0032] Specifically, the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA are mixed and injected into the diffusion model to obtain the inference model. During the inference process of the inference model, the target product low-rank adapter P-LoRA is activated separately, and the target product prompt word is displayed. As input to the reasoning model, calculate the target-specific product. Predicted velocity field Simultaneously activate the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA, and use mixed inference prompts. As input to the inference model, the joint predicted velocity field is calculated. The velocity field will be jointly predicted. Along the direction away from the predicted velocity field Extrapolate the direction to calculate the corrected velocity field and using the corrected velocity field For noisy normal images Denoising is performed to obtain industrial defect images; this method can improve the quality of the source reference product. Seamlessly integrate the defects into the target-specific product On the surface texture, an image of industrial defects is obtained that blends seamlessly with the background.

[0033] In this embodiment, during the training phase, a dual-route flow mechanism precisely decouples the textual concepts and image attention dimensions. Furthermore, it innovatively utilizes the parallel product-independent flow (PI-Flow) constraint to completely block the leakage path from the source reference product texture to the target specific product, ensuring the purity of the captured defect features. Additionally, during the inference phase, this invention abandons the traditional joint cross-attention generation method, injecting a hybrid source defect low-rank adapter and a target product low-rank adapter into the diffusion model. These adapters are used in stages during inference, overcoming the feature rejection problem of independent training adapters during cross-domain coupling and effectively mitigating the technical defect of defect features being easily swallowed by the background. Through the collaborative cooperation of adapters in the training and inference phases of this invention, high-quality, seamlessly integrated, and high-fidelity industrial defect images can be generated.

[0034] In one embodiment, in step 102, the low-rank adapter injected into the diffusion model includes a trainable source product low-rank adapter P-LoRA and a trainable source defect low-rank adapter A-LoRA. This invention uses normal product prompts. With source reference product Normal image set The training process involves injecting a diffusion model with a trainable source product low-rank adapter P-LoRA. After training, the source product low-rank adapter P-LoRA is frozen. Based on the diffusion model with the source product low-rank adapter P-LoRA, a trainable source defect low-rank adapter A-LoRA is injected. Subsequently, using the frozen source product low-rank adapter P-LoRA and the trainable source defect low-rank adapter A-LoRA, the total loss value is calculated, and the parameters of the source defect low-rank adapter A-LoRA are updated based on the total loss value, completing the training of the source defect low-rank adapter A-LoRA. (See [link to documentation]). Figure 2 Step 102 specifically includes the following steps: Step 201: Inject a trainable source product low-rank adapter P-LoRA into the diffusion model, using normal product cue words. With source reference product Normal image set The diffusion model is trained, and the source product low-rank adapter P-LoRA is frozen after training.

[0035] In one embodiment, normal product prompts (such as "a") will be used. <product>) and source reference product Normal image set The input diffusion model is used to train the source product low-rank adapter P-LoRA using flow matching loss, so that the source product low-rank adapter P-LoRA learns the source reference product. The features of normal images are extracted, and the parameters of the low-rank adapter P-LoRA of the source product are frozen during subsequent training.

[0036] Step 202: Based on the diffusion model with the source product low-rank adapter P-LoRA, inject the trainable source defect low-rank adapter A-LoRA, and use a dual-route flow mechanism to process the joint text prompt words. Source reference product Defect image set By performing dual-structure decoupling, a joint predicted velocity field is obtained. and in the defect mask The flow matching loss based on the velocity field is constrained to obtain the joint loss value. .

[0037] In one embodiment, the joint loss value The joint loss function is obtained as follows: ; ; in, For the joint loss function, For defect masking, To jointly predict the velocity field, For a constant target speed, For time steps Latent variables of noise defects at that time For combined text prompts, It represents the Hadamardi (or Hadama) stack.

[0038] Among them, for the source reference product Defect image set For each defect image in the dataset, the defect image is mapped to the latent space and noise is added to obtain the noise defect latent variable. .

[0039] Step 203: Compare with normal product labeling. Compared to pure defect warning words The difference in flow matching loss is calculated, and parallel product-independent flows are used as constraints to calculate the product-independent loss value. .

[0040] In one embodiment, product-irrelevant loss value The irrelevant flow loss function is obtained through the irrelevant flow loss function, which is as follows: ; in, For irrelevant flow loss function, For defect masking, For products unrelated to flow, For residual target flow, For time steps Latent variables of noise defects at that time These are pure defect warning words. It represents the Hadamardi (or Hadama) stack.

[0041] Step 204: Combine the joint loss values Product-independent loss value Weighted fusion is performed to obtain the total loss value. The parameters of the trainable source defect low-rank adapter A-LoRA are updated based on the total loss value to complete the training of the source defect low-rank adapter A-LoRA.

[0042] In one embodiment, the formula for calculating the total loss value is as follows: ; in, This is the total loss value. For the joint loss value, This is a product-irrelevant loss value. The loss weight balancing hyperparameter.

[0043] By using the total loss value, the source-defect low-rank adapter A-LoRA is forced to strip all source reference products during training. Specific features, extract only the source reference product The pure defect.

[0044] In one implementation, the loss weight balancing hyperparameter Other values ​​can also be set, and the specific value can be determined according to the actual situation.

[0045] In one embodiment, the dual-routing mechanism includes AP-Routing on the text side and QK-Routing on the image side, see [reference]. Figure 3 Step 202 specifically includes the following steps: Step 301: Use AP-Routing on the text side to process joint text prompts. Text tokens are segmented, and the segmented text tokens are processed using the frozen source-product low-rank adapter P-LoRA and the trainable source-defect low-rank adapter A-LoRA to obtain the query features of the text modality. Key features of text modality and text modality value features .

[0046] Step 302: Image-side QK-Routing of the source reference product Defect image set Spatial geometric structure extraction is performed to obtain query features of image modalities. Key features of image modalities and image modality value features .

[0047] Step 303: Combine the text modality and the image modality into a unified vector set. The system performs joint self-attention calculations within the multimodal transformer, and then outputs the jointly predicted velocity field through subsequent network layers. .

[0048] In one embodiment, the query features of the text modality obtained in step 301 are... Key features of text modality and text modality value features The query features of the image modality obtained in step 302 Key features of image modalities and image modality value features Combined into a unified vector set And perform joint self-attention calculation within the multimodal transformer. During this process, isolated defect text tags are used. As the sole text-conditionally driven attention map interacts with image defect features, it prompts the source defect low-rank adapter A-LoRA to bind the defect structure to the defect text tag. The joint predicted velocity field is then calculated and output by subsequent network layers. and masking the defect area. The internal constraint optimization is performed using flow matching loss based on the velocity field.

[0049] In one embodiment, the frozen source product low-rank adapter P-LoRA includes the frozen text product LoRA, and the trainable source defect low-rank adapter A-LoRA includes the trainable text defect LoRA. (See also...) Figure 4 Step 301 specifically includes the following steps: Step 401: Combine the text prompt words Word segmentation yields text tags. .

[0050] In one embodiment, the combined text prompt words will be used. (e.g., "an") <anomaly>on <product>The text tags are obtained through word segmentation. .

[0051] Here, text markers can be understood as semantically meaningful lexical units. Step 401 essentially involves combining text prompts. The text is broken down into multiple semantic units, and then each unit is categorized in step 402. The categories include defect text, product text, and other background fill.

[0052] Step 402: Mark the text Classified as defect text markup Product text markings and other background fill marks .

[0053] Step 403: Process defective text tags separately using trainable text defect LoRA. , obtain query features Key features Sum value characteristics ;Utilize frozen text products LoRA to process product text tags separately , obtain query features Key features Sum value characteristics The remaining background-filled markers are processed jointly using trainable text defect LoRA and frozen text product LoRA. To obtain the query features of the text modality. Key features Sum value characteristics .

[0054] Step 404: Concatenate the three sets of query features, key features, and value features respectively to obtain the query features of the text modality. Key features of text modality and text modality value features .

[0055] This invention utilizes AP-Routing on the text side to process joint text prompts and QK-Routing on the image side to process the defect image set of the source reference product, achieving dual structural-level decoupling of text and image. Specifically, in the AP-Routing on the text side, defect text markers are processed separately using text defect LoRA, product text markers are processed separately using text product LoRA, and the remaining background filling markers are processed jointly using text defect LoRA and text product LoRA, thereby further decoupling the text processing process.

[0056] In one embodiment, the frozen source product low-rank adapter P-LoRA also includes a frozen image product LoRA, and the trainable source defect low-rank adapter A-LoRA also includes a trainable image defect LoRA. See [reference needed]. Figure 5 Step 302 specifically includes the following steps: Step 501: Place the source reference product Defect image set Feature extraction is performed to obtain image labels. .

[0057] Step 502: Process image labels using trainable image defect LoRA Get query features ; Use the frozen image product LoRA to process image tags Obtain key features Sum value characteristics .

[0058] In one embodiment, query features are computed using trainable image defect LoRA. ,in, To query features, As the basic weights of the diffusion model, These are the weights of LoRA for image defects. Label the image.

[0059] In one embodiment, LoRA computational key features of the frozen image product are utilized. ,in, Key features, As the basic weights of the diffusion model, It is the weight of LoRA for image products. Label the image.

[0060] In one embodiment, LoRA-calculated value features of the frozen image product are utilized. ,in, For value characteristics, As the basic weights of the diffusion model, It is the weight of LoRA for image products. Label the image.

[0061] In image-side QK-Routing, image defect LoRA is used to generate query features to encode the spatial geometric arrangement and structure of defects; image product LoRA is used to generate key and value features to provide visual appearance information. This design restricts the underlying texture to only come from normal products, so that the trainable source defect low-rank adapter A-LoRA can only focus on extracting source reference products when it cannot obtain appearance information. Pure structural information about defects in the middle.

[0062] See Figure 6 Step 203 specifically includes the following steps: Step 601: Activate only the frozen source product low-rank adapter P-LoRA, and enter the normal product prompt. To predict the velocity field of pure products and the pure product velocity field From constant target speed Subtract from the middle to construct the residual target flow representing the pure defect. .

[0063] Specifically, only activate the frozen source product low-rank adapter P-LoRA, and the normal product prompt word will be displayed. As a textual guide, calculate the pure product velocity field. .

[0064] The pure product velocity field obtained above From constant target speed Subtract from the middle to construct the residual target flow representing the pure defect. .

[0065] Step 602: Activate only the trainable source defect low-rank adapter A-LoRA and input pure defect prompt words. Predicting product-irrelevant flows .

[0066] In one embodiment, only the source defect low-rank adapter A-LoRA is activated, and the pure defect prompt word is displayed. As a text-guided predictive product-irrelevant flow .

[0067] Step 603: In the defect mask Internal residual-based target flow Unrelated to the product Parallel constraint calculations are performed on the flow matching loss to obtain the product-independent loss value. .

[0068] based on Figures 2-6 The technical details of the training phase are clearly explained below. Figure 7 and Figure 8 For detailed technical information on the reasoning phase, please refer to [link / reference]. Figure 7 The reasoning stage specifically includes the following steps: Step 701: Place the source reference product Defect name and target specific product By combining the category names, we can obtain mixed reasoning clues. .

[0069] Step 702: Activate only the target product's low-rank adapter P-LoRA, and select the target product prompt word. As input to the reasoning model, calculate the target-specific product. Predicted velocity field .

[0070] In one embodiment, to compensate for the re-coupling texture conflict caused by the independent training of the target product low-rank adapter P-LoRA and the source defect low-rank adapter A-LoRA, this embodiment first activates only the target product low-rank adapter P-LoRA, and then uses the target product prompt word... Guided by the target specific product, calculate Predicted velocity field .

[0071] During the inference phase, the first weight of the target product's low-rank adapter P-LoRA is... The original weights of the backbone network of the injection diffusion model This hybrid injection rule is then applied to each attention linear projection layer of the multimodal transformer, thereby improving the targeting of product prompts. Guided by the first weight and original weights For noisy normal images The predicted velocity field is obtained through processing. The first weight Is it using a specific target product? Normal image set It was acquired through training.

[0072] Step 703: Simultaneously activate the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA, and use the hybrid inference prompt. As input to the inference model, the joint predicted velocity field is calculated. .

[0073] During the inference phase, both the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA are activated simultaneously to adjust the first weight of the target product low-rank adapter P-LoRA. The second weight of the source defect low-rank adapter A-LoRA Original weights of the backbone network in the co-injection diffusion model This hybrid injection rule is then applied to each attention linear projection layer of the multimodal transformer, thereby improving the hybrid inference cue word performance. Guided by the first weight Second weight and original weights For noisy normal images The joint predicted velocity field is obtained through processing. , where the second weight Is using the source reference product It was obtained after training with defective samples.

[0074] In one embodiment, the first weight Second weight and original weights Constituting inference weights Inference weight The expression is as follows: ; in, For inference weights, These are the original weights of the backbone network of the diffusion model. The first weight for the low-rank adapter of the target product. The second weight is the source defect low-rank adapter.

[0075] It should be noted that during the inference phase, the joint predicted velocity field in step 703 is calculated using the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA. During the training phase, the joint predicted velocity field in step 202 is calculated using the source product low-rank adapter P-LoRA and the source defect low-rank adapter A-LoRA. The two use different low-rank adapters, but the specific calculation methods are the same, so they will not be described in detail here.

[0076] Step 704: Jointly predict the velocity field Along the direction away from the predicted velocity field Extrapolate the direction to calculate the corrected velocity field and using the corrected velocity field For noisy normal images Denoising is performed to generate images of industrial defects.

[0077] In one embodiment, the joint predicted velocity field Along the velocity field away from the product Extrapolating in the direction of the correction velocity field causes it to focus solely on guiding the generation of the target defect, rather than on the background texture of the target product. The expression is as follows: ; in, To correct the velocity field, For product speed field, To jointly predict the velocity field, A scalar hyperparameter for controlling the compatibility correction intensity.

[0078] In one implementation, scalar hyperparameters The value is 2.0, but other values ​​can also be set. The specific value can be determined according to the actual situation.

[0079] See Figure 8 Step 704 specifically includes the following steps: Step 801: Target specific product Normal image Mapped to latent space features by variational autoencoder .

[0080] In one embodiment, a variational autoencoder is used to target a specific product. Normal image Mapping to the latent space yields the latent space features. .

[0081] Step 802: For the potential spatial features Perform forward denoising to obtain the denoised reference latent features for the corresponding time step. .

[0082] In one embodiment, the latent spatial features are directed according to a predefined noise schedule (such as the linear schedule of DDPM). By gradually adding noise, the latent noise defect variables at the corresponding time steps are obtained. This will transform the normal image. It degenerates to a form close to pure noise in order to keep the same as the input to the model during the training phase.

[0083] Step 803: Utilize the corrected velocity field Driven inverse denoising to obtain denoising latent variables And within each inverse denoising iteration time step, the noisy reference latent features are... and the denoising latent variables Perform a mask-based region-specific hybrid diffusion stitching to obtain an image of industrial defects.

[0084] In one embodiment, the implementation process of step 803 can be represented by the following expression: ; in, For industrial defect images, To utilize the corrected velocity field The denoised latent variables obtained from the calculation The mask defines the region after random spatial transformation and combination enhancement, and 1-m represents the region outside the mask definition region. As a noisy reference latent feature, It represents the Hadamardi (or Hadama) stack.

[0085] In the above expression, the denoising latent variable By performing the Hadamard product calculation with the mask-defined region, the corrected velocity field can be obtained. The generated defects are confined to the masked region; simultaneously, the noisy reference latent features are... Performing the Hadamard product with the region outside the mask-defined area preserves the normal image. Background information is provided to ensure that non-defective areas remain unchanged, thereby ensuring that the target-specific product is fully preserved. Normal image While maintaining normal detail textures in non-defect areas, defects are generated with high quality within the mask-defined area, resulting in a seamlessly integrated, high-fidelity industrial defect image.

[0086] It is important to note that masks are used in both the inference and training phases. During the inference phase, masks are used to define a region that is taken from the source reference product. The defects are coupled to the defect region of the normal image of the target product; during the training phase, through defect masking... The training process is constrained to the defect area corresponding to the source reference product.

[0087] The following is combined with Figure 9 Here is a brief description of the specific process of the training phase of this invention: The reference source image (i.e., the source reference product) is used... (defect image) and combined text prompts As input, the combined text prompt words will be used. Word segmentation yields text tags. ; mark the text Split into defect text tags Product text markings and other background fill marks On the one hand, in the text-side AP-Routing, the trainable text defect LoRA is used to process defective text tags separately. Using frozen text products, LoRA processes product text tags separately. The remaining background-filled markers are processed jointly using trainable text defect LoRA and frozen text product LoRA. To obtain the query features of the text modality. Key features of text modality and text modality value features On the other hand, in image-side QK-Routing, trainable image defect LoRA is used to compute query features of image modalities. ; Calculate the key features of image modalities using LoRA of frozen image products and image modality value features ; Query features of text modality Key features of text modality and text modality value features Query features of image modalities Key features of image modalities and image modality value features Combined into a unified vector set The system performs joint self-attention calculations within the multimodal transformer, and then outputs the jointly predicted velocity field through subsequent network layers. The flow matching loss value is calculated based on the joint predicted velocity field and the constant target velocity.

[0088] At the same time, compare with normal product prompts Compared to pure defect warning words The difference in flow matching loss is calculated, and parallel product-independent flows are used as constraints to calculate the product-independent loss value. For the joint loss value Product-independent loss value Weighted fusion is performed to obtain the total loss value. The parameters of the trainable source defect low-rank adapter A-LoRA are updated based on the total loss value to complete the training of the source defect low-rank adapter A-LoRA.

[0089] The following is combined with Figure 10 Here is a brief description of the specific process of the inference stage of this invention: Only the low-rank adapter P-LoRA of the target product is activated, and the noise is added... Normal image As input, in the target product prompt words Under the guidance of the target specific product, calculate Predicted velocity field Simultaneously activate the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA to add noise. Normal image As input, in the mixed reasoning prompts Guided by this, the joint predicted velocity field is calculated. The velocity field will be jointly predicted. Along the direction away from the predicted velocity field Extrapolate the direction to calculate the corrected velocity field and using the corrected velocity field For noisy normal images Denoising is performed to generate images of industrial defects.

[0090] In summary, this invention proposes a zero-shot industrial defect generation method based on decoupling and recoupling, overcoming the long-standing technical bottleneck that defect features in zero-shot data amplification are easily contaminated by the entanglement of source background textures. Through a carefully designed text tag splitting and spatial geometric structure extraction mechanism, it accurately achieves dual structural-level decoupling of text concepts and image attention dimensions; and innovatively utilizes residual constraint fields as strong conditions to completely block the leakage path from the source reference product texture to the target object, ensuring the purity of captured defect features. Experiments show that the industrial defect images generated by the method provided by this invention exhibit excellent realism and diversity. Compared with existing synthesis algorithms and few-shot generation algorithms, the method provided by this invention achieves significant advantages in both metrics. In the training and application of downstream defect detection models, it enables a leapfrog improvement of pixel-level positioning accuracy of up to 5.1% and 8.2% for various complex industrial products (such as MVTec AD and VisA). It also demonstrates significant advantages in industrial data amplification under cold-start conditions, cross-domain adaptive generation, and improved robustness of defect detection.

[0091] Example 2: Based on the zero-sample industrial defect generation method based on decoupling and recoupling provided in the foregoing embodiments, the present invention also provides a zero-sample industrial defect generation device based on decoupling and recoupling that can be used to implement the above method, such as... Figure 11 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. This embodiment of the zero-sample industrial defect generation device based on decoupling and recoupling includes one or more processors 21 and a memory 22. Figure 11 Take a processor 21 as an example.

[0092] Processor 21 and memory 22 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0093] The memory 22, as a non-volatile computer-readable storage medium for a zero-sample industrial defect generation method based on decoupling and recoupling, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the zero-sample industrial defect generation method based on decoupling and recoupling in the foregoing embodiments. The processor 21 executes various functional applications and data processing of the zero-sample industrial defect generation device based on decoupling and recoupling by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the zero-sample industrial defect generation method based on decoupling and recoupling in the foregoing embodiments.

[0094] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The program instructions / modules are stored in memory 22 and, when executed by one or more processors 21, execute the zero-sample industrial defect generation method based on decoupling and recoupling in the foregoing embodiments.

[0096] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 11 One of the processors 21 can enable the one or more processors to execute the zero-sample industrial defect generation method based on decoupling and recoupling in the foregoing embodiments.

[0097] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0098] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / product> < / anomaly> < / product>

Claims

1. A zero-sample industrial defect generation method based on decoupling and recoupling, characterized in that, include: Use target-specific products Normal image set The diffusion model injected with the low-rank adapter was trained to obtain the target product low-rank adapter P-LoRA; Use combined text prompts Source reference product Defect image set Defect Image Set Defect mask The diffusion model injected with the low-rank adapter was trained, and the parameters of the low-rank adapter were adjusted by the dual-route flow mechanism and the parallel product-independent flow PI-Flow constraint to obtain the source defect low-rank adapter A-LoRA. The source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA are mixed and injected into the diffusion model to obtain the inference model. The inference model is then used to analyze the target specific product. The surface texture is seamlessly integrated from the original reference product. To identify defects and generate high-fidelity images of industrial defects.

2. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 1, characterized in that, During the training process, the method further includes: Inject a trainable source-product low-rank adapter P-LoRA into the diffusion model, using normal product prompt words. With source reference product Normal image set Train the diffusion model, and freeze the source product low-rank adapter P-LoRA after training is complete; Based on the diffusion model with the source product low-rank adapter P-LoRA, a trainable source defect low-rank adapter A-LoRA is injected, and a dual-route flow mechanism is used to process the joint text prompts. Source reference product Defect image set By performing dual-structure decoupling, a joint predicted velocity field is obtained. and in the defect mask The flow matching loss based on the velocity field is constrained to obtain the joint loss value. ; Compare with normal product prompts Compared to pure defect warning words The difference in flow matching loss is calculated, and parallel product-independent flows are used as constraints to calculate the product-independent loss value. ; Combined loss value Product-independent loss value Weighted fusion is performed to obtain the total loss value. The parameters of the trainable source defect low-rank adapter A-LoRA are updated based on the total loss value to complete the training of the source defect low-rank adapter A-LoRA.

3. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 2, characterized in that, The method further includes: AP-Routing on the text side for combined text prompts Text tokens are segmented, and the segmented text tokens are processed using the frozen source-product low-rank adapter P-LoRA and the trainable source-defect low-rank adapter A-LoRA to obtain the query features of the text modality. Key features of text modality and text modality value features ; QK-Routing of the source reference product on the image side Defect image set Spatial geometric structure extraction is performed to obtain query features of image modalities. Key features of image modalities and image modality value features ; Combine text modalities and image modalities into a unified vector set. The system performs joint self-attention calculations within the multimodal transformer, and then outputs the jointly predicted velocity field through subsequent network layers. .

4. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 3, characterized in that, The frozen source product low-rank adapter P-LoRA includes a frozen text product LoRA, and the trainable source defect low-rank adapter A-LoRA includes a trainable text defect LoRA. The method further includes: The combined text prompt words Word segmentation yields text tags. ; The text mark Classified as defect text markup Product text markings and other background fill marks ; Using trainable text defect LoRA to process defective text tags separately , obtain query features Key features Sum value characteristics ;Utilize frozen text products LoRA to process product text tags separately , obtain query features Key features Sum value characteristics The remaining background-filled markers are processed jointly using trainable text defect LoRA and frozen text product LoRA. To obtain the query features of the text modality. Key features Sum value characteristics ; The query features of the text modality are obtained by concatenating the three sets of query features, key features, and value features respectively. Key features of text modality and text modality value features .

5. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 3, characterized in that, The frozen source product low-rank adapter P-LoRA also includes a frozen image product LoRA, and the trainable source defect low-rank adapter A-LoRA also includes a trainable image defect LoRA. The method further includes: Source reference product Defect image set Feature extraction is performed to obtain image labels. ; Image tagging using trainable image defect LoRA Get query features ; Use the frozen image product LoRA to process image tags Obtain key features Sum value characteristics .

6. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 2, characterized in that, The method further includes: Activate only the frozen source product low-rank adapter P-LoRA, and enter the normal product prompt. To predict the velocity field of pure products and the pure product speed field From constant target speed Subtract from the middle to construct the residual target flow representing the pure defect. ; Activate only the trainable source defect low-rank adapter A-LoRA and input pure defect cue words. Predicting product-irrelevant flows ; In defect mask Internal residual-based target flow Unrelated to the product Parallel constraint calculations are performed on the flow matching loss to obtain the product-independent loss value. .

7. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 1, characterized in that, During the reasoning process, the method further includes: Source reference product Defect name and target specific product By combining the category names, we can obtain mixed reasoning clues. ; Only activate the target product's low-rank adapter P-LoRA, and display the target product prompt. As input to the reasoning model, calculate the target-specific product. Predicted velocity field ; Simultaneously activate the source defect low-rank adapter A-LoRA and the target product low-rank adapter P-LoRA, and use the hybrid inference prompt. As input to the inference model, the joint predicted velocity field is calculated. ; Jointly predict the velocity field Along the direction away from the predicted velocity field Extrapolate the direction to calculate the corrected velocity field and using the corrected velocity field For noisy normal images Denoising is performed to generate images of industrial defects.

8. The zero-sample industrial defect generation method based on decoupling and recoupling according to claim 7, characterized in that, The method further includes: Target specific products Normal image Mapped to latent space features by variational autoencoder ; For the latent spatial features Perform forward denoising to obtain the denoised reference latent features for the corresponding time step. ; Using the corrected velocity field Driven inverse denoising to obtain denoising latent variables And within each inverse denoising iteration time step, the noisy reference latent features are... and the denoising latent variables Perform a mask-based region-specific hybrid diffusion stitching to obtain an image of industrial defects.

9. A zero-sample industrial defect generation device based on decoupling and recoupling, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the zero-sample industrial defect generation method based on decoupling and recoupling as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the zero-sample industrial defect generation method based on decoupling and recoupling as described in any one of claims 1-8.

Images