Abnormal sample generation system and abnormal detection device for pre-manufacturing quality control

Abstract

The application relates to the technical field of industrial quality inspection, in particular to an abnormal sample generation system and an abnormal detection device for pre-manufacturing quality control, which comprises a data acquisition module used for acquiring a historical abnormal data set and a normal image data set of a target product, the historical abnormal data set comprising a historical product category, corresponding abnormal text and an abnormal image, and the normal image data set comprising a target product normal image; an abnormal reference retrieval module used for acquiring potential abnormal text and corresponding reference abnormal images of the target product based on the target product category; and a unified abnormal synthesis module used for generating pre-manufacturing synthetic abnormal samples based on the reference abnormal images and the normal image. The application realizes the generation of unknown abnormalities of the target product before manufacturing to fill the early quality control data blank, improves the synthetic sample diversity, can accurately balance the generation of abnormal fidelity and rationality, efficiently reuses the historical abnormal data to reduce the complexity, and finally improves the downstream industrial abnormal detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial quality inspection, in particular to an abnormal sample generation system and an abnormal detection device for pre-manufacturing quality control. BACKGROUND

[0002] Industrial anomaly detection is the core link of modern manufacturing systems to achieve high-quality control, and its core task is to accurately and timely identify the subtle deviations from the standard mode in the production process, which is of great significance to ensure product quality and reduce production loss. However, this field has long been faced with two major challenges: first, the high requirements for detection accuracy and operation timeliness, which requires accurate capture of minor abnormalities and rapid response; second, the extreme data imbalance problem, i.e., normal samples dominate in production data, and real abnormal samples are extremely rare, which directly leads to limited performance of traditional detection models and makes it difficult to meet actual production needs.

[0003] To address the above challenges, the industry's technological development has mainly gone through two stages, but both have significant limitations and cannot adapt to the quality control goal of "prevention in advance" of modern manufacturing.

[0004] The first stage is the traditional defect detection technology, whose core idea revolves around feature engineering driven by human experience and specific scene adaptation. Early technologies rely on image preprocessing methods (such as histogram equalization, Gaussian filtering) to eliminate noise interference, combine edge detection algorithms to extract visible defect features, and then use mathematical morphology operations (such as erosion, dilation) and template matching techniques to locate abnormal areas; although shallow machine learning methods are introduced later, they still classify abnormalities based on human-designed texture, shape, and other features. This type of technology has obvious shortcomings: first, it is highly dependent on human-defined rules (such as threshold segmentation, edge threshold), and needs to design separate detection processes for specific defect types such as scratches and holes, which has poor adaptability; second, it has weak generalization ability and can only recognize existing abnormal patterns in training data, lacking the ability to respond to new types of abnormalities or complex texture disturbances such as light changes; third, it cannot generate new defect samples, and is limited by the data imbalance problem, with insufficient model training data; fourth, it cannot meet the needs of prevention in advance, and can only passively detect after the occurrence of abnormalities, lagging behind the quality control rhythm of modern manufacturing.

[0005] The second stage is the application of advanced generative architecture, which aims to alleviate the impact of data imbalance on model performance by synthesizing abnormal samples. This type of method usually follows a fixed process: first, collect existing real abnormal samples from the factory manufacturing process, then train a generative model based on these samples, and finally generate synthetic abnormal data through the model. However, this process still has four key limitations that further exacerbate the industrial detection problem: first, it cannot generate abnormalities before manufacturing, relying entirely on existing abnormal samples from past production, resulting in a lack of abnormal data for the target product before the new production line starts or the new product goes into mass production, making it impossible to train a generative model in advance, causing a data gap in early quality control; second, the synthetic samples lack diversity, with most of the generated abnormalities being repetitive known patterns in the training data, lacking the variability needed to train robust detection models, making it difficult to handle unexpected abnormalities in production; third, the model training efficiency is low, as the lack of abnormal data requires training a generative model for each specific abnormal category, making it impossible to utilize the potential connections between different abnormal types, significantly increasing the complexity of model development and maintenance; fourth, historical abnormal data cannot be reused across scenarios, wasting valuable historical data resources and limiting the model's ability to learn from a wider range of abnormal patterns.

[0006] In summary, existing technologies, whether traditional defect detection methods or improved solutions based on generative architecture, cannot address the core needs of pre-incident prevention, and have obvious shortcomings in sample diversity, data utilization, model efficiency, and technology implementation, making it difficult to meet the requirements of modern manufacturing for efficient and forward-looking quality assurance. SUMMARY

[0007] Therefore, the present application aims to solve the problems of traditional data generation methods in the industrial abnormal detection scenario, such as the inability to generate unknown abnormalities before the manufacturing of new production lines or new products, poor diversity of synthetic samples, low efficiency of model training and data utilization, and the difficulty of accurately aligning text descriptions and visual abnormal features in implementation, as well as the difficulty of balancing rationality and fidelity in generating abnormalities, thereby providing an abnormal sample generation system for pre-manufacturing quality control and an abnormal detection device.

[0008] Specifically, the abnormal sample generation system for pre-manufacturing quality control includes a data acquisition module, an abnormal reference retrieval module, and a unified abnormal synthesis module. The data acquisition module is used to acquire a historical abnormal data set and a normal image data set of a target product. The historical abnormal data set includes historical product categories, corresponding abnormal texts, and abnormal images. The normal image data set includes abnormal-free images of the target product to be controlled for pre-manufacturing quality control.

[0009] The abnormality reference retrieval module is configured to obtain potential abnormal texts of a target product and corresponding reference abnormal images from the historical abnormal data set based on a product category of the target product, and specifically includes:

[0010] A structured abnormal knowledge base is constructed, the knowledge base being a collection of multiple tuples, each tuple consisting of a historical product category and a set of known abnormalities of products of the category, the data source of the tuple being from the historical abnormal data set;

[0011] A target product category is used as a retrieval keyword to retrieve a preset number of similar historical products and their corresponding sets of known abnormalities in the structured abnormal knowledge base, obtaining a similar retrieval result;

[0012] The target product category and the similar retrieval result are integrated into a structured prompt, which is input into a preset large language model to generate a list of potential abnormal texts of the target product;

[0013] A CLIP model optimized by an abnormality perception triple loss is used to embed and encode each potential abnormal text in the list of potential abnormal texts and each abnormal image in the historical abnormal data set, calculate the cosine similarity between the text embedding of each potential abnormal text and the image embedding of each abnormal image, and filter reference abnormal images corresponding to each potential abnormal text according to the similarity;

[0014] The unified abnormality synthesis module is configured to generate a pre-manufacturing synthetic abnormal sample of the target product based on the reference abnormal images and the target product non-abnormal images in the normal image data set.

[0015] In an embodiment of the present application, the unified abnormality synthesis module generates a pre-manufacturing synthetic abnormal sample of the target product based on the reference abnormal images and the target product non-abnormal images in the normal image data set, specifically including:

[0016] A programmatic mask is set to specify the area of the target product non-abnormal image where the abnormality is to be generated;

[0017] The target product non-abnormal image, the reference abnormal image, the programmatic mask and random noise are fused by a preset improved diffusion model to generate an initial candidate abnormal image with reference abnormal features in the area specified by the programmatic mask;

[0018] The fidelity loss of the initial candidate abnormal image and the reference abnormal image in the latent space is calculated, and the latent variable in the generation process of the improved diffusion model is adjusted according to the fidelity loss to obtain an optimized candidate abnormal image;

[0019] The optimization candidate abnormal image is evaluated and screened to obtain a pre-manufacturing synthetic abnormal sample of the target product that meets a preset fidelity and rationality balance condition.

[0020] In an embodiment of the present application, the preset improved diffusion model takes the Stable Diffusion model as the basic architecture, which includes an added mapping layer and a cross-attention layer. The mapping layer is used to encode the reference abnormal image into a vector and adjust it to the same dimension as the text embedding vector of the Stable Diffusion model. The cross-attention layer is arranged in the UNet architecture of the Stable Diffusion model, which is used to make the text embedding vector fused with the reference abnormal image features cross-modal interaction with the image features output by the UNet architecture.

[0021] In an embodiment of the present application, when the fidelity loss of the initial candidate abnormal image and the reference abnormal image in the latent space is calculated , the following method is adopted:

[0022] The reference abnormal image and the region covered by the procedural mask in the initial candidate abnormal image are projected into the latent space by the pre-trained VAE encoder respectively to obtain the reference latent variable and the generated latent variable .

[0023] The mean square error of the reference latent variable and the generated latent variable in the procedural mask region is calculated as the fidelity loss , and its expression is as follows:

[0024] ,

[0025] wherein, is the generated image latent variable at the t-th step in the diffusion process, is the procedural mask, is the reference abnormal image latent variable projected by the pre-trained VAE encoder, is the element product operation.

[0026] In an embodiment of the present application, the update formula of the latent variable is: , is the latent variable updated by gradient correction, is a hyperparameter for controlling the updating degree, is the gradient of the fidelity loss with respect to ;

[0027] The total denoising step number of the preset improved diffusion model is T, and the latent variable Gradient correction is performed only at time steps where the denoising initial stage or the fidelity loss is greater than a preset threshold, and the corrected latent variable is input into the UNet architecture of the preset improved diffusion model to perform a denoising step, generating an image latent variable at the t-1 time step .

[0028] In an embodiment of the present application, the evaluation and screening of the optimization candidate abnormal image to obtain the pre-manufacturing synthetic abnormal sample of the target product satisfying the preset fidelity and rationality balance condition specifically comprises:

[0029] In the preset control latent variable update degree hyperparameter , a plurality of values are sampled in the value range of the preset control latent variable update degree hyperparameter , and a corresponding optimization candidate abnormal image is generated based on each

[0030] Based on the candidate image set, the fidelity and the degree of adaptation to the target product of each candidate image are scored, and all optimization candidate abnormal images are sorted in descending order according to the score results, and the top k candidate synthetic abnormal images in the score sorting are selected as the pre-manufacturing synthetic abnormal sample of the target product.

[0031] In an embodiment of the present application, the programmatic mask is a preset binary mask for limiting the abnormal generation position, and the area with a mask value of 1 is the abnormal generation area, and the area with a mask value of 0 is the normal reserved area.

[0032] In an embodiment of the present application, the tuple set of the structured abnormal knowledge base is expressed as:

[0033] ,

[0034] wherein, represents a historical product category, represents a known abnormality set corresponding to the historical product category, , represents the number of historical product categories, is the number of known abnormalities corresponding to a single historical product category.

[0035] In an embodiment of the present application, the calculation method of the abnormal perception triple loss is as follows:

[0036] Using the text embedding of the potentially anomalous text as the anchor point, the image embedding of the real anomalous image corresponding to the potentially anomalous text as the positive sample, and the image embedding of the irrelevant anomalous image as the negative sample, triples are constructed. ,in, For the first Anchor points for historical product categories, For the first Positive samples of historical product categories, For the i-th historical product category, there are negative samples;

[0037] Based on the triplet Calculate the anomaly-sensing triplet loss. The method is as follows:

[0038] ,

[0039] in, The number of triples. For marginal parameters, This represents the cosine similarity between vectors a and b.

[0040] Based on the aforementioned abnormal sample generation system for pre-manufacturing quality control, the present invention also provides an anomaly detection device, which includes the aforementioned abnormal sample generation system for pre-manufacturing quality control, a model building and training unit, and an anomaly detection unit.

[0041] The abnormal sample generation system for pre-manufacturing quality control is configured to generate pre-manufacturing synthetic abnormal samples of the target product.

[0042] The model building and training unit is configured to: build an anomaly detection model, and form a training set by combining the pre-manufacturing synthetic anomaly samples and the target product non-anomaly images, and optimize the parameters of the anomaly detection model using the training set to obtain the trained anomaly detection model;

[0043] The anomaly detection unit is configured to detect images of products in manufacturing that are acquired in real time based on the trained anomaly detection model, for quality control before manufacturing.

[0044] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0045] This invention possesses several core advantages through the construction of a dual-agent collaborative architecture consisting of an anomaly reference retrieval module and a unified anomaly synthesis module:

[0046] First, it breaks through the limitation of traditional methods that rely on abnormal data of target products after manufacturing. Without abnormal data of target products itself, it can generate synthetic abnormal images of target products before manufacturing by using normal images and reference information retrieved from a cross-scenario historical abnormal database, filling the data source gap of pre-manufacturing quality control and meeting the quality control needs of modern manufacturing industry for prevention in advance.

[0047] Second, it greatly improves the diversity of synthetic samples and the generality of the model. It can generate new types of abnormalities of target products using cross-product historical reference abnormalities, and does not need to train a generation model for a specific abnormality category, achieving the goal of one-class model adapting to multiple products and multiple abnormalities, and providing sufficient and diverse sample support for training robust downstream abnormality detection models.

[0048] Third, it effectively solves the problems of semantic drift and imbalance between fidelity and rationality in traditional generation models. It forces the generation of abnormalities in the latent space to be close to reference abnormalities, and uses a visual language model to automatically select the optimal candidate image, accurately balancing the consistency of generated abnormalities with reference abnormalities and the logical adaptability of generated abnormalities to target products.

[0049] Fourth, it takes into account the practicality and efficiency of industrial scenarios. It enhances the generation of historical abnormal data to reduce data collection costs, eliminates the need for category-specific training to reduce model development and maintenance complexity, avoids the waste of efficiency caused by excessive generation, and provides a practical technical solution for real-time quality control on industrial production lines, ultimately helping to improve the performance of downstream abnormality detection tasks. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to make the content of the present application easier to be clearly understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the accompanying drawings.

[0051] Figure 1 is a structural schematic diagram of an abnormal sample generation system for pre-manufacturing quality control provided in an embodiment of the present application;

[0052] Figure 2 is a flowchart of generating a pre-manufacturing synthetic abnormal sample of a target product by a system as shown in Figure 1 is a flowchart of generating a pre-manufacturing synthetic abnormal sample of a target product by a system as shown in

[0053] Figure 3 is a flowchart of an abnormal reference retrieval module retrieving potential abnormal text and corresponding reference abnormal images of a target product from a historical abnormal data set based on the category of the target product provided in an embodiment of the present application;

[0054] Figure 4is a flowchart of a process of generating a pre-manufacturing synthetic abnormal sample of a target product based on the reference abnormal image and the target product non-abnormal image in the normal image dataset by the unified abnormality synthesis module provided in the embodiments of the present application.

[0055] Description of the figures: 10, data acquisition module; 20, abnormal reference retrieval module; 30, unified abnormality synthesis module. DETAILED DESCRIPTION

[0056] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it. The embodiments are not limiting to the present application.

[0057] Embodiment one:

[0058] Referring to Figures 1-3 As shown in the drawings, the present application provides an abnormal sample generation system and an abnormality detection device for pre-manufacturing quality control, specifically comprising: a data acquisition module 10, an abnormal reference retrieval module 20 and a unified abnormality synthesis module 30.

[0059] The data acquisition module 10 is used to acquire a historical abnormal data set and a normal image data set of a target product, the historical abnormal data set includes historical product categories, corresponding abnormal texts and abnormal images, and the normal image data set includes non-abnormal images of the target product to be controlled for pre-manufacturing quality control;

[0060] The abnormal reference retrieval module 20 is used to acquire potential abnormal texts and corresponding reference abnormal images of the target product from the historical abnormal data set based on the target product category, specifically including:

[0061] A structured abnormal knowledge base is constructed, the knowledge base is a collection containing multiple tuples, each tuple is composed of a historical product category and a known abnormality set of the category product, and the data source of the tuple comes from the historical abnormal data set;

[0062] The target product category is taken as a retrieval keyword, a preset number of similar historical products and their corresponding known abnormality sets are retrieved in the structured abnormal knowledge base to obtain a similar retrieval result;

[0063] The target product category and the similar retrieval result are integrated into a structured prompt, input into a preset large language model, and a list of potential abnormal texts of the target product is generated;

[0064] The CLIP model optimized by the anomaly perception ternary tuple loss is used to respectively embed and encode each potential abnormal text in the list of potential abnormal texts and each abnormal image in the historical abnormal data set, calculate the cosine similarity between the text embedding of each potential abnormal text and the image embedding of each abnormal image, and obtain the reference abnormal image corresponding to each potential abnormal text according to the similarity.

[0065] The unified abnormal synthesis module 30 is configured to generate a pre-manufacturing synthetic abnormal sample of a target product based on the reference abnormal image and a target product normal image in the normal image data set.

[0066] In the application scenario of pre-manufacturing quality control of plastic bottles in a new production line, the target is to generate a synthetic abnormal sample of a plastic bottle before it is formally mass-produced and train an abnormal detection model to achieve pre-incident quality prevention.

[0067] Further, in the data acquisition module 10, the historical abnormal data set is selected as the Real-IAD data set, which contains 12 categories of industrial products (such as glass bottles, metal nuts, etc.), category texts, corresponding abnormal texts (such as pits, stains), and more than 2000 abnormal images. The images with missing labels or blurred images are removed from the Real-IAD data set, and samples with a completeness of ≥95% are selected, finally obtaining 8 categories of products, 10-15 types of abnormalities for each category, and a total of 1800 abnormal images as the knowledge base for abnormal reference retrieval and the historical abnormal library.

[0068] The normal image data set is selected as the MVTec-AD data set, and 100 normal images with uniform background and no defects (resolution is uniformly adjusted to 512x512) are selected from the plastic bottle category training set of the MVTec-AD data set as the abnormal target product images of the unified abnormal synthesis module 30. The test set is only used for performance evaluation of the subsequent abnormal detection model and does not participate in sample generation and model training.

[0069] The historical abnormal data set (containing 1800 abnormal images and 8 categories of product structured abnormal tuples) and the target normal image data set (100 plastic bottle normal images) are transmitted to the abnormal reference retrieval module 20.

[0070] Through data label comparison and image feature clustering, it is confirmed that there is no overlapping sample between the Real-IAD data set and the MVTec-AD data set, and there is no cross-relation between the product categories and abnormal types of the two data sets, so as to avoid affecting the objectivity of the detection model evaluation due to data leakage.

[0071] Further, the tuple set of the structured abnormal knowledge base constructed by the abnormal reference retrieval module 20 The mathematical expression is:

[0072] ,

[0073] wherein, represents a historical product category, such as a plastic bottle, a toothbrush; represents a set of known abnormalities corresponding to the historical product category, such as D1 = {“dimple”, “stain”, “bottle mouth crack”}, , represents the number of historical product categories, is the number of known abnormalities corresponding to a single historical product category. The knowledge base is stored in dictionary format, with the key being the product category and the value being the list of abnormality sets, supporting fast retrieval based on product categories.

[0074] The target product category is used as the retrieval keyword to retrieve a preset number of similar historical products and their corresponding known abnormality sets in the structured abnormality knowledge base, obtaining a similar retrieval result. For example, using the target product “plastic bottle” as the retrieval keyword, the cosine similarity algorithm is used to retrieve the top-3 similar products in the tuple set of the structured abnormality knowledge base, with the specific steps as follows:

[0075] The “plastic bottle” and the category text of the 8 product categories in the structured abnormality knowledge base are encoded into 768-dimensional vectors through the CLIP text encoder;

[0076] The cosine similarity of the text vector of the “plastic bottle” category and the text vectors of the other 8 product categories is calculated, and the top 3 similar products are selected: “glass bottle” (similarity 0.82), “plastic cap” (similarity 0.75), and “metal can” (similarity 0.68);

[0077] The abnormality sets of these 3 product categories are extracted to obtain the similar retrieval result D = { (glass bottle, {dimple, stain, bottle mouth crack}), (plastic cap, {scratch, deformation, missing corner}), (metal can, {corrosion, indentation, label offset})}.

[0078] The target product “plastic bottle” and the similar retrieval result D are integrated into a structured prompt, and the GPT-4o model is used to generate a list of potential abnormality texts, with the specific operations as follows:

[0079] The construction prompt is: "The known target object is a 'plastic bottle', its similar objects and corresponding abnormalities are as follows: 1. Glass bottle (dimple, stain, bottle mouth crack); 2. Plastic cap (scratch, deformation, missing corner); 3. Metal can (corrosion, indentation, label offset). Please combine the material (polyethylene), structure (bottle body, bottle mouth, bottle bottom) and production process (injection molding, blow molding) of the plastic bottle to list all possible abnormalities that may occur during production, and require specific and visual abnormality types."

[0080] Call the GPT-4o model to generate a list of potential abnormality texts for the plastic bottle Q = {“plastic bottle body yellow stain” “plastic bottle mouth scratch” “plastic bottle bottom indentation” “plastic bottle body deformation” “plastic bottle mouth missing corner”}.

[0081] Use the fine-tuned CLIP model to retrieve reference abnormal images corresponding to each abnormality text in the potential abnormality text list Q from the historical abnormality data set composed of the Real-IAD data set, with the following specific steps:

[0082] Take the text embedding of each potential abnormality text in the potential abnormality text list Q as the anchor point , and take the image embedding of the real abnormal image corresponding to the potential abnormality text (such as the “plastic bottle body yellow stain” corresponding to the “glass bottle stain” image in the Real-IAD data set) as the positive sample , and take the image embedding of an unrelated abnormal image (such as the “metal can corrosion” image) as the negative sample , construct 500 triplets ;

[0083] Set the margin parameter , use the abnormality perception triplet loss Fine-tune the CLIP encoder, and the loss calculation formula is as follows:

[0084] ,

[0085] where is the number of triplets, represents the cosine similarity between vector a and vector b, , is the Euclidean norm;

[0086] The attention map of the fine-tuned CLIP model changes from a state of no target dispersion to a state of focusing on the abnormal area. Encode the 5 abnormality texts in the potential abnormality text list Q and the 1800 abnormal images in Real-IAD using the fine-tuned CLIP model, calculate the cosine similarity between each text embedding and all image embeddings, and select the top 5 images in descending order of similarity as candidates, and then randomly select 1 as the final reference abnormal image for the abnormality text Finally, 5 reference abnormal images are obtained, such as "plastic bottle body yellow stain" corresponding to "glass bottle stain" image, and "plastic bottle mouth scratch" corresponding to "plastic cover scratch" image.

[0087] As shown in Figure 4 , the unified abnormal synthesis module 30 takes the normal plastic bottle image and the reference abnormal image as inputs to generate a pre-manufacturing synthetic abnormal sample of the target product through the logic of "reference condition diffusion model-fidelity gradient refinement-rationality-fidelity control", and the specific implementation is as follows:

[0088] A binary program mask M is generated using the Python OpenCV library, which is used to specify the area to be generated abnormal in the target product normal image :

[0089] The mask resolution is consistent with the target product normal image , and the area with mask value 1 is the abnormal generation area (according to the plastic bottle structure, set as the middle of the bottle body and the bottle mouth), and the area with mask value 0 is the normal reserved area (bottle bottom and bottle body edge);

[0090] Taking the Stable Diffusion 1.4 model as the basic architecture, an improved diffusion model is constructed, which includes a newly added mapping layer and a cross-attention layer. The mapping layer encodes the reference abnormal image into a 1024 vector through a ResNet-50 encoder, and then adjusts it to 768 dimensions through a linear mapping layer to adjust to the same dimension as the text embedding vector of the Stable Diffusion 1.4 model, replacing the placeholder vector of the prompt word in the original model to obtain a text embedding vector fused with reference features;

[0091] The cross-attention layer is set in the 3-5 layers of the UNet architecture of the Stable Diffusion 1.4 model, which is used to make the text embedding vector fused with the reference abnormal image features cross-modal interaction with the image features output by the UNet architecture, to ensure that the generated abnormal image matches the reference abnormal image in vision;

[0092] The improved diffusion model is used to fuse the target product normal image , the reference abnormal image , the program mask M and random noise ε (subject to normal distribution) to generate an initial candidate abnormal image with reference abnormal features in the area specified by the program mask M and normal features in the normal reserved area after a preset total denoising step T (50 steps) diffusion denoising. ;

[0093] calculating the initial candidate abnormal image with the reference abnormal image fidelity loss in latent space , and adjusting the latent variable in the improved diffusion model generation process according to the fidelity loss to obtain an optimized candidate abnormal image ;

[0094] evaluating and screening the optimized candidate abnormal image to obtain a pre-manufacturing synthetic abnormal sample of a target product that meets the preset fidelity and rationality balance condition .

[0095] Further, when calculating the fidelity loss in latent space of the initial candidate abnormal image and the reference abnormal image , the following method is adopted:

[0096] projecting the areas covered by the procedural mask in the reference abnormal image and the initial candidate abnormal image respectively to the latent space through the pre-trained VAE encoder to obtain reference latent variable and generated latent variable ;

[0097] calculating the mean square error of the reference latent variable and the generated latent variable in the procedural mask area as the fidelity loss , the expression of which is as follows:

[0098] ,

[0099] wherein, is the generated image latent variable at the t-th step in the diffusion process, is the procedural mask, is the reference abnormal image latent variable after projection by the pre-trained VAE encoder, is the element product operation.

[0100] Further, within the total denoising iteration cycle (the total step number is set to T) of the preset improved diffusion model, for each time step t, first determine whether the correction update trigger condition is met, i.e., in the early stage of denoising at time step or the fidelity loss is greater than the preset threshold:

[0101] If the condition is met, perform gradient correction on the latent variable at the current time step using the formula to obtain the gradient correction updated latent variable , a hyper-parameter for controlling the updating degree of the latent variable, the gradient of the fidelity loss with respect to ;

[0102] inputting the corrected latent variable into the UNet architecture of the preset improved diffusion model to perform a standard denoising step, to generate an image latent variable at the t-1 time step, the process strictly follows the conditional transition probability , c is a conditional set composed of a reference abnormal image and a programmed mask, and only depends on the corrected latent variable at the previous step, so as to maintain the Markov property of the diffusion process.

[0103] If the correction trigger condition is not met, the current latent variable is directly input into the UNet architecture of the preset improved diffusion model to perform a standard denoising step, to generate an image latent variable at the t-1 time step.

[0104] The iteration process of the gradient correction and the standard denoising continues until the T-step denoising is completed, and the latent variable at the terminal time step is mapped to the pixel space through the pre-trained VAE decoder, to obtain an optimized candidate abnormal image .

[0105] Specifically, in the embodiment, the optimized candidate abnormal image is evaluated and screened by the BLIP-2 visual language model, to obtain a pre-manufacturing synthetic abnormal sample of a target product that meets a preset fidelity and rationality balance condition, specifically including:

[0106] a plurality of values are sampled within a preset value range of a hyper-parameter for controlling the updating degree of the latent variable, a corresponding optimized candidate abnormal image is generated based on each value, and a candidate image set S is formed;

[0107] Based on the candidate image set, the fidelity and the degree of adaptation of each candidate image to the target product are scored, and all the optimized candidate abnormal images are sorted in descending order according to the score results, and the top k candidate synthetic abnormal images in the score sorting are selected as the pre-manufacturing synthetic abnormal sample of the target product .

[0108] For example, 5 values (0.005, 0.008, 0.01, 0.015, 0.02) are uniformly sampled within the range of , and 1 optimized candidate abnormal image is generated based on each value , form a candidate set S containing 5 images;

[0109] The prompt word "Please score from two dimensions (1-10 points): 1. The similarity (fidelity) of the abnormality (such as a yellow stain) in the image to the reference abnormality (glass bottle stain); 2. The physical rationality (such as whether the stain conforms to the characteristics of the plastic material) of the abnormality on the plastic bottle (rationality), and the final score is the weighted sum of the fidelity and the rationality, and the BLIP-2 visual language model is input to score each image in the candidate set S;

[0110] In descending order of the final score, the Top-1 image is selected as the final synthetic abnormal image (such as the "plastic bottle body yellow stain" image generated when η=0.01, with a score of 9.2).

[0111] Through the above steps, 100 synthetic abnormal images of each of the 5 types of plastic bottles before manufacturing are finally generated, a total of 500 synthetic abnormal samples, all of which have pixel-level masks (marking abnormal areas) for subsequent detection model training.

[0112] Embodiment Two:

[0113] The application also provides an abnormality detection device, which comprises the abnormal sample generation system for pre-manufacturing quality control, the model construction and training unit, and the abnormality detection unit described in Embodiment One.

[0114] The abnormal sample generation system for pre-manufacturing quality control is configured to generate pre-manufacturing synthetic abnormal samples of target products.

[0115] The model construction and training unit is configured to construct an abnormality detection model, and to form a training set by combining the pre-manufacturing synthetic abnormal samples and target product non-abnormal images, to optimize the parameters of the abnormality detection model through the training set, and to obtain a trained abnormality detection model.

[0116] The abnormality detection unit is configured to detect images of products in manufacturing in real time according to the trained abnormality detection model, for pre-manufacturing quality control.

[0117] Those skilled in the art will understand that the embodiments of the application can be provided as methods, systems, or computer program products. Therefore, the application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0118] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0119] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0120] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0121] Obviously, the above-described embodiments are only examples for clarity of description and are not limiting on the embodiments. Based on the above description, one of ordinary skill in the art can further make other different forms of changes or modifications. Here, all the embodiments are not required to be exhausted. The obvious changes or modifications derived from the above are still within the protection scope of the present application.

Claims

1. An abnormal sample generation system for pre-manufacturing quality control, characterized in that, include: The data acquisition module is used to acquire historical abnormal datasets and normal image datasets of the target product. The historical abnormal dataset includes historical product categories, corresponding abnormal texts and abnormal images. The normal image dataset includes images of the target product without abnormalities to be subject to pre-manufacturing quality control. The anomaly reference retrieval module is used to retrieve potential anomaly text and corresponding reference anomaly images of target products from the historical anomaly dataset based on the target product category. Specifically, it includes: Construct a structured anomaly knowledge base, which is a collection of multiple tuples. Each tuple consists of a historical product category and a set of known anomalies for that category of products. The data source for the tuples comes from the historical anomaly dataset. Using the target product category as the search keyword, a preset number of similar historical products and their corresponding known anomaly sets are retrieved from the structured anomaly knowledge base to obtain similar search results; The target product category and the similar search results are integrated into a structured prompt, which is then input into a preset large language model to generate a list of potential abnormal texts for the target product. The CLIP model, optimized by anomaly-aware triplet loss, is used to embed and encode each potential anomalous text in the potential anomalous text list and each anomalous image in the historical anomalous dataset. The cosine similarity between the text embedding of each potential anomalous text and the image embedding of each anomalous image is calculated, and reference anomalous images corresponding to each potential anomalous text are obtained based on the similarity. And a unified anomaly synthesis module, used to generate a pre-manufacturing synthesized anomaly sample of the target product based on the reference anomaly image and the target product without anomalies in the normal image dataset; Among them, the anomaly-sensing triplet loss The calculation method is as follows: Using the text embedding of the potentially anomalous text as the anchor point, the image embedding of the real anomalous image corresponding to the potentially anomalous text as the positive sample, and the image embedding of the irrelevant anomalous image as the negative sample, triples are constructed. ,in, For the first Anchor points for historical product categories, For the first Positive samples of historical product categories, For the i-th historical product category, there are negative samples; Based on the triplet Calculate the anomaly-sensing triplet loss. The method is as follows: ,in, The number of triples. For marginal parameters, This represents the cosine similarity between vectors a and b.

2. The abnormal sample generation system for pre-manufacturing quality control according to claim 1, characterized in that, The unified anomaly synthesis module generates a pre-manufacturing synthetic anomaly sample of the target product based on the reference anomaly image and the target product without anomalies in the normal image dataset, specifically including: Set a procedural mask, which is used to specify the area in the target product's image without anomalies where anomalies are to be generated; The target product's anomaly-free image, the reference anomaly image, the programmed mask, and random noise are fused using a preset improved diffusion model to generate an initial candidate anomaly image with reference anomaly characteristics in the region specified by the programmed mask. Calculate the fidelity loss between the initial candidate anomaly image and the reference anomaly image in the latent space, and adjust the latent variables in the generation process of the improved diffusion model according to the fidelity loss to obtain the optimized candidate anomaly image; The optimized candidate abnormal images are evaluated and screened to obtain pre-manufacturing synthetic abnormal samples of the target product that meet the preset balance between fidelity and rationality.

3. The abnormal sample generation system for pre-manufacturing quality control according to claim 2, characterized in that, The preset improved diffusion model is based on the Stable Diffusion model, which includes a newly added mapping layer and a cross-attention layer. The mapping layer is used to encode the reference anomalous image into a vector and adjust it to the same dimension as the text embedding vector of the Stable Diffusion model. The cross-attention layer is set in the UNet architecture of the Stable Diffusion model and is used to enable the text embedding vector that fuses the features of the reference anomalous image to perform cross-modal interaction with the image features output by the UNet architecture.

4. The abnormal sample generation system for pre-manufacturing quality control according to claim 2, characterized in that, The fidelity loss between the initial candidate anomaly image and the reference anomaly image in the latent space is calculated. When this happens, the following method shall be used: By using a pre-trained VAE encoder, the regions covered by the procedural mask in the reference anomaly image and the initial candidate anomaly image are projected into the latent space to obtain the reference latent variables. and generating latent variables ; The mean squared error between the reference latent variable and the generated latent variable in the programmed mask region is calculated as the fidelity loss. Its expression is as follows: ,in, Let be the latent variables of the generated image at step t in the diffusion process. For programmatic masking, For reference abnormal images The latent variables projected by the pre-trained VAE encoder, This is an element-wise product operation.

5. The abnormal sample generation system for pre-manufacturing quality control according to claim 4, characterized in that, The latent variables The update formula is: , For the latent variables updated after gradient correction, Hyperparameters for controlling the update rate, To preserve fidelity loss The gradient; The total number of denoising steps in the preset improved diffusion model is T, and the latent variables... Gradient correction is performed only at time step In the initial stage of denoising or loss of fidelity Execute when the value exceeds a preset threshold; corrected latent variables. The input is fed into the UNet architecture of the preset improved diffusion model to perform a denoising step, generating the image latent variables at time step t-1. .

6. The abnormal sample generation system for pre-manufacturing quality control according to claim 2, characterized in that, The process of evaluating and screening the optimized candidate anomaly images to obtain pre-manufacturing synthetic anomaly samples of the target product that meet the preset balance between fidelity and reasonableness conditions specifically includes: The hyperparameters controlling the degree of latent variable updates are preset. Sample multiple values ​​within the range Value, based on each The values ​​generate corresponding optimized candidate anomaly images, forming a candidate image set; Based on the candidate image set, the fidelity and suitability of each candidate image are scored, and all optimized candidate abnormal images are sorted in descending order according to the scoring results. The top k candidate synthetic abnormal images in the score ranking are selected as synthetic abnormal samples before manufacturing the target product.

7. The abnormal sample generation system for pre-manufacturing quality control according to claim 2, characterized in that, The programmed mask is a pre-set binary mask used to limit the location of anomaly generation. The area with a mask value of 1 is the anomaly generation area, and the area with a mask value of 0 is the normal preservation area.

8. The abnormal sample generation system for pre-manufacturing quality control according to claim 1, characterized in that, The tuple set of the structured anomaly knowledge base Represented as: ,in, Indicates historical product categories, This represents the set of known anomalies corresponding to this historical product category. , Indicates the number of historical product categories, This represents the number of known anomalies corresponding to a single historical product category.

9. An anomaly detection device, characterized in that, include: The abnormal sample generation system for pre-manufacturing quality control as described in any one of claims 1 to 8 is configured to: generate pre-manufacturing synthetic abnormal samples of the target product; The model building and training unit is configured to: build an anomaly detection model, and form a training set by combining the pre-manufacturing synthetic anomaly samples and the target product non-anomaly images, and optimize the parameters of the anomaly detection model using the training set to obtain the trained anomaly detection model; An anomaly detection unit is configured to detect images of products in manufacturing that are acquired in real time based on the trained anomaly detection model, for quality control before manufacturing.

Images