Mask r-cnn industrial product few sample defect detection method and system based on stable diffusion image generation and computer equipment

Abstract

The application discloses a MASK R-CNN industrial product few-sample defect detection method and system based on Stable Diffusion image generation and a computer device, relates to the technical field of industrial product defect detection, and mainly comprises the following steps: constructing a data set, training an improved image generation network model, training an improved classification detection model, performing industrial product defect category semantic segmentation, and measuring the actual damage range of different defects of a product. The application makes the model meet production requirements under the condition of few samples, realizes the purpose that image generation and defect detection are synchronously performed, and greatly improves the training efficiency and detection accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of industrial product defect detection technology, and more specifically to a method, system, and computer equipment for detecting few-sample defects in industrial products based on MASK R-CNN generated from StableDiffusion images. Background Technology

[0002] The stability of product supply and quality safety have a significant impact on market stability. To ensure product quality, it is necessary to pay attention to every stage of the product manufacturing process. Product appearance defects are one of the important indicators of quality inspection in industrial product manufacturing; therefore, the detection of product appearance defects is of significant research importance.

[0003] Traditional defect detection methods rely heavily on manual inspection, which is time-consuming, costly, subjective, and inefficient, making it difficult to meet the demands of rapid and accurate product inspection in large-scale production. In recent years, with the development of artificial intelligence, machine vision and deep learning-based inspection methods have been increasingly applied to product appearance defect detection. Machine vision overcomes the shortcomings of manual inspection, offering advantages such as high speed and non-destructive testing. However, machine vision-based methods typically require manual feature extraction and combination testing to select the optimal feature combination, a cumbersome process with relatively low model robustness. Deep learning methods, compared to machine vision, can automatically extract target features through convolutional neural networks. However, they require a large number of defect samples to train the model until it reaches the accuracy required for production. With increasingly sophisticated manufacturing processes and production management, this undoubtedly incurs significant costs and time in the sample collection phase.

[0004] Therefore, how to solve the problem of needing to collect various defect samples over a long period of time in existing technologies and reduce the model's dependence on the number of defect samples collected is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method, system and computer equipment for detecting defects in industrial products with few samples based on MASK R-CNN generated from Stable Diffusion images, which solves the problems existing in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A few-shot defect detection method for industrial products based on MASK R-CNN generated from stable diffusion images includes the following steps:

[0008] S1. Capture product defect images in a real industrial production environment to build an initial dataset;

[0009] S2. Input the initial dataset into the improved image generation model for training, and use the trained image generation model to generate new product defect samples.

[0010] S3. Merge the initial dataset with the new product defect samples into a mixed dataset, and train the improved classification and detection model based on the mixed dataset;

[0011] S4. Use the trained classification and detection model to segment the product defect image to obtain the product defect category discrimination result;

[0012] S5. Based on the detection results of various product defects, obtain the counting results and pixel values ​​of different types of defects; combine the camera parameters when collecting data, establish the correspondence between the pixel area occupied by the defect in the image and the area in the actual three-dimensional space, construct a mathematical mapping relationship model, and then realize the measurement of the damage range of various product defects.

[0013] Optionally, the specific steps of S1 are as follows:

[0014] S101. Build a data image acquisition platform and collect a small number of product images as raw image data;

[0015] S102. Crop the original image data to the size required by the improved image generation model to obtain the initial dataset;

[0016] S103. Use Labeling to label the different defects in the initial dataset.

[0017] Optionally, the specific steps of S2 are as follows:

[0018] S201. Use the Labelme tool to perform contour annotation on the product defect images, and convert the annotated file format into a txt file to form a segmentation training dataset.

[0019] S202. Based on the Stable Diffusion model framework, adjust the parameters to determine the basic training configuration, taking into account the characteristic that the segmentation training dataset is a grayscale image.

[0020] S203. Add a text-image correspondence module to the Stable Diffusion model with complete parameter configuration. The text-image correspondence module automatically polls the initial dataset labeled in S103 to make the text prompts correspond to different categories of defect data in S103. Continuously poll and evaluate the accuracy of defect category matching until the accuracy reaches the production requirements, then stop polling and fix the text-image correspondence relationship.

[0021] S204. Using the segmented training dataset and the image-text correspondence as joint inputs, start training the improved StableDiffusion image generation model. After continuous polling, generate new product defect samples.

[0022] Optionally, the specific steps of S204 are as follows:

[0023] During the forward phase, multiple noise addition operations are performed on the input product defect image and corresponding contour annotation information at preset time steps, with each noise addition step following a probability distribution. This continues until the input product defect image is completely converted into a noisy image. ;in, It is the noise variance at each step. yes t Images with added noise at any given time;

[0024] In the reverse phase, with a full-noise image Using the corresponding text prompts and contour annotations as references, a neural network is used to predict the denoised mean. and variance According to probability distribution From full noise image Inversely recovering defect features, and ensuring that the generated new product defect samples are consistent with the contour annotation information in terms of contour; among which, It is a Gaussian distribution;

[0025] Repeat the above forward noise addition-inverse noise reduction training process until the generated product defect samples meet the quality standards required for production, thus completing the model training.

[0026] Optionally, the specific steps of S3 are as follows:

[0027] S301. Merge the initial dataset with the new product defect samples into a mixed dataset, and divide it into a training set and a validation set in a 10:1 ratio;

[0028] S302. Improve the MASK R-CNN model by adjusting the input channel parameters of the backbone;

[0029] S303. Set appropriate batch size and input image size as training parameters. Train the improved MASK R-CNN classification and detection model by first using real samples in the training set and then gradually adding generated samples. During training, the model extracts candidate region features through RolAlign, completes defect classification regression and masking operations, and determines qualified products if no suspicious defects are detected.

[0030] Optionally, the specific steps for S5 are as follows:

[0031] S501. Based on the product defect category discrimination results, determine the number of defects, pixel values, and target bounding box area for each category. ; where the target box width Target bounding box height , , These are the maximum and minimum values ​​of the target bounding box width, respectively. , These are the maximum and minimum heights of the target bounding box, respectively.

[0032] S502. Determine the focal length of the camera used for data acquisition. f Shooting distance D and camera sensor size w×h Image resolution W×H ;in, w It is the physical width of the camera sensor, h It is the physical height of the camera sensor. W It is the horizontal resolution of the acquired image. H It is the vertical resolution of the acquired image;

[0033] S503. Establish the mathematical mapping relationship between the pixel area of ​​the defect image and the actual three-dimensional spatial area:

[0034]

[0035]

[0036]

[0037] In the formula: That is the actual width. This is the actual height;

[0038] S504. By using root mean square error, mean absolute error, coefficient of determination, and residual analysis, compare the differences between the model measurement results and the actual measurement data to verify the accuracy of the mapping relationship.

[0039] A MASK R-CNN industrial product few-shot defect detection system based on Stable Diffusion image generation, performing any of the methods described above, includes:

[0040] The data acquisition module is used to capture images of product defects in actual industrial production environments and build an initial dataset.

[0041] The image generation model training module is used to input the initial dataset into the improved image generation model for training, and then use the trained image generation model to generate new product defect samples.

[0042] The classification and detection model training module is used to merge the initial dataset with new product defect samples into a mixed dataset, and to train the improved classification and detection model based on the mixed dataset.

[0043] The semantic segmentation module is used to segment the product defect image to be identified using the trained classification and detection model, and obtain the product defect category discrimination result;

[0044] The actual damage range measurement module is used to obtain the count results and pixel values ​​of different types of defects based on the detection results of various product defects; combined with the camera parameters when collecting data, it establishes the correspondence between the pixel area occupied by the defect in the image and the area in the actual three-dimensional space, constructs a mathematical mapping relationship model, and then realizes the measurement of the damage range of various product defects.

[0045] A computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the instruction, program, code set, or instruction set being loaded and executed by the processor to implement any of the methods described above.

[0046] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method, system, and computer equipment for detecting few-sample defects in industrial products based on MASK R-CNN generated from StableDiffusion images, which has the following beneficial effects:

[0047] (1) This invention selects MASK R-CNN as the framework basis and integrates Stable Diffusion into MASK R-CNN. Under the condition of few samples, the model meets the production requirements and achieves the purpose of simultaneous image generation and defect detection, which greatly improves the training efficiency of the model. The multi-scale idea is adopted to improve the detection capability of small defects through multi-resolution fusion. The above improvements not only improve the detection accuracy while maintaining efficient computing performance, but also have stronger adaptability and generalization ability for small targets and complex shape defects, further improving the performance of the model in the actual application of product quality inspection.

[0048] (2) This invention proposes a collaborative mode of offline training and online evaluation. The online evaluation process does not require image generation, which greatly saves production computing power.

[0049] (3) The optimized semantic segmentation model in this invention has significant advantages in product defect detection. By classifying each pixel, semantic segmentation can accurately locate and distinguish different types of defects on the product surface, achieving fine-grained defect detection. Compared with traditional target detection methods, semantic segmentation can capture more complex and irregular defect shapes, and is particularly suitable for detecting subtle defects with unclear edges or varied shapes. In addition, semantic segmentation can effectively process large-scale high-resolution images, reducing false detections and missed detections while ensuring accuracy, and significantly improving the efficiency and reliability of product quality inspection.

[0050] (4) Based on the model output results, the present invention further realizes the measurement of defect damage area. Deep learning method has significant advantages in the measurement of product defect damage area. Through automated feature extraction and learning, it can accurately identify and quantify various defect morphologies on the product surface. Compared with traditional image processing methods, deep learning can more accurately segment and detect irregularly shaped defect areas and measure their area in real time, which greatly improves detection efficiency. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart of the MASK R-CNN method for detecting few-sample defects in industrial products based on Stable Diffusion image generation provided by the present invention;

[0053] Figure 2 Network structure diagram of the improved MASK R-CNN model provided by this invention;

[0054] Figure 3 This is a detailed structural diagram of the U-Net network provided by the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This invention discloses a few-shot defect detection method for industrial products based on MASK R-CNN generated from Stable Diffusion images, such as... Figure 1 As shown, it includes the following steps:

[0057] S1. Capture product defect images in a real industrial production environment to build an initial dataset;

[0058] S2. Input the initial dataset into the improved image generation model for training, and use the trained image generation model to generate new product defect samples.

[0059] S3. Merge the initial dataset with the new product defect samples into a mixed dataset, and train the improved classification and detection model based on the mixed dataset;

[0060] S4. Use the trained classification and detection model to segment the product defect image to obtain the product defect category discrimination result;

[0061] S5. Based on the detection results of various product defects, obtain the counting results and pixel values ​​of different types of defects; combine the camera parameters when collecting data, establish the correspondence between the pixel area occupied by the defect in the image and the area in the actual three-dimensional space, construct a mathematical mapping relationship model, and then realize the measurement of the damage range of various product defects.

[0062] To address the issues raised in the background section regarding the need for lengthy collection of various defect samples in existing technologies, based on... Figure 1 As shown in the steps, this embodiment reduces the dependence on the number of defects collected by improving the network model, and provides an integrated service for quality inspection tasks in the industrial product production chain.

[0063] Furthermore, in this embodiment, the specific steps of S1 are as follows:

[0064] S101. Build a data image acquisition platform and collect a small number of product images as raw image data;

[0065] S102. Data preprocessing: The original image data is cropped to the image size required by the improved image generation model to obtain the initial dataset;

[0066] S103. Training Image Labeling: Use Labeling to label different defects in the initial dataset.

[0067] Furthermore, in this embodiment, the specific steps of S2 are as follows:

[0068] S201. Image annotation: Use the Labelme tool to annotate the contours of product defect images, and convert the annotated files into txt format files to form a segmentation training dataset.

[0069] S202. Based on the Stable Diffusion model framework, adjust the parameters to determine the basic training configuration, taking into account the characteristic that the segmentation training dataset is a grayscale image.

[0070] Specifically, since the processed dataset is grayscale, the encoder input in the Stable Diffusion model is changed to (in_channels:1, out_channels:128, kernel_size=3, padding=1); for the text encoder module in the model, the relevant settings are n_vocab=49408, the longest sentence can be n_token=77, and the length of the expression vectors for the two embeddings is n_embd=768; the relevant settings for the latent space Y: the standard Gaussian distribution Z~N(0,1) is linearly transformed to generate a new Gaussian distribution Y~N(μ,σ) 2 Y was optimized, and for practical engineering considerations, in order to ensure training stability, Y was finally set to 0.18215.

[0071] S203. Add an image-text matching module to the Stable Diffusion model with completed parameter configuration. Limit the range of text prompts. The image-text matching module automatically polls the initial dataset marked in S103 to match text prompts with different categories of defect data in S103. Continuously poll and evaluate the accuracy of defect category matching until the accuracy meets production requirements. Stop polling and fix the image-text matching relationship. For example, if the product surface is dirty, it should only match defect image data with dirt in the dataset.

[0072] S204. Using the segmented training dataset and the image-text correspondence as joint input, start training the improved StableDiffusion image generation model. Through continuous polling, generate new product defect samples. This step, based on the labeled dataset and the defined text description, uses the improved image generation model to generate defect image data, evaluates the generated defect images, adjusts the text description, and through multiple iterations, ensures that the model can generate images that meet the requirements based on the text condition information.

[0073] Specifically, in this embodiment, the specific steps of S204 are as follows:

[0074] During the forward phase, multiple noise addition operations are performed on the input product defect image and corresponding contour annotation information at preset time steps (the amount of noise added in each step is different), and the noise addition in each step follows a probability distribution. This continues until the input product defect image is completely converted into a noisy image. ;in, It is the noise variance at each step (controlling the noise intensity). yes t Images with added noise at any given time;

[0075] In the reverse phase, with a full-noise image Using the corresponding text prompts and contour annotations as references, a neural network is used to predict the denoised mean. and variance According to probability distribution From full noise image Inversely recovering defect features, and ensuring that the generated new product defect samples are consistent with the contour annotation information in terms of contour; among which, It is a Gaussian distribution;

[0076] Repeat the above forward noise addition-inverse noise reduction training process until the generated product defect samples meet the quality standards required for production, thus completing the model training.

[0077] Furthermore, in this embodiment, the specific steps of S3 are as follows:

[0078] S301. Merge the initial dataset with the new product defect samples into a mixed dataset, and divide it into a training set and a validation set in a 10:1 ratio;

[0079] S302, reference Figure 2 , Figure 3 The MASK R-CNN model was improved by adjusting the input channel parameters of the backbone and resetting them to (in_channels:1, out_channels:128, kernel_size=3, padding=1).

[0080] S303. Set an appropriate batch size (batch=16) and input image size (imgsz=1024). Using 1024 as training parameters, the improved MASK R-CNN classification and detection model was trained by first using real samples in the training set and then gradually adding generated samples to continue training. During the training process, the model extracted candidate region features through RolAlign, completed defect classification regression and masking operations, and determined that the product was qualified if no suspicious defects were detected.

[0081] Furthermore, in this embodiment, the specific steps of S5 are as follows:

[0082] S501. Based on the product defect category discrimination results, determine the number of defects, pixel values, and target bounding box area for each category. ; where the target box width Target bounding box height , , These are the maximum and minimum values ​​of the target bounding box width, respectively. , These are the maximum and minimum heights of the target bounding box, respectively.

[0083] S502. Determine the focal length of the camera used for data acquisition. f (mm), shooting distance D (mm) and camera sensor size w ×h Image resolution W×H ;in, w It is the physical width of the camera sensor (mm). h It is the physical height of the camera sensor (mm). W It is the horizontal resolution (pixels) of the acquired image. H It is the vertical resolution (pixel) of the acquired image;

[0084] S503. Establish the mathematical mapping relationship between the pixel area of ​​the defect image and the actual three-dimensional spatial area:

[0085]

[0086]

[0087]

[0088] In the formula: That is the actual width. This is the actual height;

[0089] S504. By using root mean square error, mean absolute error, coefficient of determination, and residual analysis, compare the differences between the model measurement results and the actual measurement data to verify the accuracy of the mapping relationship.

[0090] and Figure 1 Corresponding to the method described above, this embodiment of the invention also provides a MASK R-CNN industrial product few-shot defect detection system based on Stable Diffusion image generation, used for detecting defects in industrial products. Figure 1 The specific implementation of the method, as provided in this embodiment of the invention, is a MASK R-CNN industrial product few-shot defect detection system based on Stable Diffusion image generation, which can be applied to computer terminals or various mobile devices, specifically including:

[0091] The data acquisition module is used to capture images of product defects in actual industrial production environments and build an initial dataset.

[0092] The image generation model training module is used to input the initial dataset into the improved image generation model for training, and then use the trained image generation model to generate new product defect samples.

[0093] The classification and detection model training module is used to merge the initial dataset with new product defect samples into a mixed dataset, and to train the improved classification and detection model based on the mixed dataset.

[0094] The semantic segmentation module is used to segment the product defect image to be identified using the trained classification and detection model, and obtain the product defect category discrimination result;

[0095] The actual damage range measurement module is used to obtain the count results and pixel values ​​of different types of defects based on the detection results of various product defects; combined with the camera parameters when collecting data, it establishes the correspondence between the pixel area occupied by the defect in the image and the area in the actual three-dimensional space, constructs a mathematical mapping relationship model, and then realizes the measurement of the damage range of various product defects.

[0096] This embodiment also proposes a computer device, including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the instruction, program, code set or instruction set is loaded and executed by the processor to implement the method described above.

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0098] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting defects in industrial products using a few samples based on MASK R-CNN generated from Stable Diffusion images, characterized in that, Includes the following steps: S1. Capture product defect images in a real industrial production environment to build an initial dataset; S2. Input the initial dataset into the improved image generation model for training, and use the trained image generation model to generate new product defect samples. S3. Merge the initial dataset with the new product defect samples into a mixed dataset, and train the improved classification and detection model based on the mixed dataset; S4. Use the trained classification and detection model to segment the product defect image to obtain the product defect category discrimination result; S5. Based on the detection results of various product defects, obtain the counting results and pixel values ​​of different types of defects; combine the camera parameters when collecting data, establish the correspondence between the pixel area occupied by the defect in the image and the area in the actual three-dimensional space, construct a mathematical mapping relationship model, and then realize the measurement of the damage range of various product defects. The specific steps of S1 are as follows: S101. Build a data image acquisition platform and collect a small number of product images as raw image data; S102. Crop the original image data to the size required by the improved image generation model to obtain the initial dataset; S103. Use Labeling to label the different defects in the initial dataset; The specific steps of S2 are as follows: S201. Use the Labelme tool to perform contour annotation on the product defect images, and convert the annotated file format into a txt file to form a segmentation training dataset. S202. Based on the Stable Diffusion model framework, adjust the parameters to determine the basic training configuration, taking into account the characteristic that the segmentation training dataset is a grayscale image. S203. Add a text-image correspondence module to the Stable Diffusion model with complete parameter configuration. The text-image correspondence module automatically polls the initial dataset labeled in S103 to make the text prompts correspond to different categories of defect data in S103. Continuously poll and evaluate the accuracy of defect category matching until the accuracy reaches the production requirements, then stop polling and fix the text-image correspondence relationship. S204. Using the segmented training dataset and the image-text correspondence as joint inputs, start training the improved StableDiffusion image generation model. After continuous polling, generate new product defect samples. The specific steps in S204 are as follows: During the forward phase, multiple noise addition operations are performed on the input product defect image and corresponding contour annotation information at preset time steps, with each noise addition step following a probability distribution. This continues until the input product defect image is completely converted into a noisy image. ;in, It is the noise variance at each step. yes t Images with added noise at any given moment; In the reverse phase, with a full-noise image Using the corresponding text prompts and contour annotations as references, a neural network is used to predict the denoised mean. and variance According to probability distribution From full noise image Inversely recovering defect features, and ensuring that the generated new product defect samples are consistent with the contour annotation information in terms of contour; among which, It is a Gaussian distribution; Repeat the above forward noise addition-inverse noise reduction training process until the generated product defect samples meet the quality standards required for production, thus completing the model training.

2. The MASK R-CNN method for detecting defects in industrial products with few samples based on Stable Diffusion image generation according to claim 1, characterized in that, The specific steps for S3 are as follows: S301. Merge the initial dataset with the new product defect samples into a mixed dataset, and divide it into a training set and a validation set in a 10:1 ratio; S302. Improve the MASK R-CNN model by adjusting the input channel parameters of the backbone; S303. Set appropriate batch size and input image size as training parameters. Train the improved MASK R-CNN classification and detection model by first using real samples in the training set and then gradually adding generated samples. During training, the model extracts candidate region features through RolAlign, completes defect classification regression and mask operation, and determines qualified products if no suspicious defects are detected.

3. The MASK R-CNN method for detecting defects in industrial products using few samples, based on Stable Diffusion image generation, as described in claim 1, is characterized in that... The specific steps of S5 are as follows: S501. Based on the product defect category discrimination results, determine the number of defects, pixel values, and target bounding box area for each category. ; where the target box width Target bounding box height , , These are the maximum and minimum values ​​of the target bounding box width, respectively. , These are the maximum and minimum heights of the target bounding box, respectively. S502, Determine the focal length of the camera used for data acquisition. f Shooting distance D and camera sensor size w×h Image resolution W×H ;in, w It is the physical width of the camera sensor, h It is the physical height of the camera sensor. W It is the horizontal resolution of the acquired image. H It is the vertical resolution of the acquired image; S503. Establish the mathematical mapping relationship between the pixel area of ​​the defect image and the actual three-dimensional spatial area: In the formula: This is the actual width. This is the actual height; S504. By using root mean square error, mean absolute error, coefficient of determination, and residual analysis, compare the differences between the model measurement results and the actual measurement data to verify the accuracy of the mapping relationship.

4. A MASKR-CNN industrial product few-sample defect detection system based on Stable Diffusion image generation, performing the method as described in any one of claims 1-3, characterized in that, include: The data acquisition module is used to capture images of product defects in actual industrial production environments and build an initial dataset. The image generation model training module is used to input the initial dataset into the improved image generation model for training, and then use the trained image generation model to generate new product defect samples. The classification and detection model training module is used to merge the initial dataset with new product defect samples into a mixed dataset, and to train the improved classification and detection model based on the mixed dataset. The semantic segmentation module is used to segment the product defect image to be identified using the trained classification and detection model, and obtain the product defect category discrimination result; The actual damage range measurement module is used to obtain the count results and pixel values ​​of different types of defects based on the detection results of various product defects; combined with the camera parameters when collecting data, it establishes the correspondence between the pixel area occupied by the defect in the image and the area in the actual three-dimensional space, constructs a mathematical mapping relationship model, and then realizes the measurement of the damage range of various product defects.

5. A computer device, characterized in that, It includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the instruction, program, code set, or instruction set being loaded and executed by the processor to implement the method as described in any one of claims 1-3.

Images