A multi-scene product defect detection method based on a diffusion model

Abstract

The present application belongs to the field of computer vision, and particularly relates to a multi-scene product defect detection method based on a diffusion model. The disclosed method realizes a multi-scene product defect detection large model method based on a diffusion model, uses dynamic multi-scale feature fusion to improve the perception of targets of different sizes, and integrates all scenes into one model to improve the accuracy and generalization ability of the device defect detection large model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a multi-scenario product defect detection method based on a diffusion model. Background Technology

[0002] Computer vision is the science of enabling machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, followed by image processing to create images more suitable for human observation or transmission to instruments. Product defect detection is a subfield of computer vision, focusing on defective product detection solutions for industrial production scenarios. Industrial production typically encompasses various scenarios and diverse product types, including welded products, components, lenses, and other products of different sizes and characteristics. Traditional detection models are usually trained only for a single scenario, often requiring retraining when a new production line is introduced.

[0003] In recent years, with the rapid development of computer vision, methods to improve product defect detection have been proposed. However, multiple different models are often required for multiple scenarios, and the number of defect samples in many scenarios is extremely small, affecting the detection results. Therefore, how to design a large model that can perform defect detection with few samples and can play a role in a variety of different scenarios is an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-scenario product defect detection method based on a diffusion model, combining the image augmentation capability of the diffusion model with the multiple task functions of a large model.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A multi-scenario product defect detection method based on a diffusion model includes the following steps: S1. Data preprocessing and labeling; S2. Feature Extraction: A neural network with a self-attention mechanism is used to extract features from the image to obtain a feature pyramid; The neural network with attention mechanism uses the Swin-transformer model to extract image features. For each input image, it is divided into blocks. After four stages of feature extraction, it outputs four different sizes of features, namely P2, P3, P4 and P5. S3. Scene Classification: Use the fused feature map to perform preliminary classification of the product scene in the image to be detected; After dimensionality reduction of the P5 features via convolution, global average pooling is applied, followed by a fully connected layer for adjustment. , The scene categories are defined, and the scene classification probability distribution vector is output after activation by the Softmax function. The detection scenarios are initially classified. S4. Multi-scale feature fusion: Dynamic feature fusion is performed on the features output by the feature pyramid to obtain multi-scale fused features; S5. Defect Detection: Further classify defects in different scenarios; defect detection includes a classification module, a detection module, and a segmentation module. The classification module is used for tasks where a single target occupies a large proportion of the overall image and determines the target type; the detection module is used to detect targets with a small area in the image that need to be counted or extracted for subsequent processing and outputs the target bounding box; the segmentation module is mainly suitable for tasks that require area calculation or detection of whether the target is covered and outputs an image mask.

[0006] Preferably, the feature extraction module in step S2 includes image preprocessing: Calculate the gray-level histogram of the image and the cumulative histogram of the image, specifying the mapping relationship between the input gray levels and the output gray levels as follows: ; in The input image pixel gray levels, To output the pixel grayscale levels of the image, The grayscale value is The total number of pixels, The height of the input image. Given the width of the input image, output the gray level of each pixel in the image in a loop according to the mapping relationship.

[0007] Preferably, the neural network for the attention mechanism in step S2 uses a Swin-transformer model to extract image features: For each input image Divided into indivual Small pieces, each piece flattened, yield... For each stage of the feature map, except for the first stage which does not require feature segmentation, the second, third and fourth stages all require feature merging. The specific steps of feature merging are to divide the feature into 2×2 blocks, stack the channels of the 2×2 blocks, and then perform a 1×1 convolution. The height and width are reduced to half of the original, and the channels are expanded to twice the original. Each stage contains an even number of Swin-T blocks, which are first normalized using LN, as shown in the following formula: ; in For pixel feature values, The mean of a single sample feature. For variance, To prevent division by zero for the minimum value, Learnable scaling and offset parameters; The W-MSA module first divides the feature map into windows, each window being 7×7 in size, and then initializes it. The matrix is ​​then used to calculate the attention score, as shown in the following formula: ; in Let be the query matrix. For the key matrix, For the value matrix, For single-head dimension, This is a bias term.

[0008] Preferably, step S3 scene classification: for Features are reduced to dimensionality using 1×1 convolution. Then perform two more 3×3 convolutions with 2x downsampling to After global average pooling, the flattened output is a 1×128 vector, which is then adjusted by a fully connected layer. , The scene categories are defined, and the scene classification probability distribution vector is output after activation by the Softmax function. The detection scenarios are initially classified.

[0009] Preferably, step S4, multi-scale feature fusion, includes a feature routing module: first, it determines whether the task is a pure classification task based on the scene category output in step S3; if it is a pure classification task, it proceeds directly to step S5; if it is not a pure classification task, it first processes the features output by the Swin-T module. After 3×3 convolution and deconvolution, the feature map size is adjusted to the same value. Same size, that is .

[0010] Preferably, step S4, multi-scale feature fusion, includes a feature fusion module: for the classification result of step S3, the classification probability is multiplied by the corresponding class weight, and the sum of the products is as follows: ; in, for The class of Each weight value for Class classification probability; Features output by the feature routing module The weighted summation formula is as follows: ; in It is the final characteristic; After feature fusion, the final output size is Feature map Adjusted to using deconvolution and convolution. Initial size, output .

[0011] Preferably, step S5, defect detection, includes a classification task module: for the output of step S4... Dimensionality reduction to 1×1 convolution. The feature size is then adjusted by performing a 3×3 convolution without downsampling and two 3×3 convolutions with double downsampling. After global average pooling, the vector flattens to 1×256. After passing through two fully connected layers, the output is... The vector is activated by the Softmax function, and the output probability distribution is then calculated.

[0012] Preferably, step S5, defect detection, includes a detection task module, comprising: RPN stage: for the output of S4 Each feature point generates 9 anchor boxes, totaling... ,for Generate one feature point A score is used to predict whether the anchor box belongs to the foreground or background, and Use these data points as anchor coordinates, then sort them by score and select the top ones. The input RoI stage, called the candidate region, involves two types of losses: classification loss and anchor regression loss, as shown in the following formula: ; ; ; ; ; ; ; ; in , These are the coordinates of the center point of the target bounding box. , The target bounding box width and height, , , , These are the coordinates of the anchor center point and its width and height, respectively. These are the coordinates and width / height offsets of the horizontal bounding box. For RPN loss, Foreground and background classification loss, The loss function for the horizontal bounding box. For smoothing functions, For feature weight hyperparameters, For predicting categories, Based on the prediction , , , Calculated , To be based on reality , , , Calculated z is the independent variable of the function.

[0013] ROI stage: Mapping the predicted bounding boxes to feature maps Above, features of a fixed size are obtained according to proportions; bbox head phase: For each RoI region there is Features, flattened A long one-dimensional vector passes through two fully connected layers, then through two branch fully connected layers, one of which passes through... get One branch outputs a vector representing the classification result; the other branch outputs a 4×1 dimensional variable, where 4 variables represent the horizontal box offset. The loss at this stage mainly includes classification loss, level regression loss, and rotational regression loss, with the specific formulas as follows: ; ; ; in For the real category, For predicting categories, This represents the actual horizontal bounding box offset. To predict the horizontal target box offset, To detect the loss, For classifying losses, For the target bounding box loss, For feature weight hyperparameters, It is a smoothing function.

[0014] Preferably, for the corrected bounding boxes, the class with the highest classification score is selected, and those below a set threshold are directly discarded. Different classes are sorted in descending order of score, and the class with the highest score is selected as the benchmark class. Those with an intersection-union ratio (IU) with the benchmark class that exceeds a threshold are discarded, while those below it are temporarily retained. Then, the filtering is iterated until all target boxes are processed, and only the last remaining detection boxes are retained.

[0015] Preferably, step S5, defect detection, includes a segmentation task module, comprising: ROI stage: Mapping the predicted bounding boxes to feature maps Above, high-resolution spatial details are preserved proportionally to obtain fixed-size features; SEGM head stage: For each RoI region, obtain The features are preserved through multiple convolutional layers to maintain the spatial structure, and then adjusted to [the desired structure] through deconvolution. Then, the output is determined by the mask prediction branch. The feature map represents the classification probability distribution of each pixel; the ground truth mask of the original image is downsampled to... For each class in the original image mask and the classification image, calculate the binary cross-entropy loss using the following formula: ; in For the number of categories, As the current category number, The coordinates of the points on the graph are: coordinates The number of pixels is The probability of a class coordinates Are the actual pixels? kind; During reasoning Upsample to the original image, use bilinear interpolation to create a mask on the original image, and output the image.

[0016] Compared with the prior art, the beneficial effects of this application are as follows: The method disclosed in this invention realizes a large-scale model method for multi-scenario product defect detection based on a diffusion model. It uses dynamic multi-scale feature fusion to improve the perception of targets of different sizes, and integrates all scenarios into a single model to improve the accuracy and generalization ability of the large-scale model for device defect detection. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of a multi-scenario product defect detection method based on a diffusion model disclosed in an embodiment of the present invention; Figure 2 A diffusion model architecture diagram used in the preparation stage of embodiments of the present invention; Figure 3This is a diagram of the Swin-transformer architecture used in the feature extraction stage of this invention. Figure 4 This is a window change diagram of the SW-MSA module used in the feature extraction stage of this embodiment of the invention; Figure 5 This is an architecture diagram of the lightweight classification module used in the scene classification stage of this invention. Figure 6 This is a diagram illustrating the feature routing and feature fusion architecture used in the multi-scale feature fusion stage of this invention. Figure 7 This diagram illustrates the classification of tasks in an embodiment of the present invention. Figure 8 This is an example diagram illustrating the detection task in an embodiment of the present invention; Figure 9 This is a classification architecture diagram used in the product defect detection stage of this invention. Figure 10 This is a diagram of the detection segmentation architecture used in the product defect detection stage of this invention. Figure 11 This is a diagram illustrating the architecture of the special task module used in the product defect detection stage of this invention. Figure 12 This is an example diagram illustrating a text recognition task in an embodiment of the present invention; Figure 13 This is an example diagram illustrating the area percentage task in an embodiment of the present invention; Figure 14 This is an example diagram illustrating a counting task in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0019] A multi-scenario product defect detection method based on a diffusion model includes the following steps: S1. Data Preprocessing and Labeling: S11. Specific annotation method during training: For each image, a primary category is labeled as the scene classification. For each image, based on the component defect situation, corresponding annotations such as defect category, target box, and mask are applied. Semantic defects with the same meaning across different scenes are grouped into one defect category. For each scene class, multiple sets of parameters are preset: fusion weight parameters... This parameter will be used as the weight for feature fusion in step S4; the task type parameter is used by the subsequent detector to distinguish different tasks, including classification, detection, and segmentation; the special task parameter is a list used to mark the categories that require special tasks, including tasks such as area calculation, text recognition, quantity counting, and component anomaly detection.

[0020] S12. Diffusion Model Data Augmentation: For cases where the number of defect samples is small, background images and defects are selected to generate images to supplement the dataset. The diffusion model structure is attached. Figure 2 As shown, input a background image and various defect images, paste the defect or normal target onto the background, and use a diffusion model to redraw the area to make it more realistic.

[0021] S13. The model is deployed on the working machine and receives images from the camera.

[0022] S14. Perform a series of data preprocessing steps on each image uploaded to the working machine: cut or fill to the model input size. .

[0023] S2. Feature Extraction: A neural network with a self-attention mechanism is used to extract features from the image to obtain a feature pyramid.

[0024] S21. Image preprocessing: Due to the uncertainty of the assembly line shooting environment, a series of problems such as blurring may occur, thus affecting the accuracy of remote sensing target detection. Therefore, image preprocessing is required, mainly implemented using Python's OpenCV library, including the following steps: S21-1. First, calculate the grayscale histogram of the image using the hist function in the Matplotlib library, and then calculate the cumulative histogram of the image.

[0025] S21-2. The mapping relationship between input gray levels and output gray levels is specified as follows: ; in For input pixels, To output pixels, the above mapping relationship is equivalent to first obtaining the cumulative probability histogram of the grayscale histogram (i.e., Then, this range is expanded to between 0 and 255 to obtain the pixels of the output image.

[0026] S21-3. Output the grayscale level of each pixel in the image cyclically according to the mapping relationship. The above steps achieve histogram equalization, solving the low contrast problem caused by the small grayscale range of the image and improving the recognition effect.

[0027] S21-4. Detect image blur. For images that are too blurry to be recognized, output a blur error directly.

[0028] S22. Self-attention mechanism network, using the Swin-transformer model to extract image features. The Swin-transformer structure is attached. Figure 3As shown, for the input image Divided into indivual Small pieces, each piece flattened, yield... For the feature map, in each stage, except for the first stage which does not require feature block division, the second, third, and fourth stages all require feature merging. Feature merging divides the features into 2×2 blocks, stacks the channels of the 2×2 blocks, and then performs a 1×1 convolution. After this, the height and width are reduced to half of the original, and the channels are expanded to twice the original. There are an even number of Swin-T blocks in each stage. The blocks are first normalized by LN, as shown in the following formula: ; in The mean of a single sample feature. For variance, To prevent division by zero for the minimum value, These are learnable scaling and offset parameters.

[0029] Next, the feature map is processed by the W-MSA module. The W-MSA module first divides the feature map into windows, initializing Q, K, and V matrices within each window with the same size as the window. Then, the attention score is calculated using the following formula: ; Similar to the W-MSA module, the SW-MSA module's window will change, as shown in the attached image. Figure 4 As shown.

[0030] After four stages of feature extraction, four different sized features are output, namely: : , : , : , : .

[0031] S3. Scene Classification: Use fused feature maps to perform preliminary classification of product scenes in the images to be detected.

[0032] The lightweight scene classification module has the following structure: Figure 5 As shown, for Features are reduced to dimensionality using 1×1 convolution. Then perform two more 3×3 convolutions with 2x downsampling to After global average pooling, the flattened output is a 1×128 vector, which is then adjusted to 1×n through a fully connected layer, where n is the scene category. Finally, after activation by the Softmax function, the scene classification probability distribution is output, which performs preliminary classification of the detected scene.

[0033] S4. Multi-scale feature fusion: Dynamic feature fusion is performed on the features output by the feature pyramid to obtain multi-scale fused features.

[0034] S41. Feature-based routing module, its structure is shown in the attached figure. Figure 6 As shown, firstly, based on the scene category output in step S3, it is determined whether it is a pure classification task. If it is a pure classification task, proceed directly to step S5. If it is not a pure classification task, firstly, the features P2, P3, P4, and P5 output by Swin-T are subjected to 3×3 convolution and deconvolution to adjust the feature map size to be the same as the size of P3. Then, spatial attention is used to adjust the weight distribution of the convolutional feature maps; S42. The structure of the feature fusion module is as shown in the attached figure. Figure 6 As shown, for the classification result in step S3, the sum of the product of the classification probability and the corresponding class weight is calculated using the following formula: ; Where T i For the final weight, for The class of Each weight value for Class classification probability.

[0035] Subsequently, the features output by the feature routing module were analyzed. The weighted summation formula is as follows: ; After feature fusion, the final output size is The feature maps are adjusted to their initial sizes P2, P4, and P5 through deconvolution and convolution, and the output is... .

[0036] S5. Defect Detection: Further classify defects in different scenarios; defect detection includes a classification module, a detection module, and a segmentation module.

[0037] Defect detection comprises a main task module and special task modules. The main tasks include classification, detection, segmentation, and both detection and segmentation. The classification module is primarily suitable for tasks involving a single target that occupies a large portion of the overall image; it determines the target type: defective parts are discarded; incompletely processed parts are reprocessed. The detection module is mainly suitable for detecting targets with a small area in the image, requiring target counting or extraction for subsequent processing, and outputs the target bounding box. The segmentation module is mainly suitable for tasks requiring area calculation or detection of overlap, and outputs an image mask.

[0038] Special tasks include text recognition, counting, area calculation, and component anomaly detection. The text recognition module obtains the bounding box of the detection module's results, identifies the characters in the bounding box, and compares them with a preset format to determine if the component is correct. The counting module obtains the results from the detection module, counts special categories, compares them with a preset base number, and determines the defect type. The area calculation module obtains the results from the segmentation module, calculates the area ratio of specific categories, compares it with a threshold, and determines the defect type. The component anomaly detection module obtains the results from the detection module, compares the type and position of the center target box of the template image and the actual image, and determines the component defect type.

[0039] S51. Detector Routing Phase: Based on the task parameter set obtained from the classification results in step S3, the subsequent detector structure is determined, including the classification task module, detection task module, segmentation task module, and detection-segmentation task module, such as... Figure 7 and Figure 8 As shown.

[0040] S52. Classification Task Branches: Structure as shown in the appendix Figure 9 As shown, the output of step S4 Dimensionality reduction to 1×1 convolution. The feature size is then adjusted by performing a 3×3 convolution without downsampling and two 3×3 convolutions with double downsampling. After global average pooling, it is flattened into a 1×256 one-dimensional vector. After passing through two fully connected layers, it outputs a 1×n vector, where n is the number of classification categories for the scene. After activation by the Softmax function, it outputs the probability distribution.

[0041] The target detection and segmentation structure is shown in the attached figure. Figure 10 As shown.

[0042] S53. Object Detection Branch: RPN stage: Generate 9 Anchors for each feature point in the feature map output by S4, totaling... ,for Generate one feature point A score is used to predict whether the anchor belongs to the foreground or background, and Use 10 data points as anchor coordinates, then sort them by score and select the top N. The input RoI stage is called the candidate region. The loss at this stage includes two types: classification loss (foreground / background) and Anchor regression loss, as shown in the following formula: ; ; ; ; ; ; ; ; in These are the coordinates and width / height offsets of the horizontal bounding box. For RPN loss, Foreground and background classification loss, The loss function for the horizontal bounding box. It is a smoothing function.

[0043] ROI stage: Map the predicted target bounding box onto the feature map and obtain features of a fixed size according to the proportion.

[0044] Example: The original image has an anchor of size [size missing]. The feature map size is Then the Anchor is mapped to the feature map as follows: Assuming a fixed mapping to Size, then in Upper division One equal area is For each region, the feature value of the center point is taken, and bilinear interpolation is used.

[0045] bbox head phase: For each RoI region there is Features, flattened A long one-dimensional vector passes through two fully connected layers, then through two branch fully connected layers, one of which passes through... get One branch outputs a vector representing the classification result; the other branch outputs a 4×1 dimensional variable, where four variables represent the horizontal bounding box offset. The loss at this stage mainly includes classification loss, horizontal regression loss, and rotational regression loss, with the specific formulas as follows: ; ; ; in For the real category, For predicting categories, This is the horizontal bounding box offset, the calculation method of which has been given above. To detect the loss, For classifying losses, For the target bounding box loss, For feature weight hyperparameters, It is a smoothing function.

[0046] In the post-processing stage, for the corrected bounding boxes, the class with the highest classification score is selected. Those with scores below a set threshold are directly discarded. Different classes are sorted in descending order of score, and the class with the highest score is selected as the baseline class. Those with an intersection-union ratio (IU) with the baseline class that exceeds a threshold are discarded, while those with an IU that is lower are temporarily retained. Then, the filtering is iterated until all target boxes are processed, and only the last remaining detection boxes are retained.

[0047] S54. Branching: RPN phase: Shares the same RPN as S53.

[0048] ROI stage: The predicted target box is mapped onto the feature map, and high-resolution spatial details are preserved proportionally to obtain fixed-size features.

[0049] Example: The original image has an anchor of size [size missing]. The feature map size is Then the Anchor is mapped to the feature map as follows: Assuming a fixed mapping to Size, then in Upper division One equal area is Each sub-region takes the feature value of the center point, and uses bilinear interpolation for accurate sampling to preserve pixel-level spatial location and edge details, which are then used by the subsequent segmentation head for pixel-by-pixel mask prediction.

[0050] SEGM head stage: For each RoI region, obtain The features are preserved through multiple convolutional layers to maintain the spatial structure, and then adjusted to [the desired structure] through deconvolution. Then, the output is determined by the mask prediction branch. The feature map, where K is the number of classes, represents the classification probability distribution of each pixel; the original image's ground truth mask is downsampled to... For each class in the original image mask and the classification image, calculate the binary cross-entropy loss using the following formula: ; Where K is the number of categories. As the current category number, The coordinates of the points on the graph are: coordinates The number of pixels is The probability of a class coordinates Are the actual pixels? kind.

[0051] Post-processing stage: During reasoning, Upsample to the original image, use bilinear interpolation to create a mask on the original image, and output the image.

[0052] S55. Special Mission Phase: Overall structure as shown in the appendix. Figure 11 As shown, based on the scene category obtained in step S3, a parameter list for special tasks is obtained, specifically one list for detection tasks and one list for segmentation tasks. The length of the list is the number of categories for each task. Each parameter is a 5-bit binary number, where one bit represents whether it is a text recognition task, one bit represents whether it is the occupant in the area occupancy task, one bit represents whether it is the occupied in the area occupancy task, one bit represents whether it is a counting task, and one bit represents whether it is a component abnormality task. Example:

[0053] Text recognition task: For the target text to be recognized, the text localization contour and offset angle are pre-defined on the template image. All target boxes of this type output from stage S53 are retrieved. Using the center of the target box as the localization center, all pixels within the localization contour are selected. The text region is transformed to horizontal using an affine transformation. All text regions in this image are used as input to the CRNN model, with a size of [missing information]. K represents the number of targets, and the text result is output after passing through CRNN.

[0054] Area percentage task: For the two objectives that require calculating the area percentage, see the attached table. Figure 13 As shown in the figure, the tin area needs to occupy a certain proportion of the pad area. Retrieve the two types of masks output by the S54 stage, and calculate the ratio of the number of overlapping pixels to the number of pixels of the occupied type.

[0055] Counting task: For targets that need to be counted, see attached... Figure 14 As shown, calculate the number of solder balls, retrieve all target boxes of this type output by stage S53, and calculate the number.

[0056] Component abnormal task: For targets that need to be checked for incorrect components, all target boxes of this type output in stage S53 are retrieved and matched with the center points of all target boxes of this type in the template diagram. If no match is found, it is determined to be an incorrect or missing component. If the center point is significantly offset, it is determined to be an offset.

[0057] This method expands the number of images by using a diffusion model, effectively reducing the performance degradation caused by insufficient samples. It also generates multi-scale feature maps through feature fusion, performs scene classification, and subsequently adds a special task component to complete multi-scene tasks.

[0058] This method expands the number of images of defects with few samples in industrial scenarios by using a diffusion model, effectively reducing the performance degradation caused by insufficient samples. It also improves the model's ability to recognize small targets by generating multi-scale feature maps through feature fusion. Through scene pre-classification, parameter matching branch scheduling, and task-specific post-processing link layering, it achieves integrated processing of defect recognition, segmentation, measurement, and character recognition in multiple industrial scenarios, solving the pain point that traditional single models cannot adapt to multiple working conditions, multiple types of defects, and multiple auxiliary measurement tasks.

[0059] Tested in real-world industrial applications, this method achieved a 100% accuracy rate in classifying scenarios; the classification module is included in the appendix. Figure 7 In the welding task scenario shown, the defect detection rate reached 99.5%, and the defect classification accuracy rate reached 98%; for the attached... Figure 8 In detection tasks, the defect detection rate reaches 98%, and the accuracy of the target bounding box reaches 95% with an intersection-union ratio (IU) ≥ 0.5. In segmentation tasks, the IU reaches 92%. In special task scenarios, with additional... Figure 12 In text recognition tasks such as [list of tasks], the character recognition accuracy rate reached 99%; in the attached [list of tasks]... Figure 13 In area calculation tasks such as [specific tasks], the relative error in calculating the area percentage should be ≤3%; in the appendix Figure 8 In counting tasks such as [unspecified tasks], the overall average counting error does not exceed 0.1.

[0060] Regarding model inference efficiency, considering the integration of special tasks such as character recognition, defect count, and defect pixel area calculation with the same input image size, this model directly outputs quantization results through a dedicated post-processing parameter link, eliminating the need for multi-model cascade inference. This reduces end-to-end inference time by 42% compared to multi-model cascade schemes. Furthermore, since the model backbone is pre-trained on the initial multi-scene dataset, transfer training in new scenes is 50% faster than retraining a single-scene model.

[0061] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-scenario product defect detection method based on a diffusion model, characterized in that, Includes the following steps: S1. Data preprocessing and labeling; S2. Feature Extraction: A self-attention neural network is used to extract features from the image, resulting in a feature pyramid; The neural network with attention mechanism uses the Swin-transformer model to extract image features. For each input image, it is divided into blocks. After four stages of feature extraction, it outputs four different sizes of features, namely P2, P3, P4 and P5. S3. Scene Classification: Use the fused feature map to perform preliminary classification of the product scene in the image to be detected; After dimensionality reduction of the P5 features via convolution, global average pooling is applied, followed by a fully connected layer for adjustment. , The scene categories are defined, and the scene classification probability distribution vector is output after activation by the Softmax function. The detection scenarios are initially classified. S4. Multi-scale feature fusion: Dynamic feature fusion is performed on the features output by the feature pyramid to obtain multi-scale fused features; S5. Defect Detection: Further classify defects in different scenarios; defect detection includes a classification module, a detection module, and a segmentation module. The classification module is used for tasks involving a single target that occupies a large proportion of the overall image and determines the target type; the detection module is used to detect targets with a small area in the image that require target counting or extraction for subsequent processing and outputs the target bounding box; the segmentation module is used for tasks that require area calculation or detection of whether the target is covered and outputs the image mask.

2. The multi-scenario product defect detection method based on a diffusion model according to claim 1, characterized in that, The feature extraction module in step S2 includes image preprocessing: Calculate the gray-level histogram of the image and the cumulative histogram of the image, specifying the mapping relationship between the input gray levels and the output gray levels as follows: ； in The input image pixel gray levels, To output the pixel grayscale levels of the image, The grayscale value is The total number of pixels, The height of the input image. Given the width of the input image, output the gray level of each pixel in the image in a loop according to the mapping relationship.

3. The multi-scenario product defect detection method based on a diffusion model according to claim 1, characterized in that, In step S2, for each input image Divided into indivual Small pieces, each piece flattened, yield... For each stage of the feature map, except for the first stage which does not require feature segmentation, the second, third and fourth stages all require feature merging. The specific steps of feature merging are to divide the feature into 2×2 blocks, stack the channels of the 2×2 blocks, and then perform a 1×1 convolution. The height and width are reduced to half of the original, and the channels are expanded to twice the original. Each stage contains an even number of Swin-T blocks, which are first normalized by LN. The W-MSA module first divides the feature map into windows, each window being 7×7 in size, and then initializes it. The matrix is ​​then used to calculate the attention score, as shown in the following formula: ； in Let be the query matrix. For the key matrix, For the value matrix, For single-head dimension, This is a bias term.

4. The multi-scenario product defect detection method based on a diffusion model according to claim 3, characterized in that, Step S3 Scene Classification: For Features are reduced to dimensionality using 1×1 convolution. Then perform two more 3×3 convolutions with 2x downsampling to After global average pooling, the flattened output is a 1×128 vector, which is then adjusted by a fully connected layer. .

5. The multi-scenario product defect detection method based on a diffusion model according to claim 3, characterized in that, Step S4, multi-scale feature fusion, includes a feature routing module: First, based on the scene category output in step S3, it is determined whether it is a pure classification task. Pure classification tasks proceed directly to step S5; non-pure classification tasks first process the features output by the Swin-T module. After 3×3 convolution and deconvolution, the feature map size is adjusted to the same value. Same size, that is .

6. The multi-scenario product defect detection method based on a diffusion model according to claim 4, characterized in that, Step S4, multi-scale feature fusion, includes a feature fusion module: for the classification results of step S3, the classification probability is multiplied by the corresponding class weight, and the sum of the products is as follows: ； in, for The class of Each weight value for Class classification probability; Features output by the feature routing module The weighted summation formula is as follows: ； in It is the final characteristic; After feature fusion, the final output size is Feature map Adjusted to using deconvolution and convolution. Initial size, output .

7. The multi-scenario product defect detection method based on a diffusion model according to claim 3, characterized in that, Step S5, defect detection, includes a classification task module: This module processes the output of step S4. Dimensionality reduction to 1×1 convolution. The feature size is then adjusted by performing a 3×3 convolution without downsampling and two 3×3 convolutions with double downsampling. After global average pooling, the vector flattens to 1×256. After passing through two fully connected layers, the output is... The vector is activated by the Softmax function, and the output probability distribution is then calculated.

8. The multi-scenario product defect detection method based on a diffusion model according to claim 3, characterized in that, Step S5, defect detection, includes a detection task module, comprising: RPN stage: for the output of S4 Each feature point generates 9 anchor boxes, totaling... ,for Generate one feature point A score is used to predict whether the anchor frame belongs to the foreground or background, and Use these data points as anchor box coordinates, then sort them by score and select the top ones. The input RoI stage, called the candidate region, involves two types of losses: classification loss and anchor box regression loss, as shown in the following formula: ； ； ； ； ； ； ； ； in , These are the coordinates of the center point of the target bounding box. , The target bounding box width and height, , , , These are the coordinates of the anchor frame's center point and its width and height, respectively. These are the coordinates and width / height offsets of the horizontal bounding box. For RPN loss, Foreground and background classification loss, The loss function for the horizontal bounding box. For smoothing functions, For feature weight hyperparameters, For predicting categories, Based on the prediction , , , Calculated , To be based on reality , , , Calculated z is the independent variable of the function; ROI stage: Mapping the predicted bounding boxes to feature maps Above, features of a fixed size are obtained according to proportions; bbox head phase: For each RoI region there is Features, flattened A long one-dimensional vector passes through two fully connected layers, then through two branch fully connected layers, one of which passes through... get One branch outputs a vector representing the classification result; the other branch outputs a 4×1 dimensional variable, where 4 variables represent the horizontal box offset. The loss at this stage mainly includes classification loss, level regression loss, and rotational regression loss, with the specific formulas as follows: ； ； ； in For the real category, For predicting categories, This represents the actual horizontal bounding box offset. To predict the horizontal target box offset, To detect the loss, For classifying losses, For the target bounding box loss, For feature weight hyperparameters, It is a smoothing function.

9. The multi-scenario product defect detection method based on a diffusion model according to claim 8, characterized in that, For the corrected bounding boxes, the class with the highest classification score is selected. Those with scores below a set threshold are discarded. Different classes are sorted in descending order of score. The class with the highest score is selected as the baseline class. Those with an intersection-union ratio (IU) exceeding the threshold with the baseline class are discarded, while those with a IU below the threshold are temporarily retained. Then, the filtering is iterated until all target boxes are processed, and only the last remaining detection boxes are retained.

10. The multi-scenario product defect detection method based on a diffusion model according to claim 3, characterized in that, Step S5, defect detection, includes a segmentation task module, comprising: ROI stage: Mapping the predicted bounding boxes to feature maps Above, high-resolution spatial details are preserved proportionally to obtain fixed-size features; SEGM head stage: For each RoI region, obtain The features are preserved through multiple convolutional layers to maintain the spatial structure, and then adjusted to [the desired structure] through deconvolution. Then, the output is determined by the mask prediction branch. The feature map represents the classification probability distribution of each pixel; the ground truth mask of the original image is downsampled to... For each class in the original image mask and the classification image, calculate the binary cross-entropy loss using the following formula: ； in For the number of categories, As the current category number, The coordinates of the points on the graph are: coordinates The number of pixels is The probability of a class coordinates Are the actual pixels? kind; During reasoning Upsample to the original image, use bilinear interpolation to create a mask on the original image, and output the image.

Images