Industrial defect rating method, electronic equipment and storage medium

Abstract

The invention provides an industrial defect rating method, electronic equipment and a storage medium, and belongs to the field of industrial defect detection. The industrial defect rating method comprises the following steps: acquiring an image of a to-be-detected product; inputting the image of the product to be detected into a trained defect feature extraction model to obtain defect features; the defect features comprise an abnormal thermodynamic diagram, a feature map and boundary coordinates of candidate defect areas; based on the defect features, generating a defect region mask through a seed region growing algorithm based on a feature distance; based on the defect area mask, extracting a defect area image from the product image; and inputting the defect area image into an improved NaViT model to obtain an industrial defect rating result. According to the method, any defect type can be graded, the model performance is remarkably improved, and the method has a wider application scene.

Description

Technical Field

[0001] The present invention belongs to the field of industrial defect detection, and in particular relates to an industrial defect rating method, electronic equipment and storage medium. Background Art

[0002] During the production process, products may have various types of defects, which directly affect the overall performance of the product and its competitiveness in the market. Therefore, product defects have become one of the important factors affecting the production of industrial products.

[0003] Existing quality inspection methods, such as Figure 1 As shown, a "one-size-fits-all" approach is often adopted. After defective products are discovered, they are typically identified by professional quality inspectors or classified using detection methods tailored to the specific product defects. Products with minor defects are classified as acceptable, while others are reworked. While this approach improves production efficiency to a certain extent, its reliance on manual screening often leads to subjectivity and inconsistency in the inspection process. Furthermore, manual screening consumes significant human resources, showing significant potential for improvement. Therefore, defect grading has become a crucial step in ensuring product quality and optimizing production processes in industrial production. As an indicator reflecting the severity of product surface defects, the industrial defect grade can help companies optimize production processes, reduce rework and scrap, and thus improve production efficiency.

[0004] Numerous previous studies have focused on introducing detailed parameter standards for defect analysis and assessment. For example, some studies use the PACP standard to analyze pipeline defects, while others use measured equipment corrosion parameters to assess defect severity. These methods are effective for specific products and applications, but due to the diverse morphology and nature of defects, a single objective standard cannot comprehensively cover all possible defect types. Therefore, existing defect rating methods have limitations when dealing with diverse products and complex application scenarios.

[0005] With the development of deep learning and computer vision technologies, pre-trained defect classification networks have gradually become a popular research direction in the field of defect rating. However, when these pre-trained networks are trained using defect severity datasets, the performance is less than satisfactory. Studies have shown that traditional image classification methods face multiple bottlenecks and challenges when dealing with defect rating tasks. For example, defects need to be independently extracted from complex backgrounds, and even after cropping, background information irrelevant to the rating may still exist. In addition, the different sizes and resolutions of defects make it difficult to apply a unified pre-processing method to different defect images, resulting in difficulty in model training convergence. More importantly, defect rating is not a simple image classification task. There is a clear sequential relationship between different levels, which places higher demands on traditional classification methods.

[0006] In summary, existing industrial defect rating methods typically use deep learning techniques to rate specific defects. However, this approach is costly and difficult to generalize to other application scenarios. Consequently, when training defect rating networks using defect severity datasets from different scenarios, model performance often suffers, making them ineffective in completing rating tasks. Summary of the Invention

[0007] The purpose of the present invention is to solve the difficulties existing in the above-mentioned prior art and to provide an industrial defect rating method, electronic device and storage medium, which effectively solves the problem of poor performance of existing industrial defect rating models, can rate any defect type, and has a wider range of application scenarios.

[0008] The present invention is achieved through the following technical solutions:

[0009] A first aspect of the invention provides a method for rating industrial defects, the method comprising:

[0010] Obtain an image of the product to be inspected;

[0011] Inputting the image of the product to be inspected into a trained defect feature extraction model to obtain defect features; the defect features include an abnormality heat map, a feature map, and boundary coordinates of a candidate defect area;

[0012] Based on the defect features, a defect region mask is generated by a seed region growing algorithm based on feature distance;

[0013] extracting a defect area image from the product image based on the defect area mask;

[0014] The defect area image is input into the improved NaViT model to obtain the industrial defect rating result.

[0015] Preferably, the loss function in the improved NaViT model is:

[0016]

[0017] Among them, y i is the hot encoding of the true label, is the jth column of the last fully connected layer, x i is the input of the last fully connected layer corresponding to the i-th sample in the training data N, where N is the number of categories, Show x i The L2 normalized vector of Show x j The L2 normalized vector of .

[0018] Preferably, in the improved NaViT model, the attention weight is calculated based on the cosine similarity between the query vector and the key vector.

[0019] Preferably, the defect feature extraction model includes a target detection network and an anomaly detection network, and the image of the product to be detected is input into the defect feature extraction model to obtain defect features including:

[0020] Inputting the product image to be inspected into the object detection network to obtain the boundary coordinates of the candidate defect area;

[0021] The product image to be detected is input into the anomaly detection network to obtain the anomaly heat map and the feature map.

[0022] Preferably, the anomaly detection network is constructed using a Patch Distribution Modeling (PaDiM) model.

[0023] Preferably, the method for generating a defect region mask by using a seed region growing algorithm based on feature distance includes:

[0024] Normalize the regional anomaly probability map within each candidate defect area to obtain a normalized regional anomaly heat map;

[0025] According to a preset background point set threshold and a defect point set threshold, the pixel points in each of the abnormal probability maps of the region are divided into a defect point set, a background point set and a point set to be marked;

[0026] Calculating a feature mean of the candidate defect area based on the feature map;

[0027] Taking the points in the defect point set and the background point set as seed points, and calculating the characteristic distance between the neighboring pixels of the seed point and the seed point;

[0028] The region growth is performed based on the characteristic distance, and the region growth is terminated when a preset termination condition is met, thereby obtaining the defect region mask.

[0029] Preferably, the value range of the background point set threshold is 0≤θ1≤0.1, and the value range of the defect point set threshold is 0.9≤θ2≤1, where θ1 is the background point set threshold and θ2 is the defect point set threshold.

[0030] Preferably, the feature distance is the Euclidean distance between the neighborhood pixels of the seed point and the feature mean of the seed point.

[0031] According to a second aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the industrial defect rating method as described in any one of the above items.

[0032] A third aspect of the present invention provides a storage medium having a computer program stored thereon. The computer program runs on a computer, and when the computer program runs, the computer executes the industrial defect rating method as described in any one of the above items.

[0033] Compared with existing technologies, the present invention offers the following advantages: through defect feature extraction, an improved seed region growing algorithm, and an improved NaViT model, it effectively blocks background information while retaining key defect information, enabling the rating of any defect type. Compared with other models, the model presented in this invention significantly improves performance, enabling effective detection and rating of defects in various industrial products. Model training utilizes surface defect images, eliminating the need for segmentation and annotation, reducing data acquisition costs and enabling a wider range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other objects, features, and advantages of the present invention will become more apparent through a more detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and are not intended to limit the present invention.

[0035] Figure 1 The following is a schematic diagram of the existing quality inspection method;

[0036] Figure 2 Schematic diagram of a process for rating industrial defects according to some embodiments of the present invention;

[0037] Figure 3 This is a diagram showing the overall framework of an industrial defect rating method according to an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the principle of an industrial defect rating method according to an embodiment of the present invention;

[0039] Figure 5 The relationship diagram between the threshold change and the proportion of correctly classified points in the defect point set and the background point set;

[0040] Figure 6 A schematic diagram of a flow chart of a seed region growing algorithm according to an embodiment of the present invention;

[0041] Figure 7It is a scatter plot of the width and height distribution of the defect area;

[0042] Figure 8 These are some sample examples from the two datasets Spandex and Spindle;

[0043] Figure 9a This is the confusion matrix diagram of model A;

[0044] Figure 9b This is the confusion matrix diagram of model B;

[0045] Figure 9c This is the confusion matrix diagram of model C;

[0046] Figure 10 Example graph of the results of defect rating for two industrial products;

[0047] Figure 11 Schematic diagram of the structure of electronic devices according to some embodiments of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more apparent, exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0049] To address the problem that existing methods can only rate specific defects, the present invention proposes an industrial defect rating method that can rate any defect type and has a wider range of applications. The present invention is further described in detail below with reference to the accompanying drawings.

[0050] To facilitate understanding of the present invention, we first provide a detailed introduction to an industrial defect rating method disclosed in an embodiment of the present invention. The execution entity of the industrial defect rating method provided in the embodiment of the present invention is generally a computer device with certain computing capabilities. The computer device includes, for example: a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, etc. In some possible implementations, the industrial defect rating method can be implemented by a processor calling computer-readable instructions stored in a memory. For example, the execution entity of the industrial defect rating method can be an industrial defect rating system set up on a product production line.

[0051] The following reference Figure 2The following describes the industrial defect rating method according to some embodiments of the present invention.

[0052] Figure 2 Schematic diagram of the process of industrial defect rating method according to some embodiments of the present invention, such as Figure 2 As shown, the industrial defect rating method of the present invention includes at least the following steps S100 to S500.

[0053] Step S100: Acquire an image of a product to be inspected.

[0054] In the present invention, the image of the product to be inspected at least contains the product to be inspected. The image of the product to be inspected may be an original image captured by an image acquisition device, or an image obtained by preprocessing the original image captured by the image acquisition device. Preprocessing may include normalization, scaling, smoothing, and other processing. Preprocessing may also include extracting a portion of the image region containing the product to be inspected from the original image captured by the image acquisition device to obtain the image of the product to be inspected.

[0055] Step S200: Input the image of the product to be inspected into a trained defect feature extraction model to obtain defect features; the defect features include an abnormal heat map, a feature map, and boundary coordinates of a candidate defect area.

[0056] Specifically, the defect feature extraction model may include a target detection network and an anomaly detection network. The target detection network is used to detect the defect area of ​​the product based on the image of the product to be detected, and output a detection frame, that is, to obtain the boundary coordinates (positioning frame) of the candidate defect area.

[0057] The anomaly detection network is used to detect anomalies of industrial products based on the images of the products to be detected, and obtain anomaly heat maps and feature maps.

[0058] Anomaly Heatmap: The output of the anomaly detection network is essentially the distance between abnormal and normal features. For ease of processing, these distance values ​​are mapped to anomaly probabilities from high to low, resulting in an anomaly probability map, or heatmap, with values ​​ranging from 0 to 1. This map identifies the anomaly probability of each region in the product image being inspected and is used to assist in selecting the initial seed points for region growing in the seed region growing algorithm. Using the heatmap, pixels with a higher probability of anomaly can be identified as seed points, while pixels with a lower probability of anomaly are considered background.

[0059] Feature map: This map extracts the deep feature information of the image and provides a quantitative basis for judging the similarity between pixels during the region growing process.

[0060] Specifically, the target detection network and the anomaly detection network can adopt existing networks. For example, the target detection network can be implemented using Faster Region-based Convolutional Neural Network (Faster R-CNN), You Only Look Once (YOLO) or Single Shot MultiBox Detector (SSD), and is not limited to this, as long as it can output the target detection box. The anomaly detection network can adopt Convolutional Neural Network (CNN), Autoencoder (Autoencoder), Generative Adversanal Network (GAN) or Long Short-Term Memory (LSTM), and is not limited to this, as long as it can output anomaly probability map (heat map) and image feature map.

[0061] When using these models, you can choose the network model that performs best when trained on the product dataset to ensure the best rating effect.

[0062] In the present invention, defect features are extracted separately through two networks, thereby fully utilizing the advantages of the target detection network and the anomaly detection network: the target detection network provides rough but reliable defect candidate positioning information; the anomaly detection network provides solid data support for the region growing algorithm to select the starting point and subsequent region expansion through multi-scale anomaly information and detailed feature maps.

[0063] This design provides sufficient multi-dimensional prior information for the subsequent region growing algorithm through a dual-branch feature complementarity mechanism, reducing the positioning deviation that may be caused by a single information source, while improving the robustness and accuracy of the industrial defect rating system.

[0064] Step S300 : Based on the defect features, a defect region mask is generated by a seed region growing algorithm based on feature distance.

[0065] The seeded region growing algorithm is a classic image processing technique, mainly used for image segmentation and target recognition. Its basic principle is to determine one or more initial seed points, starting from one or more initial seed points, and gradually incorporate neighboring pixels into the growing region according to predefined similarity criteria until a specific stopping condition is met.

[0066] In the seed region growing algorithm, the seed point serves as the starting point for region expansion and directly affects the starting point and direction of the subsequent segmentation process. Therefore, the selection of seed points is of great significance. In this invention, in order to capture the region as accurately and continuously as possible, the seed point selection method is as follows:

[0067] Crop the heat map according to the positioning frame to obtain the point set H of the heat map of a single area b,Norm . b,Noem Divide the grid into g*g grids, take the center point of each grid, and form a point set S={(x i ,y j )|i=1,2,...,n,j=1,2,...,m}, where h0 and w0 are respectively b The height and width of the point set H are shown in the following formula (1). b,Norm Divided into three subsets:

[0068]

[0069] Where S represents H b,Norm All points on S D represents the defect point set, which has the highest probability value; S N represents the background point set, which has the lowest probability value; S U represents the unlabeled point set, that is, the points to be marked, θ1 is the background point set threshold, and θ2 is the defect point set threshold.

[0070] The points where the defect points are concentrated are used as seed points for region growing. This method of selecting seed points not only ensures that the region growing accurately captures the essential characteristics of the target area, but also effectively prevents over-growth or under-growth caused by local feature noise or errors, thereby ensuring the stability and accuracy of the overall segmentation results.

[0071] After determining the initial seed point, the target region is gradually expanded from the initial point to the global region through a subsequent region growing process. The performance of this process directly affects the accuracy and stability of the final segmentation result. Therefore, in the present invention, the region growing process iteratively merges neighboring pixels whose characteristic distance from the current seed point meets a preset similarity standard into the current growing region. This improves the accuracy of the segmentation result while fully utilizing the local continuity and structural information of the image, ultimately obtaining a complete and detailed description of the target region.

[0072] In the present invention, the characteristic distance diff between the neighboring pixels and the current seed point D (x ′ ,y ′ ) is defined as the feature distance between the neighborhood pixels and the feature mean corresponding to the current seed point.

[0073] Preferably, the characteristic distance is the Euclidean distance, which can be calculated by the following formula (2):

[0074] diff D (x ′ ,y ′ )=‖F b (x ′ ,y ′ )-μ Seed ‖ 2 (2)

[0075] Among them, (x ′ ,y ′ ) is the seed point (x,y)∈S D Neighborhood pixels, S D Denotes the defect point set, F b (x ′ ,y ′ ) is the neighborhood pixel (x ′ ,y ′ )’s eigenvector, i.e., eigenvalue, μ Seed is the feature mean corresponding to the current seed point.

[0076] Specifically, the neighborhood pixels may be four neighborhood pixels or eight neighborhood pixels. Preferably, the neighborhood pixels are eight neighborhood pixels.

[0077] During the region expansion process, by calculating the eight-neighborhood similarity of the pixels in the feature map, it can be ensured that the expansion process covers the defect area as accurately as possible.

[0078] In the present invention, the characteristic mean is the characteristic mean of the defect point set, which can be calculated by the following formula (3):

[0079]

[0080] Among them, |S D | is the total number of initial defect point sets, F b (x,y) is the feature vector of the pixel at coordinate (x,y), and the feature atlas F is used. B ={F [b] |b∈B} is obtained by segmenting the input feature map F using the detection box B output by the target detection network.

[0081] In the present invention, the criterion for determining the similarity between a neighboring pixel and the current seed point is that the characteristic distance between the neighboring pixel and the current seed point is less than or equal to the characteristic distance between the neighboring pixel and the background point. Preferably, the characteristic distance between the neighboring pixel and the background point is the Euclidean distance between the characteristic means corresponding to the neighboring pixel and the background point, which is defined as shown in formula (4):

[0082] diff N (x ′ ,y ′ )=‖F b (x ′ ,y ′ )-μ N ‖ 2 (4)

[0083] Among them, diff N (x ′ ,y ′ ) is the neighborhood pixel (x ′ ,y ′ ) and the characteristic distance between the background point, μ N is the characteristic mean of the background point set. N It can be calculated by the following formula (5):

[0084]

[0085] Among them, |S N | is the total number of initial background points, S N Represents the background point set.

[0086] In this paper, a seeded region growing algorithm based on feature distance is used to generate a defect region mask, effectively shielding background information while retaining the key defect information. Compared to traditional methods, this method utilizes the anomaly probability map and high-level semantic feature map output by a deep learning network to provide richer guidance information to assist in region expansion. This strategy can achieve more refined defect segmentation without relying on complex manual labeling.

[0087] Step S400: extracting a defect area image from the product image using the defect area mask.

[0088] The following methods can be used to extract defect area images from product images using defect area masks:

[0089] A filter matrix corresponding to the product image is generated based on the defect mask, where 1 represents a defect point and 0 represents a background point. The defect area image is filtered using the filter matrix, that is, the filter matrix is ​​multiplied by the elements at the corresponding position to obtain the defect area image.

[0090] Step S500: input the defect area image into the improved NaViT model to obtain the industrial defect rating result.

[0091] The Native Resolution Vision Transformer (NaViT) is a network capable of processing variable-resolution images while preserving their aspect ratio. Its advantages include: eliminating the need for strict image resizing, significantly reducing training costs; and maintaining high performance across a wide range of resolutions, while preserving both the original scale and detail of the image, ensuring high-performance adaptability of the model.

[0092] In view of the advantages of NaViT in maintaining image ratio and processing multiple resolutions, this paper improves on NaViT and proposes a Native Resolution Cosine Transformer (NCViT), which, combined with an ordered classification loss function, can analyze and rate defects of any resolution.

[0093] NaViT will first be described below.

[0094] NaViT is an efficient visual transformer model that supports native resolution input (without forcing it to be scaled to a fixed size). Its flexible sequence processing mechanisms (such as packing strategies) significantly improve training and inference efficiency. It mainly includes the following functional modules.

[0095] Patch Embedding: Directly processes images of original size and uses dynamic patch extraction technology to divide the input image into fixed-size image patches. The number of patches is dynamically calculated based on the actual image resolution. Each patch is mapped to a D-dimensional vector through a shared convolutional layer (or linear layer).

[0096] Sequence Packing: Through sequence packing, multiple image patch sequences are spliced ​​into a long sequence, and an attention mask is added to mark the sequence boundaries.

[0097] Position Embedding: This method uses a learnable relative position encoding to accommodate variable-length sequences. The embedding method is to directly add the relative position encoding to the D-dimensional vector obtained by block embedding. In addition, a learnable classification token (CLS token) is added before each image sequence for classification output.

[0098] Transformer Encoder: The Transformer Encoder is composed of multiple identical layers stacked together, each of which contains two submodules: a Multi-Head Self-Attention (MSA) mechanism, which calculates the correlation between each token in the sequence and other tokens to capture global context; and a Feed Forward Neural Network (FFN), which introduces nonlinear transformations based on self-attention to enhance the model's fitting capabilities. Layer Normalization (LN) and Residual Connections are used throughout the entire architecture.

[0099] Among them, the steps of the multi-head self-attention mechanism mainly include:

[0100] Each input vector (of dimension D) is mapped into a query (Q) vector, a key (K) vector, and a value (V) vector through three linear layers.

[0101] Split Q, K, and V into h heads, and calculate attention independently for each head: When calculating the attention score, each query vector is dot-producted with all key vectors to obtain a score, which is then normalized to an attention weight through the softmax function;

[0102] The outputs of the h heads are concatenated and mapped back to dimension D through a linear layer.

[0103] Classification Head: Classifies the classification tokens output by the Transformer encoder.

[0104] Since the defect rating task requires the model to not only identify the category of defects but also to accurately sort the severity of defects, and there is a strict order relationship between different levels, in this paper, the Orderloss function φ(x i ), in order to adjust the distance between different label feature vectors according to the order of defect levels, the Orderloss function is shown in the following formula (6):

[0105]

[0106] Among them, x i is the input of the last fully connected layer corresponding to the i-th sample in the training data N, where N is the number of categories, Show x i The L2 normalized vector of Show x j The L2 normalized vector of .

[0107] As shown in the following formula (7), the cross entropy loss function is:

[0108]

[0109] Among them, y i is the hot encoding of the true label, is the j-th column of the last fully connected layer (j∈[1,C], C is the number of classes).

[0110] Finally, the loss function of NCViT in the present invention is obtained, as shown in the following formula (8):

[0111]

[0112] This loss function fully considers the order of labels during its calculation. By quantifying the order differences between predicted and true labels, it effectively guides the model to capture detailed progressive relationships between labels, thereby enhancing the model's ability to recognize patterns associated with ordered labels. This approach enables the NCViT model to more sensitively detect subtle differences between defects of varying severity levels. Experiments have shown that combining Order Loss with Cross Entropy (CE) yields the most effective results.

[0113] In the Transformer's self-attention mechanism, the dot product operation between the query vector and the key vector is susceptible to variations in feature magnitude. In particular, in defect rating tasks, random variations in defect size can lead to significant deviations in the dot product result. Increased magnitude causes the softmax function to produce extremely small gradient outputs, leading to the vanishing gradient phenomenon.

[0114] To solve the above problem, in some embodiments of the present invention, cosine similarity can be used instead of dot product operation. Cosine similarity only focuses on the direction information of the vector and ignores the amplitude change. Therefore, the Softmax function no longer needs to rely on the feature amplitude for scaling. This maintains the stability of the value when calculating the attention weight, ensures the robustness of the attention distribution, and helps improve performance. As shown in the following formula (9), the cosine attention calculation formula is:

[0115]

[0116] Among them, M q The length of the row vector representing q, M k The vectorization of the length value of the row vector representing k, q represents the query vector, k represents the key vector, represents the vector division operation, Represents vector multiplication operation, v represents the value vector.

[0117] This paper provides a rating method for common industrial defects. By extracting defect features, employing an improved seed region growing algorithm, and employing the NCViT model, which is based on an improved NaViT model, this method effectively blocks background information while retaining key defect information. This method can be used to rate any defect type. Model training utilizes surface defect images without the need for segmentation and annotation, resulting in low data acquisition costs and a wider range of applications.

[0118] Next, refer to Figure 3 and Figure 4 A method for rating industrial defects according to an embodiment of the present invention will be described.

[0119] Figure 3 The overall framework diagram of the industrial defect rating method according to one embodiment of the present invention is shown. Figure 4 FIG. 1 is a schematic diagram showing the principle of an industrial defect rating method according to an embodiment of the present invention; FIG. Figure 3 As shown, the orange part is used for defect feature extraction, such as Figure 3 and Figure 4 As shown, the green part is used to generate a defect area mask, and the pink part is used to rate the defect severity using the NCViT model. The method of this embodiment is described in detail below in conjunction with this framework.

[0120] 1. Defect feature extraction.

[0121] The input image is passed through the pre-trained anomaly detection network and object detection network to locate defects and extract defect information for subsequent defect rating.

[0122] In this embodiment, the product image I is input into the object detection network O, and the boundary coordinates of the candidate defect area (i.e., the detection box) can be obtained as shown in the following formula (10):

[0123] B=f det (I) (10)

[0124] Where B represents the boundary coordinates of the detected defect area, and its accuracy is sufficient to constrain the subsequent region growing process, f det Represents target detection.

[0125] The target detection here can provide candidate regions with higher confidence for subsequent stages, thereby limiting the region growing process.

[0126] In this embodiment, the input product image I is processed by the anomaly detection network A to obtain the following formula (11):

[0127] H,F=f ano (I) (11)

[0128] Among them, H is the abnormal heat map, F is the feature map of the product image, and f ano Indicates anomaly detection.

[0129] In this embodiment, a Patch Distribution Modeling (PaDiM) model is used to construct an anomaly detection network.

[0130] The anomaly detection network extracts abnormal information from the image from another perspective, thereby providing more fine-grained guidance information for the feature-based seed region growth algorithm (FSRG).

[0131] 2. Defect area mask generation

[0132] The defect area mask is generated based on the extracted defect features through a seed region growing algorithm based on feature distance.

[0133] Before directly applying region growing, preprocessing is performed to meet the input requirements of the FSRG algorithm. The preprocessing objects include the input image's heatmap H and feature map F. During preprocessing, the corresponding heatmap and feature map are extracted from H and F, respectively, based on the annotation results of the detection boxes.

[0134] It should be emphasized that the heat map H covers the entire image. When selecting seed points later, it is necessary to manually adjust the threshold to determine the initial point set. However, the heat map set H extracted based on the detection box B B ={H [b] |b∈B} cannot guarantee that the probability value of each small image is in the range of 0 to 1. Therefore, the preprocessing also includes normalizing these heat map sets to ensure that all probability values ​​are mapped to the range of 0 to 1, thereby facilitating the stable application of the subsequent FSRG algorithm, as shown in the following formula (12):

[0135] H B,Norm ={Normalize(H [b] )|b∈B} (12)

[0136] Among them, H [b] H is a heat map of a single region cropped according to a single set of coordinates b of the detection frame. B,Norm Represents the normalized heat map set H B .

[0137] Before selecting seed points, two probability thresholds as preset parameters can be determined based on the ideal segmentation mask annotation of the dataset: the background point set threshold θ1 and the defect point set threshold θ2.

[0138] Exemplarily, the value range of the background point set threshold is 0≤θ1≤0.1, and the value range of the defect point set threshold is 0.9≤θ2≤1.

[0139] As the defect point set threshold decreases, the point set size increases, but it may include non-defect points, causing a shift in feature centers and impacting algorithm performance. Similarly, increasing the background point set threshold has a similar effect. Therefore, the appropriate threshold is selected within the range where the accuracy of the two thresholds is higher. The final threshold is determined by averaging the intervals with the highest correct classification ratios for each threshold.

[0140] Figure 5 This is the relationship diagram of the ratio of correctly classified points in the defect point set and background point set due to the change of threshold value. Figure 5 The horizontal axis is the threshold, the vertical axis is the correct classification ratio in the point set, the blue background and red background are the threshold intervals with higher classification accuracy, the blue curve represents the change in the correct classification ratio of the threshold θ1, and the red curve represents the change in the correct classification ratio of the threshold θ2. Figure 5 As shown in Figure 3, the change of threshold has a significant impact on the proportion of correctly classified points in the defect point set and the background point set.

[0141] like Figure 6 As shown, the specific steps of the seed region growing algorithm (FSRG) in this embodiment include the following steps S1 to S6.

[0142] Step 1: Input feature image, defect point set, background point set, unlabeled point set and background feature mean;

[0143] Step 2: Determine whether the entire feature map contains unmarked pixels. If so, go to step S3; otherwise, go to step S6.

[0144] Step 3: Take the point in the defect point set as the seed point, extract all pixels in its eight-connected neighborhood, and determine whether the pixel points in the surrounding eight neighborhoods belong to the unlabeled point set. If so, go to step S4; otherwise, go to step S5;

[0145] Step 4: Determine the similarity between the neighborhood pixel points and the seed point, and merge the neighborhood pixel points into the defect point set or background point set based on the judgment result.

[0146] Specifically, it is determined whether the characteristic distance between the pixel point in the neighborhood and the current seed point is less than or equal to the characteristic distance between the pixel point in the neighborhood and the background point, as shown in the following formula (13). If so, the point is merged into the defect point set and used as the seed point for the next growth; otherwise, the point is merged into the background point set.

[0147]

[0148] Step 5: End a round of growth and determine the algorithm termination conditions. The conditions include: Condition 1: All eight-connected neighborhoods of the defect point set in the current area have been classified as background points; Condition 2: All pixels in the entire feature map have been marked. If any of the conditions are met, go to step S6; otherwise, go to step S3.

[0149] Step S6: End, the region growing process will stop, at this time a complete segmentation of each target region can be obtained, and the final defect point set and background point set are output.

[0150] By strictly adhering to the preset thresholds and similarity metrics in these steps, we can effectively suppress misjudgments or missed detections caused by local noise or subtle variations, ensuring the precise location of segmentation boundaries. In each iteration of the region growing process, the newly added pixel is treated as part of the expanded region and its features are compared and classified with those of the surrounding eight-connected pixels, thereby continuously refining and expanding the boundaries of the target region. Ultimately, a defect region mask is generated based on the resulting defect point set.

[0151] To address the spatial inconsistencies that may arise from multi-view features, different branches are used to extract features to ensure geometric consistency at the regional level. An improved seed-based region growing algorithm (FSRG) is used to generate defect area masks. This algorithm generates defect masks based on the main defect information without training, shielding background information and retaining only the defect information.

[0152] 3. Defect Severity Rating

[0153] In this embodiment, the input image is processed using the NCViT network model, severity features are extracted, and finally the defect severity level is obtained using a multi-layer perceptron (MLP). This process is described in detail below.

[0154] First, due to the size variations of the detected defect areas, this embodiment imposes upper and lower limits on the size of the NCViT input images (e.g., image 1, image 2, etc.). Therefore, some images require padding to meet the required size requirements. The specific calculation method uses the same patch size to calculate the sequence length for each image, while limiting the upper limit of the input sequence based on hardware conditions. To avoid insufficient information due to too short a sequence length, a lower limit of 10 can be set.

[0155] Then, a Fourier transform-based feature embedding operation is performed on the input image to achieve position embedding. Specifically, Fourier basis functions (such as sine or cosine) can be used to generate position encoding. For details, see Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1(2021):99-106, so that the model can perceive the relative position of the image; and a learnable classification token (cls token) cls1, cls2, cls3... is added before each image sequence for classification output.

[0156] The severity feature extraction part includes layer normalization (Layer Norm) module, multi-head self-attention (

[0157] The Multi-head Self-Attention) module and the FeedForward neural network (FeedForward) module gradually extract deep features related to the severity of the defect by repeating these operations multiple times.

[0158] The layer normalization module is used to normalize the input data and adjust the mean and variance of the data to an appropriate range, which helps to accelerate model convergence, improve training stability, and prevent gradient vanishing or exploding problems.

[0159] The multi-head self-attention module allows the model to simultaneously focus on different parts of the input sequence in different representation subspaces, captures long-distance dependencies between image features, extracts image feature information from multiple angles, and enhances the model's ability to capture complex feature relationships.

[0160] The feedforward neural network module performs further nonlinear transformation on the features processed by the self-attention mechanism, and further extracts and transforms features through the multi-layer perceptron structure to increase the expressive power of the model.

[0161] See also Figure 4The output of the NCViT network model uses a multi-layer perceptron (MLP) to perform final classification processing on the extracted deep features, mapping the features to different defect severity levels (from 1 to 10) to obtain regional severity scores. Darker colors represent higher top-1 accuracy. The final output contains the predicted defect severity labels for each image, and the predicted labels are close to the true labels. The loss function uses the cross entropy loss (CE) function and the order loss function to measure the difference between the model's predictions and the true labels, guiding model training and optimization, so that the model's predicted defect severity levels are closer to the true values.

[0162] Furthermore, to prevent mutual interference between different instances, an additional self-attention mask is introduced, ensuring that each instance focuses only on its own features. To coordinate with the Order loss function, the labels of different examples are feature-adjusted, retaining the self-attention mask but removing the combined padding of different instances. Cosine attention is used instead of dot products to calculate the similarity of the attention features, and an ordered classification loss function is used to emphasize the sequential relationship of defect ratings. This enables analysis and rating of defects of arbitrary aspect ratios and resolutions.

[0163] To evaluate the defect rating method proposed in this paper, two datasets, spandex and spindle, were used for experimental verification. The spandex dataset contains 2,867 raw images from a spandex production line, each with a resolution of 512×512 pixels. The defect type is uniformly classified as oil stains. There are 1,657 defect-free images and 1,210 defect-prone images. In the experiment, a total of 2,628 defect regions with graded labels were identified, covering 10 categories. These regions were divided into 2,102 training samples, 263 test samples, and 263 validation samples. The spindle dataset contains 3,270 raw images of spindle products, each with a resolution of 512×512 pixels. The defect type is uniformly classified as stains. There are 1,485 defect-free images and 1,785 defect-prone images. A total of 2,560 defect regions with graded labels were identified, covering 10 categories. These regions were divided into 2,048 training samples, 256 test samples, and 256 validation samples. In addition, the differences in aspect ratio, resolution, and appearance of defect regions make the dataset more challenging. Figure 7 It is a scatter plot of the width and height distribution of the defect area. The horizontal axis is the width and the vertical axis is the height. Figure 7 As shown in the figure, the width and height distribution of the defect area has no obvious regularity, and there are extremely large or small area sizes. This phenomenon further verifies the limitations of existing preprocessing methods in dealing with complex real-world scenarios. Figure 8 Some sample examples from the dataset are shown.

[0164] The experiment implemented the method of the present invention under the PyTorch framework and conducted a systematic evaluation in a unified experimental environment.

[0165] The experimental hardware configuration is as follows: CPU Intel Core i9-13900 2.00GHz, GPU NVIDIA GeForce4090D 24GB and 32GB RAM memory.

[0166] The specific settings and parameters are as follows: Regarding the model structure, the Transformer module depth was set to 3 layers. With a patch_size of 32, the length of the defect region sequence was kept within 1000 due to hardware memory limitations. The loss function used a combination of OrderLoss and Cross Entropy (CE) to fully capture the ordered relationship between labels. The optimizer used was AdamW, with a learning rate of 0.0002 and a dropout of 0.1.

[0167] The training, validation, and test sets were strictly separated and non-overlapping. To enhance the diversity of the training samples, the data preprocessing phase employed cropping, horizontal flipping, and image normalization. The entire image was resized to 512×512 to preserve the original image size, consistent with the model's design principles. Defective areas cropped from the pre-processed original image were not resized.

[0168] The anomaly detection model uses the Padim model, and the coordinates output by the object detection model are annotated using the dataset. All experiments were conducted using the same software and hardware environment to ensure comparability across all metrics and maximize the model's potential for solving real-world industrial defect rating tasks.

[0169] In the experiment, Fast R-CNN was used as a benchmark, and the proposed method was systematically compared with it on two datasets. Table 1 shows a performance comparison with Fast R-CNN on the two datasets. The experimental results, as shown in Table 1, show that the proposed method demonstrates significantly superior performance on both datasets, particularly in terms of Top-1 accuracy (Top-1 acc) and Top-5 accuracy (Top-5 acc), which are significantly improved compared to Fast R-CNN. This result demonstrates that the proposed method has superior effectiveness and robustness in the defect rating task.

[0170] Table 1

[0171]

[0172] To evaluate anomaly detection algorithms, we use Pixel-AUROC (P-AUROC). Compared to other metrics, P-AUROC effectively reflects the accuracy of pixel-level heatmap predictions and is closely related to defect area segmentation. Image-AUROC and image-F1score are also retained for reference. Table 2 compares the performance metrics of different anomaly detection methods. As shown in Table 2, the PaDiM model achieves the highest P-AUROC score, indicating that this model's heatmap predictions at the pixel level are the most accurate, making it a more suitable pre-module for the FSRG algorithm.

[0173] Table 2

[0174] Anomaly Detection P-AUROC Image-AUROC image-F1score CFA 60.4 72.2 92.9 Uflow 75.7 80.6 93.4 EfficientAD 64.8 69.9 93.1 Patchcore 65.7 81.4 92.6 Reverse_distillation 70.8 79.9 92.9 PaDiM 77.7 80.5 93.8

[0175] In order to evaluate the mask generation strategy in the present invention, the FSRG algorithm is used to generate a mask of the defect area based on the features extracted by the previous two branches. Table 3 is a validation set performance table of the rating network under different strategies. Table 3 shows the impact of different mask generation strategies on the subsequent defect rating performance. Only Bbox in Table 3 refers to the bounding box generated by directly using the target detection box as input. Hotmap is an anomaly probability map generated by the anomaly detection method, and the image is binarized in combination with the segmentation mask output by the model to obtain a more accurate defect mask. The anomaly detection method selected is Padim, which has the best performance in Table 2. The results in Table 3 show that the defect area mask generated by the FSRG algorithm has the best performance in the subsequent defect rating.

[0176] Table 3

[0177]

[0178] In order to evaluate the performance of the NCViT proposed in this paper, given that the NaViT model has the unique advantage of preserving arbitrary aspect ratios and resolutions, to ensure the fairness and effectiveness of the experiment, only the original NaViT and the NCViT modified based on CosineTransformer (NaViT+cosT) were selected for comparative experiments. In this experiment, the experimental strategy adopted was: under the condition of a block size of 32, the input sequence length was controlled within 1000 (i.e., Seq_1000), and the input scale was regarded as equivalent to a 1024×1024 resolution image. To ensure fairness in the comparison, the input image size of other comparison models was set to 1024×1024. Table 4 is a performance comparison table of the original NaViT and NCViT on the Spandex dataset. Table 4 shows the comparison results on the Spandex dataset, among which NCViT achieved the best performance in Top-1 accuracy.

[0179] Table 4

[0180] method Input size Top-1acc Top-5acc Average precision Average recall NaViT Seq_1000 57.52 97.12 64.12 58.30 NCViT Seq_1000 59.40 96.13 65.44 59.79

[0181] Industrial images differ significantly from natural images. Their defects often manifest as irregular shapes and weak semantics, making it difficult to capture key information. Therefore, this paper proposes the NCViT model, which, while preserving the original scale information of industrial images, addresses the instability of the similarity matrix caused by inconsistent input sequence lengths through a cosine attention mechanism. Experimental results (Table 4) demonstrate that the improved NCViT achieves significant performance improvements over the original NaViT in defect rating tasks, demonstrating the effectiveness of preserving the original image scale and introducing the CosT module in processing industrial defect images.

[0182] The task of industrial product defect rating can be viewed as an ordered classification problem, in which different labels have hierarchical and unequal distance relationships. To more comprehensively reflect this characteristic, experiments were conducted to compare the effects of three loss functions on the model's prediction of defect severity levels: mean squared error (MSE), standard cross entropy (CE), and an improved CE+Orderloss that incorporates label order information. Table 5 shows the experimental results of different loss functions under the same rating network parameters. The results show that the model using CE+Orderloss outperforms the model using only MSE or CE in terms of Top-1 accuracy. This result verifies the hypothesis that there is an ordered relationship between defect severity and shows that the introduction of an ordered loss function helps the model more finely capture the subtle progressive characteristics between labels.

[0183] Table 5

[0184] Loss Function Top-1acc MSE 51.84 MSE+Distance 46.15 CE+Mean-Variance 57.25 CE+Orderloss 62.65

[0185] During the region growing process, feature vector similarity assessment plays a key role in the gradual expansion of the target region, from the initial point to the entire region. Its performance directly impacts the accuracy and stability of the final segmentation results. Table 6 shows the impact of masks generated by different feature distance calculation methods on defect rating performance. Cosine similarity and Euclidean distance are both effective methods for calculating one-dimensional vector similarity, but they focus on different aspects: the former focuses more on the directional similarity between vectors, while the latter focuses on the actual distance between vectors. Experimental results show that using Euclidean distance can improve the accuracy of defect rating results.

[0186] Table 6

[0187] Feature distance Top-1acc Cosine similarity 33.95 Euclidean distance 62.65

[0188] Since the division of the initial seed point set determines the subsequent growth process, the determination of the background point set threshold and the defect point set threshold is crucial. Table 7 shows the model performance test data under different threshold settings, which shows the relationship between threshold settings and model performance.

[0189] Table 7

[0190]

[0191] To evaluate the entire model framework of the proposed industrial defect rating method, three experimental configurations were set up: Model A, which lacks FSRG and NCViT but replaces the rating component with a vision transformer (ViT); Model B, which lacks FSRG; and Model C, the complete defect rating model of our invention. Comparing Model A with Model B demonstrates the need to generate segmentation masks for defect regions and exclude irrelevant background information for industrial product rating, while comparing Model B with Model C highlights the advantages of NCViT over ViT. Figure 9a is the confusion matrix diagram of model A, Figure 9b is the confusion matrix diagram of model B, Figure 9c This is the confusion matrix for Model C. The horizontal axis represents the predicted labels, and the vertical axis represents the true labels. The values ​​represent the regional severity scores, with darker colors indicating higher top-1 accuracy. Each row represents the proportion of predictions for a particular category. The confusion matrix shows that categories 4 through 7 are difficult to distinguish using Model A, which does not consider segmentation masks. While Model B improves performance for these categories, the top-1 accuracy for categories 1 and 7 remains poor. Model C improves performance for all categories overall, but there are still instances where category 7 is mistaken for category 9. This may be due to the feature distribution of the dataset. Figure 10The figure shows the defect rating results for two industrial products. The left side shows the complete product image, and the right side shows the mask of each defect region and the severity label predicted by the defect rating system and the true value. Table 8 compares the performance of the three model configurations, showing more detailed performance metrics for the three model configurations: top-1 accuracy (Top-1acc), top-5 accuracy (Top-5acc), average precision, and average recall. Experimental results demonstrate that the proposed method has significant performance advantages.

[0192] Table 8

[0193] Model Compare Top-1acc Top-5acc Average precision Average recall A No FSRG, Vit 38.45 92.93 40.12 40.71 B No FSRG 51.51 96.26 55.20 49.12 C The present invention 62.65 93.9 69.275 58.975

[0194] The test results of the above experiments show that the industrial defect rating method proposed in this invention uses automatic mask generation technology to effectively eliminate background interference and accurately extract and evaluate defect areas. Combining the FSRG algorithm based on anomaly detection with the advanced NCViT algorithm, it can fully capture global and local features when processing industrial defect images with irregular shapes and weak semantics, thereby achieving robust defect localization and rating. The experiment effectively verified the importance of excluding background information, retaining the original image scale, and introducing loss functions with label order information in improving model performance. Experimental results show that the NCViT model and the CE+Order loss function based on label order show significant advantages on the collected dataset.

[0195] Furthermore, it's important to note that during the production process, whether a defect is severe enough to warrant return to the factory for repair is often determined subjectively by factory managers based on their experience, making defect ratings somewhat subjective. To address this issue, after detecting and locating defect areas using the method presented in this paper, users still need to manually label the defect level, incorporating expert experience into the final decision.

[0196] An embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program executed by the processor. When the computer program is executed by the processor, the device equipped with the processor executes the industrial defect rating method described in any of the above embodiments.

[0197] Next, refer to Figure 11 An example electronic device 100 for implementing the industrial defect rating method according to an embodiment of the present invention is described below.

[0198] like Figure 11 As shown, the electronic device 100 includes a processor 110, a memory 120, and a communication interface 130. The processor 110, the memory 120, and the communication interface 130 can be interconnected and communicated via a communication bus 140 and / or other forms of connection mechanisms (not shown).

[0199] It should be noted that Figure 11 The components and structure of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as needed.

[0200] Optionally, the communication interface 130 may further include a transmitter and / or a receiver.

[0201] The processor 110 may be a microcontroller unit (MCU), a central processing unit (CPU), a digital signal processor (DSP), a single-chip microcomputer, an embedded device, or other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0202] The memory 120 can be various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM), cache memory, and synchronous dynamic random access memory (SDRAM). The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, and flash memory. The computer-readable storage medium may also store one or more computer program instructions, and the memory 120 may execute the program instructions to implement the industrial defect rating method described above in the embodiments of the present invention.

[0203] An embodiment of the present application further provides a storage medium having a computer program stored thereon. The computer program runs on a computer, and when the computer program runs, the computer executes the industrial defect rating method as described in any of the above embodiments.

[0204] Finally, it should be noted that the above technical solution is only one embodiment of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and it is not limited to the method described in the above specific embodiment of the present invention. Therefore, the method described above is only preferred and does not have a restrictive meaning.

Claims

1. A method for rating industrial defects, characterized by: The rating methodologies include: Obtain an image of the product to be inspected; Inputting the image of the product to be inspected into a trained defect feature extraction model to obtain defect features; the defect features include an abnormality heat map, a feature map, and boundary coordinates of a candidate defect area; Based on the defect features, a defect region mask is generated by a seed region growing algorithm based on feature distance; extracting a defect area image from the product image based on the defect area mask; The defect area image is input into the improved NaViT model to obtain the industrial defect rating result.

2. The industrial defect rating method according to claim 1, characterized in that: The loss function in the improved NaViT model is: Among them, y i is the hot encoding of the true label, is the jth column of the last fully connected layer, x i is the input of the last fully connected layer corresponding to the i-th sample in the training data N, where N is the number of categories, Show x i The L2 normalized vector of Show x j The L2 normalized vector of .

3. The industrial defect rating method according to claim 2, characterized in that: In the improved NaViT model, the attention weight is calculated based on the cosine similarity between the query vector and the key vector.

4. The industrial defect rating method according to claim 1, wherein: The defect feature extraction model includes a target detection network and an anomaly detection network. The image of the product to be detected is input into the defect feature extraction model, and the defect features obtained include: Inputting the product image to be inspected into the object detection network to obtain the boundary coordinates of the candidate defect area; The product image to be detected is input into the anomaly detection network to obtain the anomaly heat map and the feature map.

5. The industrial defect rating method according to claim 4, characterized in that: The anomaly detection network is constructed using the PaDiM model.

6. The industrial defect rating method according to claim 1, characterized in that: Methods for generating defect area masks using a seed region growing algorithm based on feature distance include: Normalize the regional anomaly probability map within each candidate defect area to obtain a normalized regional anomaly heat map; According to a preset background point set threshold and a defect point set threshold, the pixel points in each of the abnormal probability maps of the region are divided into a defect point set, a background point set and a point set to be marked; Calculating a feature mean of the candidate defect area based on the feature map; Taking the points in the defect point set and the background point set as seed points, and calculating the characteristic distance between the neighboring pixels of the seed point and the seed point; The region growth is performed based on the characteristic distance, and the region growth is terminated when a preset termination condition is met, thereby obtaining the defect region mask.

7. The industrial defect rating method according to claim 6, characterized in that: The value range of the background point set threshold is 0≤θ1≤0.1, and the value range of the defect point set threshold is 0.9≤θ2≤1, where θ1 is the background point set threshold and θ2 is the defect point set threshold.

8. The industrial defect rating method according to claim 6, characterized in that: The feature distance is the Euclidean distance between the neighborhood pixels of the seed point and the feature mean corresponding to the seed point.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the industrial defect rating method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores a computer program, which runs on a computer. When the computer program is run, it enables the computer to execute the industrial defect rating method according to any one of claims 1 to 8.

Images