A surface defect detection method based on fine-grained prototype online learning

Abstract

This invention discloses a surface defect detection method based on fine-grained prototype online learning, belonging to the field of image processing and surface defect detection. The method includes: inputting the image to be tested into a surface defect detection model; outputting a fine-grained feature map using a pyramid scene parsing module; clustering multiple fine-grained feature vectors using an online clustering module; determining the fine-grained feature vector closest to each cluster center as the corresponding prototype vector; for each fine-grained feature vector, calculating the distance between the fine-grained feature vector and the corresponding prototype vector to obtain the anomaly score of the pixel in the image whose position matches the fine-grained feature vector; obtaining an image-level anomaly score using all anomaly scores; and determining that a surface defect exists in the image if the score is greater than a preset threshold. This invention directly obtains the anomaly score of an image region by measuring the distance between fine-grained features and prototype features, improving the accuracy of the model in detecting and locating surface defect regions.

Description

Technical Field

[0001] This invention belongs to the field of image processing and surface defect detection, specifically relating to a surface defect detection method based on fine-grained prototype online learning. Background Technology

[0002] Surface defect detection is a key task in computer vision. It involves dividing a digital image into different parts or regions based on defects. The goal is to detect and locate defective regions that deviate from normal areas, thereby separating these defective regions. Surface defect detection is currently widely used in aerospace, printing and packaging, and product yield inspection. Its importance is particularly prominent in product yield inspection. Surface defect detection identifies and locates pixels with defects in product images, obtaining information such as the shape and location of these defects to monitor product yield.

[0003] However, in modern automated production, surface defect detection still requires manual visual inspection. But the quality of manual visual inspection is affected by visual fatigue and subjective emotions, potentially leading to decreased accuracy and slow speed. Traditional surface defect detection technology based on computer vision features online control and real-time analysis. Inspection equipment integrating computer vision surface defect detection algorithms is gradually replacing traditional manual visual inspection methods to complete surface defect detection in industrial production. However, traditional computer vision surface defect detection methods are limited by the difficulty of image feature construction and stringent image quality requirements, thus failing to achieve satisfactory defect detection results.

[0004] In surface defect detection algorithms using deep learning technology, distance-based surface defect detection algorithms have improved the detection accuracy of traditional surface defects. However, existing distance-based surface defect detection algorithms obtain anomaly scores of image regions indirectly by assigning anomaly scores of coarse-grained features to obtain anomaly scores of fine-grained features, resulting in inaccurate pixel anomaly scores and poor defect region boundary localization. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a surface defect detection method based on fine-grained prototype online learning. The technical problem to be solved by this invention is achieved through the following technical solution: A surface defect detection model is obtained; wherein, the surface defect detection model includes a pyramid scene parsing module, a feature memory cache module, and an online clustering module connected in sequence; the surface defect detection model is pre-trained using several training images that do not contain surface defect regions; The image to be tested is input into the surface defect detection model, and the pyramid scene parsing module outputs a fine-grained feature map that integrates multiple scale features. The feature memory cache module is used to save the fine-grained feature map. The fine-grained feature map includes multiple fine-grained feature vectors. The online clustering module is used to cluster the multiple fine-grained feature vectors to obtain multiple cluster centers; the fine-grained feature vector closest to each cluster center is determined as the prototype vector corresponding to that cluster center; For each fine-grained feature vector, based on the distance between the fine-grained feature vector and the prototype vector corresponding to its cluster center, the anomaly score of the pixel in the image to be tested that matches the fine-grained feature vector is obtained, and the image-level anomaly score is obtained by using all the obtained anomaly scores. Determine whether the image-level anomaly score is greater than a preset threshold. If yes, determine that there is a surface defect in the image under test; otherwise, determine that there is no surface defect in the image under test.

[0006] The beneficial effects of this invention are: The solution provided in this invention extracts fine-grained feature maps from an image, calculates the metric distance between each fine-grained feature vector and the prototype vector obtained through online learning, and directly obtains the anomaly score of the pixel corresponding to the fine-grained feature vector at the original image location. This provides pixel-level anomaly scores, enabling precise localization of defect region boundaries and improving the model's localization performance in defect regions. Furthermore, the image-level anomaly scores obtained from all anomaly scores are used to determine the presence of surface defects. This invention can directly obtain the anomaly scores of fine-grained features in an image region, removing the constraint of the coarse-grained feature anomaly score redistribution strategy on the accuracy of defect region localization. It can directly obtain the anomaly scores of an image region and improve the accuracy of the model in detecting surface defect regions. Attached Figure Description

[0007] Figure 1 This is a schematic flowchart of a surface defect detection method based on fine-grained prototype online learning provided in an embodiment of the present invention; Figure 2 This is a main structural block diagram of the surface defect detection model provided in the embodiments of the present invention; Figure 3 A detailed structural diagram of the surface defect detection model provided in the embodiments of the present invention; Figure 4 This is a schematic diagram illustrating the training process of the surface defect detection model provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the prediction process of the surface defect detection model provided in an embodiment of the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] To facilitate understanding of the embodiments of the present invention, a brief introduction to the relevant prior art is given first.

[0010] Jihun Yi et al. presented a paper titled "Patch svdd: Patch-level svdd for anomaly detection and segmentation" at the 2020 Asian Conference on Computer Vision (ACCV). The method described in this paper attempts to extract finer-grained image patch features. It calculates the L2 distance between the test image patch features and its nearest normal image patch features to obtain an anomaly score for each patch. Pixels within the same patch are assigned the same anomaly score. Furthermore, the anomaly scores obtained at different patch scales are multiplied to obtain the final pixel anomaly score.

[0011] However, this method has a drawback: image patches are regular image regions, so some patches inevitably contain both normal and defective areas. In this case, the anomaly score of the image patch cannot represent the anomaly score of every pixel within that patch. Furthermore, since the choice of image scale is related to both detection accuracy and computational cost, the choice of image patch scale also affects the accuracy of surface defect detection. Therefore, this method of redistributing image patch scores to pixel scores is not conducive to accurately locating defect region boundaries.

[0012] Currently, existing methods for surface defect detection using anomaly scores assign anomaly scores from coarse-grained features to fine-grained features to obtain anomaly scores for image regions. In other words, existing distance-based surface defect detection methods assign anomaly scores from coarse-grained features to fine-grained features, resulting in all fine-grained features within the same coarse-grained feature having the same anomaly score. This fails to consider the differences in anomaly scores between defective and normal regions within the same coarse-grained feature. Such anomaly scoring reduces the accuracy of these methods in detecting surface defect regions.

[0013] In summary, existing surface defect detection algorithms employ a coarse-grained secondary allocation method for anomaly scores, leading to inaccurate pixel anomaly scores. This, in turn, limits the accurate location of defect region boundaries, resulting in low accuracy in defect region boundary localization. Therefore, this invention provides a surface defect detection method based on fine-grained prototype online learning. Figure 1 As shown, the method may include the following steps: S1, Obtain the surface defect detection model; The surface defect detection model includes a pyramid scene parsing module, a feature memory cache module, and an online clustering module connected in sequence; the surface defect detection model is pre-trained using several training images that do not contain surface defect regions.

[0014] To facilitate understanding of the embodiments of the present invention, the structure and training process of the surface defect detection model will be introduced first.

[0015] (I) Structure of Surface Defect Detection Model like Figure 2 As shown, the surface defect detection model includes a pyramid scene parsing module, a feature memory cache module, and an online clustering module. The structure of each part is described below.

[0016] (1) Pyramid Scene Analysis Module The pyramid scene parsing module includes: The module consists of a sequentially connected feature extraction submodule, a pyramid pooling submodule, a channel adjustment submodule, a bilinear interpolation upsampling submodule, a feature fusion extraction submodule, and a momentum update submodule; among which... The feature extraction submodule is a feature extraction network that includes multiple two-dimensional convolutional layers and multiple activation function layers; The pyramid pooling submodule is a pooling network containing a preset number of pooling layers of different sizes; wherein, the pooling kernel size of each pooling layer is different to represent different scales; the preset number can be set as needed, for example, it can be 4.

[0017] The channel adjustment submodule includes the preset number of... Convolutional networks with convolutional layers; The feature fusion and extraction submodule is a convolutional neural network containing a feature concatenation layer and multiple two-dimensional convolutional layers; The momentum update submodule includes a momentum updater. For a detailed explanation of the concept of a momentum updater, please refer to the relevant technical documentation; it will not be elaborated upon here.

[0018] (2) Feature memory cache module The feature memory cache module includes: The feature memory submodule and the pseudo-label memory submodule are connected sequentially; wherein the input end of the feature memory submodule is connected to the output end of the momentum update submodule.

[0019] (3) Online clustering module The online clustering module includes: K-means clustering submodule.

[0020] (4) Prototype memory cache module For the training process of the surface defect detection model, the structure of the surface defect detection model also includes a prototype memory cache module. However, the prototype memory cache module is not used in the process of using the trained surface defect detection model, so it may not contain the prototype memory cache module.

[0021] The input of the prototype memory cache module is connected to the output of the Kmeans clustering submodule, and the output of the prototype memory cache module is connected to the input of the pseudo-label memory submodule.

[0022] For a detailed structural diagram of the surface defect detection model described above, please refer to [link / reference]. Figure 3 Please note that the prototype memory cache module is indicated by a dashed line, meaning that the usage process (prediction process) may not include this module. The function of each module will be explained in detail later.

[0023] (II) Training process of surface defect detection model In one optional implementation, the training process of the surface defect detection model includes: Step a1: Preprocess several training images that do not contain surface defect regions to form a training set; Specifically, for different surface defect detection scenarios, several images without surface defect areas can be pre-collected as training images. As an optional implementation, this embodiment of the invention can use the existing MVTec dataset, pre-processed, as the training set. The MVTec dataset is an existing unsupervised defect detection dataset, which will not be described in detail here.

[0024] In a specific example, 3466 images without surface defect regions can be obtained from the MVTec dataset. After preprocessing, they form the training set, and 1888 images containing surface defect regions can be directly used to form the test set. This set is used to verify the detection effect of the trained surface defect detection model. The images in the test set contain pixel-level accurate defect region data labels to characterize the defect pixel regions in the images, and the images in the test set do not require preprocessing.

[0025] In this embodiment of the invention, preprocessing the training images includes uniformly adjusting each training image to a preset size to meet the input requirements of the model. The preset size can be set according to needs, such as 256*256.

[0026] In step a1, the dimensions of the fine-grained feature vectors and the prototype vectors generated by the clustering process in the subsequent fine-grained feature maps can also be set. For example, corresponding to the preset size of 256*256, the dimensions of the fine-grained feature vectors and the prototype vectors can be 32.

[0027] Furthermore, in step a1, the number of prototype vectors, which is the number of cluster centers, can be set according to the type of training set. For example, for texture training sets, the number of prototype vectors can be set to 5, and for item training sets, the number of prototype vectors can be set to 8. No specific restrictions are imposed here.

[0028] For ease of subsequent description, the training set can be used. This means that the test set can be used express, Includes several test images and corresponding defect area data labels .

[0029] Step a2: Input the images in the training set into the pyramid scene parsing module of the completed surface defect detection model. For the current iteration, output a fine-grained feature map fused with multiple scale features from the feature fusion extraction submodule of the pyramid scene parsing module. The training process for the surface defect detection model can be found in [reference needed]. Figure 4 The diagram illustrating the training process is provided below. This invention uses the current iteration as an example to explain the training process.

[0030] For details, please see Figure 4 Understood, step a2 may include the following steps: Step a21: Use the feature extraction submodule to extract features from the input image to obtain the corresponding original feature map; The feature extraction submodule comprises multiple two-dimensional convolutional layers and multiple activation function layers to extract features from the input image, obtaining the original feature map. The original feature map and the input image have the same size.

[0031] Step a22: Using the preset number of pooling layers of different sizes in the pyramid pooling submodule, the original feature map is downsampled at the corresponding scale to obtain the preset number of downsampled feature maps; Figure 4The pyramid pooling submodule can contain four two-dimensional pooling layers of different sizes. Pooling cores of different sizes are spliced ​​together in a pyramid structure to form the pyramid pooling submodule. The specific structure of the pyramid pooling submodule from bottom to top is: first two-dimensional pooling layer → second two-dimensional pooling layer → third two-dimensional pooling layer → fourth two-dimensional pooling layer; wherein, the step size of the first two-dimensional pooling layer is 2; the step size of the second two-dimensional pooling layer is 3; the step size of the third two-dimensional pooling layer is 4; the second step size of the fourth two-dimensional pooling layer is 6; and the number of channels in the four two-dimensional pooling layers is the same, 512.

[0032] Each two-dimensional pooling layer performs downsampling processing on the original feature map at a corresponding scale, ultimately resulting in four downsampled feature maps. By performing downsampling processing at different scales, features of different fine granularities can be extracted.

[0033] Step a23, using the preset number of channels in the channel adjustment submodule. The convolutional layer adjusts the number of channels in the preset number of downsampled feature maps to obtain the preset number of downsampled channel-adjusted feature maps. The channel adjustment submodule can have four parallel operations. Convolutional layer Convolutional layers can be represented using CONV, which consists of four... The convolutional layer can adjust the number of channels in the four downsampled feature maps output by the pyramid pooling submodule, resulting in a total of four downsampled channel-adjusted feature maps.

[0034] Step a24: The bilinear interpolation upsampling submodule is used to upsample the preset number of downsampling channels of the feature map respectively to obtain the preset number of interpolated feature maps with the same size as the original feature map. Specifically, the bilinear interpolation upsampling submodule performs bilinear interpolation upsampling on the feature maps after adjustment of the four downsampling channels, so that the four processed feature maps have the same size as the original feature maps, which are called interpolated feature maps.

[0035] Step a25: The original feature map and the preset number of interpolated feature maps are spliced ​​together using the feature splicing layer in the feature fusion extraction submodule, and the spliced ​​feature map is fused using multiple two-dimensional convolutional layers in the feature fusion extraction submodule to obtain a fine-grained feature map.

[0036] The multiple two-dimensional convolutional layers of the feature fusion and extraction submodule can be represented by CONV. The fine-grained feature map is fused with the original feature map and fine-grained features obtained from pooling at multiple different scales, and is represented in the form of a multi-dimensional matrix containing multiple fine-grained feature vectors.

[0037] Step a3: The momentum update submodule updates the fine-grained feature map output by the feature fusion extraction submodule in the current iteration based on the fine-grained feature map obtained before the current iteration, and sends the update result to the feature memory submodule for storage; The momentum update submodule can be understood as a feature update module. In multiple iterations, the momentum update submodule does not directly overwrite and save the fine-grained feature map output by the feature fusion extraction submodule in the current iteration. Instead, it updates the fine-grained feature map obtained in the previous iteration to save the feature information of the current iteration in a way that minimizes the differences, so as to reduce training oscillations caused by frequent updates of the prototype vector.

[0038] It should be noted that for the first iteration, the previously obtained fine-grained feature map is empty, and the update result of the momentum update submodule is the fine-grained feature map output by the feature fusion extraction submodule during the first iteration; for each iteration starting from the second iteration, the previously obtained fine-grained feature map is the fine-grained feature map obtained in the previous iteration.

[0039] The update result of the momentum update submodule is a fine-grained feature map updated based on the previous iteration. This fine-grained feature map is sent to the feature memory submodule for storage so that it can be used in subsequent steps.

[0040] Step a4: The Kmeans clustering submodule uses the Kmeans clustering method to cluster all fine-grained feature vectors in the fine-grained feature map stored in the feature memory submodule to obtain multiple cluster centers; for each cluster center, the fine-grained feature vector closest to the cluster center is determined as the prototype vector corresponding to the cluster center; all determined prototype vectors are sent to the prototype memory cache module. Specifically, in step a1, the number of prototype vectors, which is the number of cluster centers, has been pre-set according to the type of the training set. In step a4, all fine-grained feature vectors in the saved fine-grained feature map are clustered using K-means to obtain multiple cluster centers matching this number. It can be understood that each cluster center represents a class with multiple fine-grained feature vectors, meaning each fine-grained feature vector has its own cluster center.

[0041] For each cluster center, the fine-grained feature vector that is closest to that cluster center is determined as the prototype vector corresponding to that cluster center; therefore, each cluster center can obtain a corresponding prototype vector.

[0042] Step a5: Use the prototype memory cache module to save all the obtained prototype vectors, and send all the saved prototype vectors to the pseudo-label memory submodule as pseudo-label vectors for the current iteration, which are used to characterize the detection result of the current iteration. In this embodiment of the invention, the distance between the fine-grained feature vector and the cluster center is calculated as a score. The fine-grained feature vector with the smallest score calculated for each cluster center is used as the corresponding prototype vector, and also as the pseudo-label vector and detection result of the current iteration.

[0043] The detection result refers to the label predicted in the current iteration, which is the pseudo-label vector obtained in the current iteration, and is stored in the pseudo-label memory submodule.

[0044] Among them, the pseudo-label vector of the current iteration is also used as the detection result of the current iteration by the variable update submodule for the next iteration.

[0045] Step a6: Based on the detection results of the current iteration, the backpropagation method is used to calculate the gradient of the network parameters through a preset loss function, and the stochastic gradient descent method is used to update the network parameters of the surface defect detection model using the obtained network parameter gradient. The formula for calculating the preset loss function is as follows: (1); in, Indicates the mean square error loss; Indicates prototype loss; Indicates the preset weight; (2); (3); (4); in, , Representing the height and width of the input image respectively; This indicates that the feature memory cache module stores fine-grained feature maps. Composed of fine-grained feature vectors; The first fine-grained feature map to be saved Line number The elements of a column refer to the corresponding fine-grained feature vectors; Represents fine-grained feature vectors The prototype vector corresponding to the cluster center; Represents the prototype vector Corresponding to all fine-grained feature vectors and prototype vectors in the cluster The degree of dispersion between them; Represents the number of prototype vectors, and is a natural number greater than 0; Indicates the first One prototype vector; Represents the prototype vector Corresponding to all fine-grained feature vectors and prototype vectors in the cluster The degree of dispersion between them; This indicates a search for the 2-norm; Indicates belonging to the prototype vector The number of fine-grained feature vectors in the corresponding cluster; Indicates belonging to the prototype vector The fine-grained feature vectors in the corresponding clusters; Represents the prototype vector Corresponding to all fine-grained feature vectors and prototype vectors in the cluster The degree of dispersion between them; Formula (4) is used to calculate and The general formula for prototype vectors for Time calculation prototype vector for Time calculation ;calculate hour, and according to Confirm, calculate hour, and according to Sure.

[0046] Specifically, use Calculate the mean squared error loss between the extracted fine-grained features and the corresponding pseudo-label vectors. Using... Calculate the prototype loss between the pseudo-label vector and the prototype vector, and finally obtain the weighted sum. .

[0047] Regarding the first In the next iteration, the backpropagation method is used, and through... The gradients of the network parameters of the surface defect detection model are calculated, and then the stochastic gradient descent method is used to apply the gradients of the network parameters of the surface defect detection model to the network parameters. The model is updated to obtain the unsupervised abnormal behavior detection network model for the current iteration. This means that the surface defect detection model uses the detection result at this time as... The updated formula is: (5); (6); (7); in, For the first Network parameters of the surface defect detection model in the next iteration Updated parameters; Specifically, these can be the parameters of the feature extraction network corresponding to the feature extraction submodule; Indicates the first Network parameters of the surface defect detection model in the next iteration; The learning rate; To update momentum; For the first The gradient of the objective function at the next iteration; For the first The gradient of the objective function at the next iteration; This is the disturbance value; For the first The objective function at the next iteration.

[0048] For the backpropagation method and stochastic gradient descent method, please refer to the prior art. The network parameter update process in step a6 can be understood in conjunction with the existing network training process, and will not be described in detail here.

[0049] Through the first The next iteration yields the surface defect detection model with updated network parameters. .

[0050] Step a7: Determine whether the current iteration has reached the preset maximum number of iterations. If yes, obtain the trained surface defect detection model; otherwise, return to step a2. Before step a1, the maximum number of iterations can be preset. The maximum number of iterations It can be greater than or equal to 350, for example, it can be... etc.

[0051] In step a7, the sequence number of the current iteration can be determined. Does it meet the requirements? If so, use the model obtained in the current iteration. As a surface defect detection model that has been trained If not, return to step a2. +1 to proceed to the next iteration.

[0052] By following the steps above, a surface defect detection model can be obtained after training. The effectiveness of this model in defect detection can be verified using a test set.

[0053] S2, input the image to be tested into the surface defect detection model, use the pyramid scene parsing module to output a fine-grained feature map that integrates multiple scale features, and use the feature memory cache module to save the fine-grained feature map; Please see Figure 5 The prediction process is illustrated in the diagram.

[0054] The image to be tested in S2 can be an image of the same scene as the images in the training set, and it needs to be detected whether it contains defective regions. For ease of understanding, this embodiment of the invention uses a test set. The images in the test set are used as examples to illustrate the concept. However, it should be noted that the images used for testing are not limited to the test set. .

[0055] The specific processing steps for S2 are explained in the relevant sections above and will not be repeated here. The fine-grained feature map includes multiple fine-grained feature vectors.

[0056] S3, use the online clustering module to cluster the multiple fine-grained feature vectors to obtain multiple cluster centers; determine the fine-grained feature vector closest to each cluster center as the prototype vector corresponding to that cluster center; For details on the S2 processing, please refer to the relevant sections above; they will not be repeated here. It is understandable that each cluster center corresponds to a prototype vector, and each fine-grained feature vector belongs to the same cluster as its corresponding cluster center. Therefore, it corresponds to the prototype vector of its respective cluster center.

[0057] S4. For each fine-grained feature vector, based on the distance between the fine-grained feature vector and the prototype vector corresponding to its cluster center, the anomaly score of the pixel in the image to be tested that matches the fine-grained feature vector is obtained, and the image-level anomaly score is obtained using all the obtained anomaly scores. First, the test set The Middle Image The image is used as input to the trained surface defect detection model for forward propagation to calculate... The Middle Line number Anomaly score of pixels in column .

[0058] Specifically, for each fine-grained feature vector, an anomaly score is obtained for the pixel in the image to be tested whose position matches the fine-grained feature vector by calculating the distance between the fine-grained feature vector and the prototype vector corresponding to its cluster center. All obtained anomaly scores are then used to obtain an image-level anomaly score, using the following formula: (8); in, Represents the image to be tested. Represents the image to be tested The Middle Line number Pixels at the column; with This represents the fine-grained feature map stored in the feature memory cache module; This indicates that the fine-grained feature map stored in the feature memory cache module is... The Middle Line number The elements of a column refer to the corresponding fine-grained feature vectors; Represents fine-grained feature vectors The prototype vector corresponding to the cluster center; This represents the position and fine-grained feature vector in the image under test. Abnormal scores for matched pixels; This indicates a search for the 2-norm; The term represents the number of prototype vectors. In the above formula (8), the numerator represents the minimum L2 distance between the fine-grained feature vector and all prototype vectors, and the denominator represents the sum of the L2 distances between the fine-grained feature vector and all prototype vectors.

[0059] Therefore, it can be understood that, through the above formula (8), each in the image to be tested... Each pixel at a given location can be assigned a corresponding anomaly score. As can be seen, the embodiments of the present invention can directly obtain the anomaly scores of the pixels at the corresponding positions in the original image from the fine-grained feature vectors. The obtained anomaly scores of all pixels can then be used to obtain image-level anomaly scores.

[0060] Specifically, the image-level anomaly score is obtained by utilizing all the obtained anomaly scores, using the following formula: (9); in, Indicates image-level anomaly score; , These represent the height and width of the input image, respectively.

[0061] Therefore, the image under test can ultimately obtain an image-level anomaly score. .

[0062] S5, determine whether the image-level anomaly score is greater than a preset threshold. If yes, determine that there is a surface defect in the image to be tested; if no, determine that there is no surface defect in the image to be tested.

[0063] The preset threshold can be set based on normal images in the training set, and can be used as follows: Indicates. If If so, then surface defects exist in the image to be tested; if If the surface defects are not present in the image being tested, then no surface defects are present in the image being tested.

[0064] S4 and S5 can be executed by the online clustering module, which directly outputs the prediction results to characterize whether the image under test has surface defects.

[0065] Existing methods for surface defect detection using anomaly scores assign anomaly scores from coarse-grained features to fine-grained features to obtain anomaly scores for image regions. In other words, existing distance-based surface defect detection methods assign anomaly scores from coarse-grained features to fine-grained features, resulting in all fine-grained features within the same coarse-grained feature having the same anomaly score. This fails to consider the differences in anomaly scores between defective and normal regions within the same coarse-grained feature. Such anomaly scoring reduces the accuracy of these methods in detecting surface defect regions.

[0066] To address the aforementioned shortcomings, the solution provided by this invention removes this limitation. By extracting fine-grained feature maps from the image and calculating the metric distance between each fine-grained feature vector and the prototype vector obtained through online learning, the anomaly score of the corresponding pixel in the original image is directly obtained. This pixel-level anomaly score enables precise localization of the defect region boundary, thereby improving the model's localization performance in defect regions. Furthermore, the image-level anomaly score obtained from all anomaly scores is used to determine the presence of surface defects. This invention can directly obtain the anomaly scores of fine-grained features in an image region, removing the constraint of the coarse-grained feature anomaly score redistribution strategy on the accuracy of defect region localization. By directly obtaining the anomaly score of the image region through the distance metric between the fine-grained feature vector and the prototype vector, the accuracy of the model in detecting surface defect regions can be improved.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A surface defect detection method based on fine-grained prototype online learning, characterized in that, include: A surface defect detection model is obtained; wherein, the surface defect detection model includes a pyramid scene parsing module, a feature memory cache module, and an online clustering module connected in sequence; the surface defect detection model is pre-trained using several training images that do not contain surface defect regions; The image to be tested is input into the surface defect detection model, and the pyramid scene parsing module outputs a fine-grained feature map that integrates multiple scale features. The feature memory cache module is used to save the fine-grained feature map. The fine-grained feature map includes multiple fine-grained feature vectors. The online clustering module is used to cluster the multiple fine-grained feature vectors to obtain multiple cluster centers; the fine-grained feature vector closest to each cluster center is determined as the prototype vector corresponding to that cluster center; For each fine-grained feature vector, based on calculating the distance between the fine-grained feature vector and the prototype vector corresponding to its cluster center, anomaly scores are obtained for pixels in the image to be tested whose positions match the fine-grained feature vector. All obtained anomaly scores are then used to obtain an image-level anomaly score. The formulas used in this step include: ； ； in, Represents the image to be tested; with This indicates that the feature memory cache module stores fine-grained feature maps. Composed of fine-grained feature vectors; This indicates that the fine-grained feature map stored in the feature memory cache module is... The Middle Line 1 The elements of a column refer to the corresponding fine-grained feature vectors; Represents fine-grained feature vectors The prototype vector corresponding to the cluster center; This represents the position and fine-grained feature vector in the image under test. Abnormal scores for matched pixels; This indicates a search for the 2-norm; Indicates the number of prototype vectors; Indicates image-level anomaly score; , These represent the height and width of the input image, respectively. Determine whether the image-level anomaly score is greater than a preset threshold. If yes, determine that there is a surface defect in the image under test; otherwise, determine that there is no surface defect in the image under test.

2. The surface defect detection method based on fine-grained prototype online learning according to claim 1, characterized in that, The pyramid scene parsing module includes: The module consists of a sequentially connected feature extraction submodule, a pyramid pooling submodule, a channel adjustment submodule, a bilinear interpolation upsampling submodule, a feature fusion extraction submodule, and a momentum update submodule; among which... The feature extraction submodule is a feature extraction network that includes multiple two-dimensional convolutional layers and multiple activation function layers; The pyramid pooling submodule is a pooling network containing a preset number of pooling layers of different sizes; wherein, the pooling kernel size of each pooling layer is different to represent different scales; The channel adjustment submodule includes the preset number of... Convolutional networks with convolutional layers; The feature fusion and extraction submodule is a convolutional neural network containing a feature concatenation layer and multiple two-dimensional convolutional layers; The momentum update submodule includes a momentum updater.

3. The surface defect detection method based on fine-grained prototype online learning according to claim 2, characterized in that, The feature memory cache module includes: The feature memory submodule and the pseudo-label memory submodule are connected sequentially; wherein the input end of the feature memory submodule is connected to the output end of the momentum update submodule.

4. The surface defect detection method based on fine-grained prototype online learning according to claim 3, characterized in that, The online clustering module includes: K-means clustering submodule.

5. The surface defect detection method based on fine-grained prototype online learning according to claim 4, characterized in that, The surface defect detection model also includes: A prototype memory cache module; wherein the input of the prototype memory cache module is connected to the output of the Kmeans clustering submodule, and the output of the prototype memory cache module is connected to the input of the pseudo-label memory submodule.

6. The surface defect detection method based on fine-grained prototype online learning according to claim 5, characterized in that, The training process of the surface defect detection model includes: Step a1: Preprocess several training images that do not contain surface defect regions to form a training set; Step a2: Input the images in the training set into the pyramid scene parsing module of the completed surface defect detection model. For the current iteration, output a fine-grained feature map fused with multiple scale features from the feature fusion extraction submodule of the pyramid scene parsing module. Step a3: The momentum update submodule updates the fine-grained feature map output by the feature fusion extraction submodule in the current iteration based on the fine-grained feature map obtained before the current iteration, and sends the update result to the feature memory submodule for storage; wherein, for the first iteration, the previously obtained fine-grained feature map is empty, and the update result of the momentum update submodule is the fine-grained feature map output by the feature fusion extraction submodule in the first iteration; Step a4: The Kmeans clustering submodule uses the Kmeans clustering method to cluster all fine-grained feature vectors in the fine-grained feature map stored in the feature memory submodule to obtain multiple cluster centers; for each cluster center, the fine-grained feature vector closest to the cluster center is determined as the prototype vector corresponding to the cluster center; all determined prototype vectors are sent to the prototype memory cache module. Step a5: Use the prototype memory cache module to save all the obtained prototype vectors, and send all the saved prototype vectors to the pseudo-label memory submodule as pseudo-label vectors for the current iteration, which are used to characterize the detection result of the current iteration. Step a6: Based on the detection results of the current iteration, the backpropagation method is used to calculate the gradient of the network parameters through a preset loss function, and the stochastic gradient descent method is used to update the network parameters of the surface defect detection model using the obtained network parameter gradient. Step a7: Determine whether the current iteration has reached the preset maximum number of iterations. If yes, obtain the trained surface defect detection model; otherwise, return to step a2.

7. The surface defect detection method based on fine-grained prototype online learning according to claim 6, characterized in that, Step a2 includes: The feature extraction submodule is used to extract features from the input image to obtain the corresponding original feature map; wherein the original feature map and the input image have the same size; Using the preset number of pooling layers of different sizes in the pyramid pooling submodule, the original feature map is downsampled at the corresponding scale to obtain the preset number of downsampled feature maps; The preset number of channels adjusted in the channel adjustment submodule The convolutional layer adjusts the number of channels in the preset number of downsampled feature maps to obtain the preset number of downsampled channel-adjusted feature maps. The bilinear interpolation upsampling submodule is used to upsample the feature maps of the preset number of downsampling channels respectively to obtain the preset number of interpolated feature maps with the same size as the original feature map. The original feature map and the preset number of interpolated feature maps are stitched together using the feature stitching layer in the feature fusion extraction submodule. The stitched feature map is then fused using multiple two-dimensional convolutional layers in the feature fusion extraction submodule to obtain a fine-grained feature map.

8. The surface defect detection method based on fine-grained prototype online learning according to claim 6, characterized in that, The formula for calculating the preset loss function is as follows: ； in, This represents the mean squared error loss; Indicates prototype loss; Indicates the preset weight; ； ； ； in, , Representing the height and width of the input image respectively; This indicates that the feature memory cache module stores fine-grained feature maps. Composed of fine-grained feature vectors; The first fine-grained feature map to be saved Line 1 The elements of a column refer to the corresponding fine-grained feature vectors; Represents fine-grained feature vectors The prototype vector corresponding to the cluster center; Represents the prototype vector Corresponding to all fine-grained feature vectors and prototype vectors in the cluster The degree of dispersion between them; Represents the number of prototype vectors, and is a natural number greater than 0; Indicates the first One prototype vector; Represents the prototype vector Corresponding to all fine-grained feature vectors and prototype vectors in the cluster The degree of dispersion between them; This indicates a search for the 2-norm; Indicates belonging to the prototype vector The number of fine-grained feature vectors in the corresponding cluster; Indicates belonging to the prototype vector The fine-grained feature vectors in the corresponding clusters; Represents the prototype vector Corresponding to all fine-grained feature vectors and prototype vectors in the cluster The degree of dispersion between them; It is a calculation and The general formula for prototype vectors for Time calculation prototype vector for Time calculation .

Images