A method, apparatus and system for detecting surface anomalies in industrial products

Abstract

This invention discloses a method, apparatus, and system for detecting surface anomalies in industrial products, belonging to the field of industrial product anomaly detection. The method includes: constructing a network model comprising a feature extractor, a feature optimization module, and a positive / negative feature vector pair generation module; the feature extractor extracts multi-scale features; the feature optimization module refines features from both single-scale and multi-scale perspectives, enhancing the interaction of multi-scale feature information to address defect regions at different scales, and finally upsampling different feature maps according to the resolution of the shallowest feature layer, while fusing certain positional information; the positive / negative feature pair generation module generates positive and negative feature vector pairs from both image and feature perspectives; and the network model is trained using a training set composed of normal images, with training losses including clustering loss and separation loss, ensuring that the nominal features in the feature memory are sufficient to distinguish between normal and abnormal feature vectors. This invention overcomes the domain drift problem and achieves pixel-level detection of surface anomalies in industrial products.

Description

Technical Field

[0001] This invention belongs to the field of industrial product anomaly detection, and more specifically, relates to a method, apparatus and system for detecting surface anomalies in industrial products. Background Technology

[0002] Due to the complexity of their backgrounds and manufacturing processes, industrial products often experience unpredictable surface defects during production, such as scratches, stains, and wear. These defects, often originating on the surface of industrial products, require timely and accurate detection to effectively ensure product quality. As more and more industrial products are integrated into people's production and daily lives, surface quality inspection significantly limits the scale and efficiency of actual production lines. Therefore, industrial product surface defect detection algorithms have gained increasing attention and become a current research hotspot. However, the types of industrial products in actual production are diverse and complex, with varying backgrounds. This complexity significantly impacts the accuracy of surface anomaly detection algorithms. Even for the same type of industrial product, unpredictable defects can occur at any stage of production due to its complex manufacturing process. The diversity of industrial products and the unpredictable types of defects place extremely high demands on the robustness of industrial product surface anomaly detection algorithms.

[0003] In recent years, with the widespread development of deep learning, deep learning-based industrial product surface anomaly detection algorithms have gradually replaced traditional machine vision-based detection methods and have been widely applied. Deep learning-based industrial product surface anomaly detection algorithms include image-level and feature-level algorithms. Image-level detection algorithms often fail due to small differences between images. Feature-level detection algorithms use the pre-trained model's representation on normal samples as the normal pattern, utilizing the difference from the normal pattern during inference to locate anomalies, effectively improving the robustness of anomaly detection and thus gaining wider application. However, pre-trained models are often pre-trained on large natural datasets, and their deep feature representations are more suited to natural datasets, which are insufficient for industrial inspection datasets. Furthermore, pre-trained models perform classification tasks on natural datasets, and there are still significant differences when directly transferred to industrial inspection tasks. In summary, mainstream feature-level detection algorithms suffer from a "domain drift" problem between the source domain (natural dataset) and the target domain (industrial inspection dataset), leading to inaccurate normal pattern recognition by pre-trained models and limiting their performance on industrial inspection datasets.

[0004] Currently, existing algorithms have proposed relatively effective solutions to the "domain drift" problem, such as early termination of training, regularization of pre-trained model weights, and adding image-assisted training set classification. Although these existing methods can achieve "domain adaptation" to some extent, most of them focus on image-level discrimination, and the accuracy of pixel-level anomaly localization still needs to be improved. That is to say, existing methods can only determine whether there are defects on the surface of industrial products and what kind of defects exist based on images, but cannot accurately locate the defect area on the surface of industrial products. Therefore, they cannot provide more effective guidance information for the traceability and improvement of the corresponding industries. Summary of the Invention

[0005] In view of the shortcomings of existing technologies and the need for improvement, this invention provides a method, device and system for detecting surface anomalies of industrial products. Its purpose is to effectively solve the "domain drift" problem of existing methods for detecting surface anomalies of industrial products, so as to adapt to complex and diverse industrial products and unpredictable defect types, and achieve high-precision anomaly positioning on the surface of industrial products.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for detecting surface anomalies in industrial products is provided, comprising:

[0007] Training phase:

[0008] Obtain a training set consisting of surface images of normal industrial products;

[0009] Multiple images are selected from the training set and the three-dimensional features of each image are extracted. The mean of the feature vectors at each position is calculated as the nominal feature vector, and all nominal feature vectors constitute the feature memory.

[0010] A network model is constructed, comprising a feature extraction module, a positive and negative feature vector pair generation module, and a feature mapping module. The network model is then trained using a training set. During training, the feature extraction module is used to extract image I. nor The three-dimensional features Z nor The positive and negative feature vector generation module is used to generate the corresponding abnormal features Z. abnor From normal feature Z nor and abnormal features Z abnor Extract the foreground features to obtain feature Z'. nor and Z' abnor and utilize feature Z' nor and Z' abnorFeature vectors at the same position are respectively used as normal feature vectors and abnormal feature vectors, forming positive and negative feature vector pairs; the feature mapping module is used to map normal feature vectors and abnormal feature vectors to a new feature space; the training loss function includes clustering loss and separation loss; the clustering loss is used to measure the abnormal distance of normal feature vectors, and the larger the abnormal distance, the larger the clustering loss; the separation loss is used to measure the difference between the abnormal distance of normal feature vectors and their corresponding abnormal feature vectors. When the difference is greater than a preset threshold margin, the separation loss is 0, and when it is not greater than the margin, the separation loss is positive, and the smaller the difference, the larger the separation loss.

[0011] Wherein, the anomaly distance of a feature vector is the average distance between that feature vector and its K nearest nominal feature vectors; K is a preset positive integer;

[0012] Testing phase:

[0013] The surface image I of the industrial product to be inspected is input into the feature extraction module to obtain the three-dimensional feature Z. Then, the feature mapping module is used to map it to a new feature space to obtain feature Z'. For the feature vector at each position in feature Z', the weighted sum of the distances between it and the nearest K nominal feature vectors is calculated as the anomaly score at the corresponding position, and the anomaly score map is obtained.

[0014] After upsampling the anomaly score map to the same size as image I, Gaussian filtering is performed to obtain the location of the defect in image I, thus completing the anomaly detection.

[0015] Furthermore, the training phase also includes updating the nominal feature vectors in the nominal feature library during the training of the network model using the training set.

[0016] In the i-th training iteration, for any t-th nominal feature vector z t it The update method is as follows:

[0017] Obtain the nominal feature vector from a pre-built normal feature vector library. The corrected feature vector z is obtained by weighted summation of the Q most recent normal feature vectors. r ;

[0018] according to nominal vector Updated to

[0019] Where α represents the momentum parameter that controls the degree of update, 0 < α < 1.

[0020] Furthermore, the feature extraction module includes a feature extractor (Backbone) and a feature optimization module (FRM).

[0021] Backbone, a feature extractor, is used to extract multi-scale features from images;

[0022] The Feature Optimization Module (FRM) is used to combine the first L layers of features from a multi-scale feature set into a three-dimensional feature set.

[0023] The feature extractor Backbone is a pre-trained model trained using a natural image dataset; 1 <L≤L max L max This indicates the maximum number of layers of features extracted by the Backbone feature extractor.

[0024] Furthermore, the Feature Optimization Module (FRM) includes: a feature fusion module, and a pyramid attention module (PAM) set between every two adjacent layers of features;

[0025] The Pyramid Attention Module (PAM) includes:

[0026] A globally average pooling layer, a 1×1 convolutional layer, and a softmax activation function layer are connected sequentially to process the high-dimensional feature Z in adjacent layers. high Perform the corresponding operations to obtain feature A;

[0027] The dot product layer is used to perform low-dimensional features in adjacent layers. Perform a dot product with feature A to obtain feature A.

[0028] And residual structures, used according to Calculate the optimized features As the output of the Pyramid Attention Module (PAM);

[0029] The feature fusion module is used to upsample the Lth layer features and the L-1 optimized features to the same size and then cascade them to obtain three-dimensional features;

[0030] Among them, feature A, feature and features The size is related to the feature Same; γ is a trainable parameter.

[0031] Furthermore, the Feature Optimization Module (FRM) also includes: a local average pooling layer set after the features of each layer and before each pyramid attention module (PAM);

[0032] Furthermore, feature Z high and characteristics All of these are features obtained by local average pooling of the original features.

[0033] Furthermore, the corresponding anomaly feature Z is generated.abnor This includes at least one of the following methods:

[0034] Method 1:

[0035] Generation and Image I nor Images of the same size with Berlin noise N And thresholding it to locate irregular defect regions I. nm ;

[0036] Calculate image I nor Foreground mask image I fm Combine it with the irregular defect area I nm Perform an AND operation to obtain mask I. m ;

[0037] Obtain another image I' from the training set. nor Add defects to it to obtain defect image I a ;

[0038] According to I abnor =I m ⊙((1-β)I a +βI nor )+(1-I m )⊙I nor Generate defect image I abnor ;

[0039] Defect image I is extracted using the feature extraction module. abnor The three-dimensional features were obtained, and the normal features Z were compared with those. nor The corresponding abnormal feature Z abnor ;

[0040] Where ⊙ represents the AND operation; β>0 represents the parameters obtained by random sampling;

[0041] Method 2:

[0042] The noise feature Z is obtained by randomly sampling from a Gaussian distribution. noise ;

[0043] Calculate image I nor Foreground mask image I fm Combine it with the noise feature Z noise After performing an AND operation to obtain the foreground noise features, they are fused into the normal feature Z. nor In the middle, the normal feature Z was obtained. nor The corresponding abnormal feature Z abnor .

[0044] Furthermore, the expression for the clustering loss is:

[0045]

[0046] Among them, L cls N represents the clustering loss. c V represents the number of normal feature vectors, K represents the number of nominal feature vectors recalled by the KNN algorithm, and D() represents the vector distance; v nor This represents the normal feature vector after mapping by the feature mapping module. Indicates with v nor The k-th nearest nominal eigenvector; Indicates the feature vector index.

[0047] Furthermore, the expression for the separation loss is:

[0048]

[0049]

[0050] Among them, L split Represents the separation loss, n s This represents the number of pairs of positive and negative eigenvectors; v p and v n These represent the normal feature vector and the abnormal feature vector after the feature mapping module has been applied to the positive and negative feature vector pairs, respectively. This represents the normal feature vector v p The nominal eigenvector closest to the k-th nearest neighbor. Represents the abnormal feature vector v n The nominal eigenvector that is closest to the k-th element; Indicates the feature vector index

[0051] According to another aspect of the present invention, an industrial product surface anomaly detection device is provided, comprising:

[0052] A computer-readable storage medium for storing computer programs;

[0053] And a processor, used to read a computer program stored in a computer-readable storage medium and execute the above-described industrial product surface anomaly detection method provided by the present invention.

[0054] According to another aspect of the present invention, an industrial product surface anomaly detection system is provided, comprising:

[0055] A camera used to capture images of the surface of industrial products to be inspected;

[0056] The above-mentioned industrial product surface anomaly detection device provided by the present invention is connected to a camera.

[0057] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0058] (1) After initializing the feature memory using normal feature vectors, this invention constructs a training loss function using the feature memory to train the trainable parameters in the feature extraction module and the 1×1 convolution used for feature mapping. The training loss function includes clustering loss and separation loss. The clustering loss increases as the distance between the normal feature vector and the K nearest nominal feature vectors increases, while the separation loss decreases as the difference between the distances between the normal feature vector and the abnormal feature vector and the K nearest nominal feature vectors increases. Therefore, the clustering loss can penalize the distance between the normal feature vector and the K nearest nominal feature vectors, enhancing the similarity between the normal feature vector and the K nearest nominal feature vectors, so that the normal feature vector will obtain a smaller abnormal score during the inference process. The separation loss maximizes the relative distance between the normal feature vector and the abnormal feature vector, establishing a more effective classification plane, that is, the distance between it and the inference benchmark (i.e., the nominal feature). Based on this loss function, this invention can effectively update trainable parameters, eliminate the "domain drift" problem, and enhance the discriminative performance of the nominal feature vector. Furthermore, based on the judgment of the feature vector, this invention can pinpoint the defects at every specific location on the surface of industrial products, achieving pixel-level anomaly detection. In summary, this invention effectively solves the "domain drift" problem existing in current industrial product surface anomaly detection methods, adapting to complex and diverse industrial products and unpredictable defect types, and achieving high-precision anomaly localization on industrial product surfaces.

[0059] (2) In a preferred embodiment of the present invention, during the training process, in addition to optimizing the trainable parameters, the nominal feature vector in the feature memory is updated by means of the normal feature vector. During the update process, when calculating the correction term, the normal feature vector that is far away from the nominal feature vector is weighted and penalized, thereby ensuring the stability of the feature update and further reducing the probability of misjudgment under normal circumstances.

[0060] (3) In the preferred embodiment of the present invention, the feature extraction module selects the first few layers of features for fusion based on the feature extractor trained on the natural dataset. In multi-scale features, deep features often have richer semantic information and stronger discriminative ability, but the feature map has a smaller resolution, which is not conducive to fine-grained defect localization. The present invention selects the first few layers of features from the multi-scale features for fusion, which can take into account both depth and resolution, and further improve the accuracy of subsequent detection.

[0061] (4) In a further preferred embodiment of the present invention, the feature optimization module further includes a pyramid attention module (PAM) set between every two adjacent layer features. This module performs global average pooling operation on the high-dimensional features in sequence to fuse global context information, then aligns the low-dimensional features through 1×1 convolution, and then uses the softmax activation function to obtain attention on the channel to establish long-range dependencies. Finally, it fuses the processed high-dimensional features and low-dimensional features together through dot product and residual structure. The features are refined and optimized from multiple scales, realizing the interaction between feature information at multiple scales, which can further improve the detection accuracy.

[0062] (5) In a further preferred embodiment of the present invention, the feature optimization module further includes a local average pooling layer disposed after each layer of features, thereby increasing the receptive field of the features and enhancing the robustness of the features to noise input.

[0063] (6) In a preferred embodiment of the present invention, positive and negative feature vector pairs are generated from both the image level and the feature level. Specifically, when constructing positive and negative feature vector pairs from the image level, an abnormal image is first generated, then the abnormal features are extracted using a feature extraction module, and then combined with the corresponding normal features to complete the construction. When constructing positive and negative feature vector pairs from the feature level, noise is directly added to the normal features as an abnormal feature, and then combined with the corresponding normal features to complete the construction. The present invention generates positive and negative feature vector pairs from both levels simultaneously, which is beneficial to improving the discrimination ability of the feature memory. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the network model established during the training phase in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the Pyramid Attention Module (PAM) in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram illustrating the generation of positive and negative feature vector pairs from an image level in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram illustrating the generation of positive and negative feature vector pairs from the feature level in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the localization of surface anomalies of different industrial products according to an embodiment of the present invention; wherein, (a) to (o) represent the detection results of different industrial products respectively. In each figure, the first row represents the surface image of the industrial product to be detected, the second row represents the truth map, and the third row represents the detection results of this embodiment.

[0069] Figure 6The image shows the feature distribution of different methods for detecting metal nuts. (a) represents the surface image of the metal nut, (b) represents the feature distribution of the target dataset before domain adaptation, (c) represents the feature distribution after adding clustering loss, and (d) represents the feature distribution after adding both clustering loss and separation loss.

[0070] Figure 7 The image shows the feature distribution of different methods for detecting leather. (a) represents a leather surface image, (b) represents the feature distribution of the target dataset before domain adaptation, (c) represents the feature distribution after adding clustering loss, and (d) represents the feature distribution after adding both clustering loss and separation loss. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0072] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0073] To address the "domain drift" problem and the inability to achieve pixel-level anomaly localization in existing industrial product surface anomaly detection methods, this invention provides an industrial product surface anomaly detection method, device, and system. The overall approach involves constructing a feature memory with strong discriminative capabilities using normal feature vectors as an inference benchmark. Through optimized design of the training loss function, the trainable parameters in the pre-trained model and other modules are effectively optimized, thereby eliminating the domain drift problem. Simultaneously, based on feature detection of anomalies at various locations, pixel-level anomaly detection is achieved.

[0074] The following is an example.

[0075] Example 1:

[0076] A method for detecting surface anomalies in industrial products. This embodiment can be divided into two stages: a training stage and a detection stage. The training stage, based on the constructed feature memory and training loss function, optimizes and updates the parameters that need to be trained in the pre-training module and other modules. The detection stage, based on the optimized and updated feature extraction module, detects specific defect areas on the surface of the industrial product. The specific implementation methods for each stage are explained in detail below.

[0077] In this embodiment, during the training phase, a training set consisting of surface images of normal industrial products is first constructed. Then, a feature memory is built as the inference basis for the anomaly detection process. The feature memory includes multiple normal feature vectors. Considering that the distribution of surface defects in the same industrial products often exhibits certain similarities, in order to reduce the size of the feature memory while ensuring its discriminative power, the initial construction method of the feature memory in this embodiment specifically includes:

[0078] Multiple images are selected from the training set and the three-dimensional features of each image are extracted. The mean of the feature vectors at each position is calculated as the nominal feature vector, and all nominal feature vectors constitute the feature memory.

[0079] For any position (i,j) in the foreground region of the image, its nominal feature vector Z m It can be represented as follows:

[0080] Z m ={μ ij ,i∈[1,H 1 ],j∈[1,W 1 ]}

[0081] in, N m =H 1 ×W 1 W 1 and H 1 C represents the height and width of the image, respectively, and C' represents the dimension of the feature space described by the nominal feature vector.

[0082] This embodiment further includes the following during the training phase: constructing a network model including a feature extraction module, a positive and negative feature vector pair generation module, and a feature mapping module, and training the network model using a training set; such as Figure 1 As shown, during training, the feature extraction module is used to extract images I from the training set. nor The three-dimensional features Z nor The positive and negative feature vector generation module is used to generate the corresponding abnormal features Z. abnor From normal feature Z nor and abnormal features Z abnor Extract the foreground features to obtain feature Z'. nor and Z' abnor and utilize feature Z' nor and Z' abnor The feature vectors at the same position are respectively used as normal feature vectors and abnormal feature vectors to form positive and negative feature vector pairs; the feature mapping module is a 1×1 convolutional layer used to map normal feature vectors and abnormal feature vectors to the feature space representing the feature vectors.

[0083] like Figure 1 As shown, in this embodiment, the feature extraction module specifically includes a feature extractor (Backbone) and a feature optimization module (FRM). In this embodiment, the feature extractor (Backbone) is a pre-trained model trained using the ImageNet natural image dataset. Optionally, the number of layers in the feature extractor (Backbone) is L. max =4.

[0084] In multi-scale features, deep features often possess richer semantic information and stronger discriminative power, but their feature map resolution is lower, which is not conducive to fine-grained defect localization. To balance sufficient depth and resolution, in this embodiment, the Feature Optimization Module (FRM) is used to fuse the first L layers of features from the multi-scale features into a three-dimensional feature; 1 <L≤L max Optionally, in this embodiment, L = 3, that is, l ∈ {1, 2, 3}.

[0085] Let φ represent the feature extraction module, then the input image I nor Subsequently, the extracted multi-scale features It can be represented as follows:

[0086]

[0087] In the formula, l∈{1,2,…L max}, W l H l and C l L represents the width, height, and number of channels of the feature output from the l-th layer of the feature extractor Backbone, respectively; max L represents the maximum number of feature layers extracted by the Backbone feature extractor. Optionally, in this embodiment, L max =4.

[0088] In actual testing, the aforementioned feature Z is directly utilized. nor During detection, the "domain drift" problem can severely affect positioning accuracy. To improve detection accuracy, in this embodiment, the feature optimization module also performs feature information interaction between single and multi-scale methods and adds trainable parameters for fine-tuning on the training set.

[0089] At multiple scales, such as Figure 1 and Figure 2 As shown, the Feature Optimization Module (FRM) specifically includes: a feature fusion module, and a Pyramid Attention Module (PAM) set between every two adjacent feature layers;

[0090] like Figure 2 As shown, the Pyramid Attention Module (PAM) includes:

[0091] A globally average pooling layer, a 1×1 convolutional layer, and a softmax activation function layer are connected sequentially to process the high-dimensional feature Z in adjacent layers. high The corresponding operation is performed. In this process, high-dimensional features are fused with global context information through global average pooling, and then aligned with low-dimensional features through 1×1 convolution to obtain feature att:

[0092] att = f l (f gap (Z l+1 ))

[0093] In the formula, l∈{1,2}. f gap and f l These represent global average pooling and 1×1 convolution operations, respectively; feature att obtains channel attention through the softmax activation function, establishing long-range dependencies, resulting in feature A:

[0094] A = softmax(att)

[0095] like Figure 2 Therefore, the Pyramid Attention Module (PAM) also includes a dot product layer for processing low-dimensional features from adjacent layers. Perform a dot product with feature A to obtain feature A.

[0096]

[0097] In the formula Feature A, Feature and features The size is related to the feature same;

[0098] like Figure 2 Therefore, the Pyramid Attention Module (PAM) also includes a residual structure for... Calculate the optimized features As the output of the Pyramid Attention Module (PAM);

[0099] In the formula, γ is a trainable parameter, and in this implementation, its initial value is 0;

[0100] In this embodiment, the features extracted from the feature extractor output consist of three layers. Correspondingly, the pyramid attention module is used twice for each layer, thereby refining and optimizing the features across multiple scales and enabling interaction between feature information at multiple scales. High-dimensional features possess stronger semantic and category information, and their global context information can guide low-dimensional features in the channel direction with prior information. Therefore, this embodiment can effectively improve detection accuracy through the feature optimization module.

[0101] In this embodiment, the feature fusion module in the feature optimization module is used to upsample the Lth layer features and the L-1 optimized features to the same size and then cascade them to obtain three-dimensional features.

[0102] At a single scale, such as Figure 1 As shown, in this embodiment, the feature optimization module also includes a local average pooling layer disposed after each layer of features and before each pyramid attention module (PAM); this can increase the receptive field of the features and enhance the robustness of the features to noisy input.

[0103] The features of layer l, after local average pooling, can be represented as:

[0104] Z l' =f lap (Z l )

[0105] In the formula l∈{1,2,3}. To maintain the resolution of the feature map, this embodiment uses average pooling with a spatial size of 3×3 and a stride and padding of 1.

[0106] It is easy to understand that the input features Z of the Pyramid Attention Module (PAM) are... high and characteristics All of these are features obtained by local average pooling of the original features.

[0107] Considering that the feature memory-based method disrupts the positional relationship between feature vectors, this embodiment introduces an absolute and untrainable two-dimensional positional encoding in the feature fusion module to fuse the positional information of each feature vector into the feature vector, thereby alleviating the above problem.

[0108] In the field of defect detection, self-supervised methods often improve defect detection accuracy by generating simulated negative samples to supervise the model's training process. Traditional methods rely on normal samples in the training set, lacking supervised learning based on defect information. In this embodiment, the positive and negative feature pair generation module generates positive and negative feature vector pairs from the image level and the feature level, respectively, facilitating the construction of a more discriminative feature memory. Image-level positive and negative feature pair generation first requires generating abnormal negative samples. Abnormal negative samples can be generated by fusing normal images and artificially generated abnormal images through a mask image. The mask image can be obtained by performing a bitwise AND operation between a noise mask with Berlin noise thresholding and a foreground mask. Artificially generated abnormal images can be generated from normal images through color perturbation and random cropping. The abnormal negative samples then obtain cascaded features through a feature extractor and a feature optimization module, and positive and negative feature vector pairs are obtained by downsampling the mask image and based on the non-zero index of the mask image. Feature-level positive and negative feature pair generation only requires fusing the foreground mask image and the noise feature vector, which can be randomly sampled from a Gaussian distribution.

[0109] like Figure 3 As shown, in this embodiment, normal feature Z is generated from the image level. nor Corresponding anomaly feature Z abnor ,include:

[0110] Generation and Image I nor Images of the same size with Berlin noise N And thresholding it to locate irregular defect regions I. nm ;

[0111] Calculate image I nor Foreground mask image I fm Combine it with the irregular defect area I nm Perform an AND operation to obtain mask I. m ;

[0112] Obtain another image I' from the training set. nor Add defects to it to obtain defect image I a ;

[0113] According to I abnor =I m ⊙((1-β)I a +βI nor )+(1-I m )⊙I nor Generate defect image I abnor ;

[0114] Defect image I is extracted using the feature extraction module. abnor The three-dimensional features were obtained, and the normal features Z were compared with those. nor The corresponding abnormal feature Zabnor ;

[0115] Where ⊙ represents the AND operation; β>0, in this embodiment, β is a parameter randomly sampled from [0,0.8]; through normal feature Z nor and the corresponding abnormal feature Z abnor It can locate pairs of positive and negative feature vectors that exist in pairs.

[0116] like Figure 4 As shown, in this embodiment, normal feature Z is generated from the feature level. nor Corresponding anomaly feature Z abnor ,include:

[0117] The noise feature Z is obtained by randomly sampling from a Gaussian distribution. noise The Gaussian distribution is denoted as N(μ,σ), with the mean μ and standard deviation σ set to 0 and 1, respectively.

[0118] Calculate image I nor Foreground mask image I fm Combine it with the noise feature Z noise After performing an AND operation to obtain the foreground noise features, they are fused into the normal feature Z. nor In the middle, the normal feature Z was obtained. nor The corresponding abnormal feature Z abnor :

[0119] Z abnor =Z nor +Z noise ⊙I fm

[0120] Because defect types are diverse and unpredictable, generating defects at the image level may lack discriminative power. This invention, based on generating positive and negative feature vector pairs at the image level, further generates positive and negative feature vector pairs at the feature level, ensuring the discriminative power of defects. During the training of the network model, the positive and negative feature vector pairs are used to establish the objective function, effectively eliminating the "domain drift" problem of feature-level industrial product surface anomaly detection algorithms and significantly improving the model's defect localization accuracy.

[0121] In this embodiment, when training the established network model using the training set, the training loss function includes clustering loss and separation loss. Clustering loss is used to measure the abnormal distance of normal feature vectors, and the larger the abnormal distance, the larger the clustering loss. Separation loss is used to measure the difference between the abnormal distances of normal feature vectors and their corresponding abnormal feature vectors. When the difference is greater than a preset threshold margin, the separation loss is 0; when it is not greater than the margin, the separation loss is positive, and the smaller the difference, the larger the separation loss. The abnormal distance of a feature vector is the average distance between the feature vector and the K nearest nominal feature vectors. K is a preset positive integer, and in this embodiment, it is specifically the number of nominal feature vectors recalled by the KNN algorithm.

[0122] In this embodiment, the clustering loss increases with the distance between the normal feature vector and the K nearest nominal feature vectors, while the separation loss decreases with the increase of the difference in distance between the normal and abnormal feature vectors and the K nearest nominal feature vectors. Therefore, the clustering loss penalizes the distance between the normal feature vector and the K nearest nominal feature vectors, reduces the intra-class variance among normal feature vectors, and enhances the similarity between the normal feature vector and the K nearest nominal feature vectors, resulting in the normal feature vector receiving a smaller abnormal score during inference. In this embodiment, the expression for the clustering loss is:

[0123]

[0124] Among them, L cls N represents the clustering loss. c V represents the number of normal feature vectors, K represents the number of nominal feature vectors recalled by the KNN algorithm, and D() represents the vector distance; v nor This represents the normal feature vector after mapping by the feature mapping module. Indicates with v nor The k-th nearest nominal eigenvector N c =H 1 ×W 1 ; This represents the feature vector index, i.e., the sequence number of the feature vector.

[0125] The inclusion of anomalous information significantly improves the model's discriminative performance. In this embodiment, the separation loss introduced into the training loss function maximizes the relative distance between normal and anomalous feature vectors, establishing a more effective classification plane, i.e., the distance to the inference baseline (i.e., the nominal feature). Specifically, the separation loss is calculated using triplet loss to maximize the relative distance between normal and anomalous feature vectors. In triplet loss, positive examples are defined as the distance between normal and nominal feature vectors, and negative examples are defined as the distance between anomalous and nominal feature vectors. In this embodiment, the specific expression for the separation loss is:

[0126]

[0127]

[0128] Among them, L split Represents the separation loss, n s This represents the number of pairs of positive and negative eigenvectors; v p and v n These represent the normal feature vector and the abnormal feature vector after the feature mapping module has been applied to the positive and negative feature vector pairs, respectively. This represents the normal feature vector v p The nominal eigenvector closest to the k-th nearest neighbor. Represents the abnormal feature vector v n The nominal eigenvector closest to the k-th nearest neighbor. This represents the feature vector index, i.e., the sequence number of the feature vector.

[0129] In this embodiment, the expression for the overall training loss function is:

[0130] L = L cls +λL split

[0131] In the formula, λ represents the equilibrium clustering loss L. cls and separation loss L split The weight.

[0132] When detecting surface anomalies in industrial products directly based on features extracted by the Backbone feature extractor, the "domain drift" problem can severely affect the positioning accuracy. This embodiment uses anomaly information to establish a loss function based on the nominal feature vector, which eliminates the "domain drift" problem and enhances the discriminative performance of the nominal feature vector, ultimately effectively improving the accuracy of surface anomaly detection in industrial products.

[0133] To further improve the similarity between enhanced normal feature vectors and nominal feature vectors, thereby improving the discriminative ability of nominal feature vectors, in this embodiment, the training phase also includes updating the nominal feature vectors in the nominal feature library during the training of the network model using the training set.

[0134] In the i-th training iteration, for any t-th nominal feature vector The update method is as follows:

[0135] Obtain the nominal feature vector from a pre-built normal feature vector library. The corrected feature vector z is obtained by weighted summation of the Q most recent normal feature vectors. r ;

[0136] according to nominal vector Updated to The update method is specifically an exponential moving average (EMA), which can eliminate noise in statistical data.

[0137] in, α represents the momentum parameter controlling the degree of update, 0 < α < 1; among the normal eigenvectors participating in the weighted summation, the nominal eigenvector... The greater the distance, the smaller its weight in the weighted summation. In this embodiment, the corrected feature vector z r The specific expression is:

[0138]

[0139] in, w q Penalizing normal eigenvectors that are far from the nominal eigenvector enhances the stability of feature updates. In practical applications, the normal eigenvector library can be constructed by extracting features from surface images of normal industrial products.

[0140] This embodiment updates the feature memory in this way, which can further reduce the probability of normal feature vectors being misjudged.

[0141] This embodiment, through the above training phase, can optimize the parameters of the feature extraction module and the feature mapping module, effectively solving the problem of domain drift.

[0142] Building upon the training phase, the detection phase of this embodiment specifically includes:

[0143] The surface image I of the industrial product to be inspected is input into the feature extraction module to obtain the three-dimensional feature Z. Then, the feature mapping module is used to map it to a new feature space to obtain feature Z'. For the feature vector at each position in feature Z', the weighted sum of the distances between it and the nearest K nominal feature vectors is calculated as the anomaly score at the corresponding position, and the anomaly score map is obtained.

[0144] After upsampling the anomaly score map to the same size as image I, Gaussian filtering is performed to obtain the location of the defect in image I, thus completing the anomaly detection.

[0145] In the anomaly score graph, for any anomaly score s at position (i,j) ij for:

[0146]

[0147] In the formula, z ij The feature vector representing the (i,j) position in feature Z'. This represents the nominal eigenvector that is closest to the eigenvector zij. Represents the distance feature vector z ij The k-th nearest nominal eigenvector.

[0148] Using the above detection method, this embodiment can accurately determine the abnormalities at each location on the surface of the industrial product to be inspected, thereby accurately locating the defective area on the surface of the industrial product. It is easy to understand that after locating the defect in the image, the image content at that location can be extracted to obtain the specific data of the defect.

[0149] The following explanation, based on specific detection results, further illustrates the beneficial effects achievable by this invention. Using the above detection method, the surface anomaly localization effect on different industrial products in the MVTec AD dataset is as follows: Figure 5 As shown, there are 15 categories. In the detection results of each category, the first to third rows are the abnormal image to be tested, the labeled ground truth, and the detection result, respectively. Each column represents the detection result of a defect type. Among them, the 15 categories shown in (a) to (o) are bottles, cables, capsules, carpets, nets, hazelnuts, feathers, metal nuts, pills, screws, tiles, toothbrushes, transistors, wood, and zippers.

[0150] The detection category is labeled below each image group, with the defect category indicated in parentheses. The detection results show that this embodiment can accurately locate anomalies under different detection categories and various defect categories, demonstrating good generalization performance.

[0151] The model implemented in this embodiment provides a visual representation of the characteristics of the domain drift problem. Figure 6 and Figure 7 As shown, where, Figure 6 The results shown are the test results related to metal nuts. Figure 7 The results shown are related to leather testing. Figure 6 and Figure 7 The second column shows the feature distribution of the target dataset before domain adaptation, indicating that a significant number of normal and anomalous feature vectors are far removed from the nominal feature vectors, becoming mixed together and indistinguishable. The third column shows the feature distribution after adding clustering loss. Clustering loss narrows the gap between normal and nominal feature vectors, avoiding a considerable proportion of false positives. However, the feature distribution in the third column shows that some difficult-to-classify anomalous feature vectors are still mixed in with normal feature vectors, reducing detection accuracy. The fourth column shows the feature distribution after further adding separation loss. The results in the fourth column show that separation loss reduces the relative distance between normal and anomalous feature vectors, further improving the model's ability to distinguish difficult samples.

[0152] In summary, this embodiment can eliminate the "domain drift" problem and achieve pixel-level detection of surface anomalies in industrial products. Its detection results effectively locate the defect areas on the surface of industrial products, which is beneficial for tracing and improving the corresponding processes and is of great significance for improving actual production efficiency.

[0153] Example 2:

[0154] An industrial product surface defect detection device, comprising:

[0155] A computer-readable storage medium for storing computer programs;

[0156] And a processor, used to read the computer program stored in the computer-readable storage medium and execute the industrial product surface anomaly detection method provided in Embodiment 1 above.

[0157] Example 3:

[0158] An industrial product surface anomaly detection system, comprising:

[0159] A camera used to capture images of the surface of industrial products to be inspected;

[0160] The industrial product surface anomaly detection device provided in Embodiment 2 above is connected to a camera.

[0161] The industrial product surface anomaly detection system provided in this embodiment can detect defects on the surface of industrial products in real time.

[0162] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting surface anomalies in industrial products, characterized in that, include: Training phase: Obtain a training set consisting of surface images of normal industrial products; Multiple images are selected from the training set and the three-dimensional features of each image are extracted. The mean value of the feature vector at each position is calculated as the nominal feature vector, and all nominal feature vectors constitute the feature memory library. A network model is constructed, comprising a feature extraction module, a positive and negative feature vector pair generation module, and a feature mapping module. The network model is then trained using the training set. During training, the feature extraction module is used to extract images. I nor 3D features Z nor The positive and negative feature vector pair generation module is used to generate corresponding abnormal features. Z abnor From normal characteristics Z nor and abnormal features Z abnor Extract foreground features to obtain features. Z' nor and Z' abnor and utilize features Z' nor and Z' abnor Feature vectors at the same position are respectively used as normal feature vectors and abnormal feature vectors, forming positive and negative feature vector pairs; the feature mapping module is used to map normal feature vectors and abnormal feature vectors to the feature space where the nominal feature vectors are located; the training loss function includes clustering loss and separation loss; the clustering loss is used to measure the abnormal distance of normal feature vectors, and the larger the abnormal distance, the larger the clustering loss; the separation loss is used to measure the difference in abnormal distance between normal feature vectors and their corresponding abnormal feature vectors, and this difference is greater than a preset threshold. margin At that time, the separation loss is 0, and not greater than 0. margin When the difference is smaller, the separation loss is positive and the separation loss is greater. The feature extraction module includes a feature extractor Backbone and a feature optimization module FRM; The feature extractor Backbone is used to extract multi-scale features from the image; The Feature Optimization Module (FRM) is used to optimize the multi-scale features. L Layer features are combined and fused into three-dimensional features; The Feature Optimization Module (FRM) includes: a feature fusion module and a Pyramid Attention Module (PAM) set between every two adjacent layers of features; The pyramid attention module (PAM) includes: A globally average pooling layer, a 1×1 convolutional layer, and a softmax activation function layer are connected in sequence to process high-dimensional features from adjacent layers. Z high Perform the corresponding operations to obtain the features. A ; The dot product layer is used to perform positional features in adjacent layers. and characteristics A Perform a dot product to obtain the features. ; And residual structures, used according to Calculate the optimized features , as the output of the Pyramid Attention Module (PAM); The feature fusion module is used to upsample the first... L Layer features, and L -1 optimized features are upsampled to the same size and then concatenated to obtain 3D features; where the anomaly distance of the feature vector is the distance between the feature vector and the nearest... K The average distance between the nominal feature vectors; K The value is a preset positive integer; the feature extractor Backbone is a pre-trained model trained using a natural image dataset; 1 < L ≤ L max , L max This indicates the maximum number of feature layers extracted by the Backbone feature extractor; features A ,feature and features The size is related to the feature same; γ These are trainable parameters; Testing phase: Image of the surface of the industrial product to be inspected I The input is fed into the feature extraction module to obtain three-dimensional features. Z Then, the feature mapping module is used to map the vectors to the feature space containing the nominal feature vectors to obtain the features. Z' For features Z' For each position in the feature vector, calculate its relationship with the nearest neighbor. K The weighted sum of the distances between the nominal feature vectors is used as the anomaly score at the corresponding position to obtain the anomaly score map; Upsample the anomaly score map to the image. I After applying Gaussian filtering to the same size, the image is obtained. I The location of the defect is identified, and anomaly detection is completed.

2. The method for detecting surface anomalies in industrial products as described in claim 1, characterized in that, The training phase further includes updating the nominal feature vectors in the nominal feature library during the training of the network model using the training set. In the it In the training iteration, for any _th t nominal eigenvectors The update method is as follows: Obtain the nominal feature vector from a pre-built normal feature vector library. Recent Q We obtain the corrected feature vector by summing the normal feature vectors and performing a weighted summation. z r Among the normal eigenvectors participating in the weighted summation, the nominal eigenvector... The greater the distance, the smaller its weight in the weighted summation; according to nominal vector Updated to ; in, α The momentum parameter represents the degree of update control; 0 < α <1.

3. The method for detecting surface anomalies in industrial products as described in claim 1, characterized in that, The Feature Optimization Module (FRM) further includes: a local average pooling layer set after each layer of features and before each pyramid attention module (PAM); Furthermore, the features Z high and characteristics All of these are features obtained by local average pooling of the original features.

4. The method for detecting surface anomalies in industrial products as described in claim 1 or 2, characterized in that, The generation of corresponding abnormal features Z abnor This includes at least one of the following methods: Method 1: Generation and Image I nor Images of the same size with Berlin noise And thresholding it to locate irregular defect areas. ; Calculate the image I nor Foreground mask image Combine it with the irregular defect area Perform an AND operation to obtain the mask. ; Obtain another image from the training set. Add defects to it to obtain a defect image. ; according to Generate defect images ; The defect image is extracted using the feature extraction module. The three-dimensional features were obtained, and the normal features were compared with those of the normal features. Z nor Corresponding abnormal features ; in, This represents the AND operation; β >0 indicates parameters obtained through random sampling; Method 2: Noise features are obtained by randomly sampling from a Gaussian distribution. ; Calculate the image I nor Foreground mask image Combine it with noise characteristics After performing an AND operation to obtain the foreground noise features, they are fused into the normal features. Z nor In the middle, the normal characteristics were obtained. Z nor Corresponding abnormal features Z abnor .

5. The method for detecting surface anomalies in industrial products as described in claim 4, characterized in that, The expression for the clustering loss is: in, L cls Represents the clustering loss. N c This represents the number of normal eigenvectors. K This indicates the number of nominal feature vectors recalled by the KNN algorithm. D ( ) represents vector distance; v nor This represents the normal feature vector after mapping by the feature mapping module. Indicates and v nor No. k The nearest nominal eigenvector; Indicates the feature vector index.

6. The method for detecting surface anomalies in industrial products as described in claim 5, characterized in that, The expression for the separation loss is: in, L split Indicates separation loss. n s This indicates the number of pairs of positive and negative eigenvectors. and These represent the normal feature vector and the abnormal feature vector after the feature mapping module has been applied to the positive and negative feature vector pairs, respectively. Represents the normal feature vector v p Distance to k The nearest nominal eigenvector, Represents anomaly feature vectors v n Distance to k The nearest nominal eigenvector; Indicates the feature vector index.

7. A device for detecting surface anomalies in industrial products, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in a computer-readable storage medium and executing the industrial product surface anomaly detection method according to any one of claims 1 to 6.

8. A surface anomaly detection system for industrial products, characterized in that, include: A camera used to capture images of the surface of industrial products to be inspected; And the industrial product surface anomaly detection device as described in claim 7, which is connected to the camera.

Images