A self-supervised surface defect anomaly positioning and detection method based on image segmentation

Abstract

The present application relates to the field of image anomaly positioning, solves the problems of less prior knowledge, less sample quantity and type, lack of supervision information, and inability to realize high-quality pixel-level anomaly fine positioning, and especially relates to a surface defect anomaly positioning and detection method based on image segmentation under self-supervision, a lightweight image segmentation subnetwork, in the case of multiple samples, using the self-supervision information provided by the data augmentation network, and the necessary information provided by the reconstruction subnetwork with channel attention mechanism to assist anomaly segmentation, completing the pixel-level anomaly fine positioning of the surface defect anomaly sample, and based on the anomaly feature layer output by the segmentation network, through processing such as aggregating local response information, obtaining the instance-level anomaly score corresponding to the sample, and simultaneously realizing the instance-level anomaly detection task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image anomaly detection, and in particular to a surface defect anomaly positioning and detection method based on image segmentation under self-supervision. BACKGROUND

[0002] Anomaly detection in the image field is an important machine learning problem. Since in many practical application scenarios, only normal image samples can be collected, abnormal image samples are rare and difficult to collect, there is no effective monitoring information that can be used, and moreover, abnormal samples or abnormal patterns are usually variable in shape, color and size, and do not have stable statistical rules, all of which will make it difficult for machine learning models to capture enough statistical information or significant features of abnormal image patterns, and thus in the field of image anomaly detection, unlike most existing machine learning methods based on the assumption of a static and closed system, it studies how machine learning models use only partial knowledge (normal image data or distribution) to process unknown and uncertain information (varied abnormalities) in an open and dynamic system environment. Because the prior knowledge used by general anomaly detection algorithms is too little, it is difficult for related models to detect complex and varied types of abnormalities, and therefore, how to provide more prior knowledge, including more number and type of training samples and supervision information, has become one of the problems to be solved in image anomaly detection.

[0003] Due to different image detection granularities, image anomaly detection can be divided into image-level and pixel-level anomaly detection. Among them, image-level anomaly detection usually only focuses on the problem of whether the entire image is normal or abnormal, while pixel-level anomaly detection needs to detect or locate the abnormal area in the image. On industrial images, surface defect anomalies mainly refer to detecting scratches, defects, foreign object occlusions, color contamination, holes and other defects on the sample surface. These defect anomalies often only occupy a small part of the entire picture and are usually very close to normal samples, so the detection of such pictures is usually fine-grained and requires pixel-level anomaly detection and positioning. At present, the commonly used image anomaly positioning methods mainly include: Grad-CAM visualization positioning based on the entire image, but it can only achieve coarse anomaly positioning and often contains many normal areas; positioning based on the reconstructed image pixel residual heat map or dividing the entire image into multiple patches and positioning based on the structural similarity SSIM between the reconstructed images in each region, but due to the model generalization problem of the reconstruction type anomaly detection method, the pixel-level anomaly positioning and instance-level anomaly detection performance cannot be optimized simultaneously, and there is a contradiction. Therefore, in industrial images, how to achieve high-performance pixel-level positioning and detection of surface defect anomalies has become one of the problems to be solved in this field.

[0004] In the field of images, a common task is to classify images, which is a relatively coarse-grained task to identify and understand images. That is, given an image, the model is expected to output the category (discrete label) to which the image belongs. However, image semantic segmentation is more refined than the image classification task. Image semantic segmentation achieves fine-grained inference by making dense predictions for each pixel to infer labels, so that each pixel is labeled with its corresponding category, thereby performing accurate segmentation of each category region, which is pixel-level image classification. In the field of image semantic segmentation, UNet network is one of the classic networks, which is designed based on convolutional neural network architecture. It was first proposed by German image processing expert Olaf Ronneberger et al. in 2015 and applied to the field of computer vision to complete the semantic segmentation task. It was first applied to biomedical image segmentation, can effectively extract image features, and has achieved high segmentation accuracy. Through the structure of the contraction path and the expansion path, the feature fusion is realized by using the jump structure, which avoids the problem of affecting the network segmentation accuracy due to the loss of part of the effective feature information in the dimension reduction process. The entire network has high segmentation accuracy, high training efficiency, is friendly to small sample data, has simple and clear structure, is easy to explain, expand and modify, and has other advantages, so it has been widely applied. However, this network uses VGG16 as a feature extraction network, and the multiple convolutional layers, up-sampling and down-sampling operations bring a large amount of parameters and computation to the entire algorithm, which affects the speed of model training and inference, has poor real-time performance, and requires higher hardware configuration, often requiring higher computing power and larger storage space to store model parameters and intermediate feature maps.

[0005] In view of the pixel-level classification function realized by image semantic segmentation, how to introduce image segmentation into the field of anomaly detection to realize pixel-level positioning and detection of unknown anomalies, and how to realize the lightweight of the model while maintaining the detection accuracy of the model are problems worthy of research and exploration. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a surface defect anomaly positioning and detection method based on image segmentation under self-supervision, which proposes a data augmentation network under the condition of only normal samples, realizes random enhancement of positive samples and construction of pseudo-anomaly samples, expands the sample types while automatically generating labels and pixel-perfect anomaly maps corresponding to the samples for subsequent network supervision, preliminarily alleviates the problems of insufficient sample types, difficulty in collecting anomaly samples and complexity of manual sample labeling in the anomaly detection field, and provides more prior knowledge; a reconstruction subnetwork based on the SE channel attention mechanism is proposed to realize high-quality anomaly-free feature reconstruction of the augmented network output samples, and the anomaly-free feature reconstruction samples are significantly different in appearance from the original input samples, thus providing effective auxiliary information for subsequent anomaly segmentation; a mobileUNet image segmentation network based on the fusion of mobileNet and UNet is proposed to realize anomaly positioning and detection, which realizes more fine positioning of local small areas compared with the Grad-CAM visualization method, and realizes more lightweight segmentation compared with the traditional UNet network, and the whole algorithm has high performance of detection and lightweight of the model.

[0007] In order to solve the above technical problems, the present application provides the following technical scheme: a surface defect anomaly positioning and detection method based on image segmentation under self-supervision, comprising the following steps:

[0008] S1, acquiring training samples;

[0009] S2, inputting the training samples into a data augmentation network to realize random enhancement of the input samples;

[0010] S3, sending the random enhanced samples output by the augmentation network into a reconstruction subnetwork;

[0011] S4, concatenating the random enhanced samples output by the augmentation network and the output samples obtained by inputting the random enhanced samples into the reconstruction subnetwork at the channel, and sending them into an image segmentation subnetwork;

[0012] S5, starting training of the model, and reversely propagating the loss function to constantly update the parameters of each layer of the reconstruction subnetwork and the image segmentation subnetwork until the training is completed;

[0013] S6, acquiring test samples;

[0014] S7, directly inputting the test samples into the reconstruction subnetwork;

[0015] S8, concatenating the test samples and the output samples obtained by inputting the test samples into the reconstruction subnetwork at the channel, and sending them into the image segmentation subnetwork to obtain a pixel-level anomaly positioning map;

[0016] S9, obtaining an instance-level anomaly score corresponding to the test samples from the anomaly positioning map.

[0017] Further, in step S2, the construction method of the data augmentation network comprises the following steps:

[0018] S21 build a data augmentation network to realize normal enhancement of input normal samples or introduce local irregular anomalies to construct pseudo abnormal samples;

[0019] S22 output the enhanced normal samples or the constructed pseudo abnormal samples, and their corresponding labels and pixel perfect anomaly maps, for subsequent supervision.

[0020] Further, in step S3, the construction method of the reconstruction subnetwork comprises the following steps:

[0021] S31 build the basic structure of the reconstruction subnetwork;

[0022] S32 build an SE channel attention module;

[0023] S33 integrate the SE module into the reconstruction subnetwork to form a reconstruction subnetwork based on the SE channel attention mechanism.

[0024] Further, in step S31, the reconstruction subnetwork is composed of an encoder-decoder pair, the encoder structure is similar to VGGNet, a deep neural network is built to reduce the dimension of the input sample and extract multi-level features, and then multi-layer up-sampling is performed to realize the reconstruction of the input sample without abnormal features.

[0025] Further, in step S32, the SE channel attention module is composed of a global pooling layer, a dimension reduction fully connected layer, a ReLU activation layer, a dimension increase fully connected layer, and a sigmoid activation layer. Through the three operations of Squeeze, Excitation, and Scale, the input feature map is enhanced in each channel, so that the neural network focuses on some feature channels, and the feature channels useful for the current task are enhanced and the feature channels useless for the current task are suppressed.

[0026] Further, in step S33, the SE module is added after the multi-level features extracted by the convolution layer in the encoder part of the reconstruction subnetwork, to effectively enhance the normal features and ignore the abnormal features that will affect the reconstruction without abnormalities.

[0027] Further, in step S4, the construction method of the image segmentation subnetwork comprises the following steps:

[0028] S41 build the basic structure of the mobileNetv1 network;

[0029] S42 build the basic structure of the UNet image segmentation network;

[0030] S43 realizes model fusion, and builds the final mobileUNet image segmentation network.

[0031] Further, in step S41, the mobileNetv1 uses a combination of multiple deep convolution layers and pointwise convolution layers to perform dimension reduction and feature extraction of the picture, reduces the parameter amount in the form of deep separable convolution, reduces the model size while ensuring the same input and output;

[0032] Further, in step S42, the left shrinkage path in the UNet network adopts a combination of 6 standard convolution layers, BN layers, ReLU activation layers and the last large pooling layer for down-sampling processing, and realizes feature extraction of the input picture. In the symmetric up-sampling process of restoring the image, the low-level and high-level features obtained by down-sampling in the left shrinkage path are transmitted to the right expansion path in a jump connection form, and the same size feature maps on both sides are cascaded and then continue to be up-sampled.

[0033] Further, in step S43, the structure of mobile-Netv1 is fused with the original UNet structure, and a deep separable convolution kernel is used instead of a large number of convolution layers in the original UNet network for deep feature extraction in the original expansion path.

[0034] Further, in step S5, the model starts training, and the loss function is back propagated. The total loss function in the training stage is the sum of the loss of abnormal feature reconstruction of the reconstruction subnetwork and the loss of accurate segmentation of the image segmentation subnetwork, which includes the following steps:

[0035] S51 loss L of abnormal feature reconstruction of the reconstruction subnetwork G-Net ;

[0036] S52 loss L of accurate segmentation of the image segmentation subnetwork Seg-Net .

[0037] Further, in step S51, the function of the reconstruction subnetwork is to realize the abnormal feature reconstruction of the output sample obtained through the data augmentation network. The closest normal sample to the output sample is the original sample input into the data augmentation network. Therefore, the loss function L of the network is set as the structural similarity loss L between the reconstructed sample of the network and the original sample input into the data augmentation network, and the image pixel loss L G-Ne . SSIM ;

[0038] Further, in step S52, the function of the image segmentation subnetwork is to realize the accurate segmentation of the input image, that is, to make the abnormal positioning map output by the network as similar as possible to the abnormal map (label map) corresponding to the sample output in the data augmentation stage, so as to reduce the difference between the network output prediction value and the true label as the objective function. Therefore, the loss function L of the network is set as the structural similarity loss L between the output of the network and the label map corresponding to the sample output in the data augmentation stage, and the image pixel loss L con .Seg-Net The focal loss is set between the network output feature map and the label map.

[0039] Further, in step S8, the test sample and the output sample obtained by the test sample entering the reconstruction sub-network are concatenated at the channel, and are sent to an image segmentation sub-network to obtain a pixel-level anomaly localization map, and the pixel-level anomaly localization map includes the following steps:

[0040] S81In the anomaly detection field, a pixel point is divided into two categories: normal and abnormal, so that the image segmentation sub-network finally outputs two feature layers, including a normal layer and an abnormal layer.

[0041] S82After the two feature layers are normalized by a softmax activation function in the channel dimension, the abnormal layer is obtained.

[0042] S83The numerical value of the abnormal layer is converted into a corresponding gray value, that is, the picture is grayed to obtain the anomaly localization map corresponding to the test sample.

[0043] Further, in step S9, an instance-level anomaly score corresponding to the test sample is obtained from the anomaly localization map, and the instance-level anomaly score includes the following steps:

[0044] S91Two-dimensional average pooling operation is performed on the abnormal layer, and the step length stride is 1. By means of 0 supplement, a smoothing operation is performed on the abnormal map without changing the output size, and local abnormal response information is aggregated.

[0045] S92The global maximum value of the smoothed abnormal map is taken as the instance-level anomaly score corresponding to the test sample.

[0046] By means of the above technical scheme, the present application provides an anomaly localization and detection method based on image segmentation under self-supervision, which has at least the following beneficial effects:

[0047] 1. In view of the problems of insufficient number of samples in the training set, and difficult collection and incomplete types of abnormal samples in the anomaly detection field, the present application expands the number of samples through a data augmentation network, including normal samples and introduced pseudo-abnormal samples with local irregularities. The present application generates labels and abnormal maps corresponding to the random enhanced samples while outputting the random enhanced samples, provides supervision information for the subsequent network training, changes the general unsupervised and supervised network into a self-supervised network, and saves the manpower and time cost in data preprocessing, sample collection, label calibration and the like.

[0048] 2. End-to-end training and detection are realized, the problems of deviation of each module to the subsequent module and inconsistency between the target function of a certain module and the system macroscopic total target are overcome, the performance cannot be optimized, and the model has more space for automatic adjustment according to data, and the overall fitting degree of the model is increased.

[0049] 3. In the industrial data set, surface defect anomalies often only occupy a small part of the whole sample, that is, only local pixel positions exist anomalies, and the present application improves the image segmentation method and realizes the pixel-level accurate positioning of local small area anomalies.

[0050] 4. The SE channel attention module is plug and play, which can exchange higher performance of the model for slight additional calculation cost, increases the attention mechanism in the reconstruction subnetwork, makes the neural network in the reconstruction subnetwork pay more attention to the normal features that can realize the sample anomaly-free reconstruction, reduces the attention and reconstruction of abnormal features as much as possible, and makes the input sample and its anomaly-free feature reconstruction sample significantly different in appearance. When the two are sent into the image segmentation network after channel concatenation, it can provide necessary information for abnormal segmentation, thereby improving the detection performance of the algorithm.

[0051] 5. The fusion of mobileNet and UNet network uses depth separable convolutional layers instead of a large number of standard convolutional layers in the original UNet network shrinkage path, greatly reduces the parameter amount of the model while retaining the image segmentation accuracy of UNet, that is, a lighter segmentation network is used to realize higher positioning and detection performance. BRIEF DESCRIPTION OF DRAWINGS

[0052] The drawings described herein are used to provide further understanding of the present application, constitute a part of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitation on the present application. In the drawings:

[0053] Figure 1 The flowchart of the anomaly detection of the present application;

[0054] Figure 2 The overall block diagram of the anomaly detection algorithm of the present application;

[0055] Figure 3 The basic structure diagram of the reconstruction subnetwork of the present application;

[0056] Figure 4 The basic structure diagram of the SE channel attention module of the present application;

[0057] Figure 5 The reconstruction subnetwork structure diagram based on the SE attention mechanism of the present application;

[0058] Figure 6 The image segmentation subnetwork structure diagram based on UNet of the present application;

[0059] Figure 7A schematic diagram of an image segmentation subnetwork structure based on mobileNet-UNet fusion of the present application;

[0060] Figure 8 A calculation example of focal loss in the image segmentation subnetwork of the present application;

[0061] Figure 9 A flowchart of the test stage of the present application;

[0062] Figure 10 An instance-level abnormal performance comparison of the present application with a baseline algorithm;

[0063] Figure 11 A pixel-level abnormal positioning result comparison of the present application with a baseline algorithm;

[0064] Figure 12 Performance exploration of the present application on the image-level AUC / image-level AP index;

[0065] Figure 13 Performance exploration of the present application on the pixel-level AUC / pixel-level AP index;

[0066] Figure 14 Performance exploration of the present application on the parameter total index. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments. The implementation process of how to apply technical means to solve technical problems and achieve technical effects of the present application can be fully understood and implemented.

[0068] Please refer to Figures 1-12 , which shows a specific embodiment of the present embodiment, which is based on a lightweight image segmentation subnetwork. In the case of multiple types of samples, the self-supervised information provided by the data augmentation network and the necessary information for auxiliary abnormal segmentation provided by the reconstruction subnetwork with channel attention are used to complete the pixel-level abnormal fine positioning of the surface defect abnormal sample. Based on the abnormal feature layer output by the segmentation network, the local abnormal response information is aggregated and processed to obtain the instance-level abnormal score corresponding to the sample, which has high performance and high computational efficiency.

[0069] Please refer to Figure 1 、 Figure 2 The present embodiment proposes an image segmentation-based abnormal positioning and detection method under self-supervision, which includes the following steps:

[0070] S1 obtains training samples, and test samples are derived from a public dataset mvtec_ad;

[0071] Mvtec_ad dataset, which simulates the actual production scene of industry and is mainly used for unsupervised anomaly detection, contains 15 object categories, each object has 4-6 fault performances, involving 73 fault categories, and is divided into 15 tasks for separate training and testing according to object categories during experiment.

[0072] S2 input the training sample data into the data augmentation network to realize random enhancement of the input sample;

[0073] S3 send the random enhanced sample output by the augmentation network into the reconstruction sub-network;

[0074] S4 cascade the random enhanced sample output by the augmentation network and the output sample obtained by inputting it into the reconstruction sub-network at the channel, and send it into the image segmentation sub-network;

[0075] S5 start training the model, and constantly update the parameters of each layer of the reconstruction sub-network and the image segmentation sub-network through loss function back propagation until the training is completed;

[0076] S6 obtain a test sample;

[0077] S7 directly input the test sample into the reconstruction sub-network;

[0078] S8 cascade the test sample and the output sample obtained by inputting it into the reconstruction sub-network at the channel, and send it into the image segmentation sub-network to obtain a pixel-level anomaly localization map;

[0079] S9 obtain an instance-level anomaly score corresponding to the test sample from the anomaly localization map.

[0080] Further, in step S2, the construction method of the data augmentation network comprises the following steps:

[0081] S21 build a data augmentation network to realize normal enhancement of the input normal sample or introduce local irregular anomaly to construct a pseudo-anomaly sample;

[0082] S22 output the enhanced normal sample or the constructed pseudo-anomaly sample, and their corresponding labels and pixel perfect anomaly maps.

[0083] Set two random values a and β between 0 and 1;

[0084] When a> 0.5, only the normal sample I is randomly enhanced, including rotation, bias, sharpness, sunlight, balance, brightness change, color change, automatic contrast change and other operations, without abnormal processing, and the enhanced normal sample I is output aug , and its label is set to 0, and since there is no anomaly, the corresponding output anomaly map M is set to a pure black picture with the same size as the input picture;

[0085] When a is less than or equal to 0.5, the normal sample is processed abnormally, a local irregular anomaly is introduced, when b is greater than 0.5, a pseudo-anomaly sample is generated based on cutpaste enhancement, when b is less than or equal to 0.5, a pseudo-anomaly sample is generated based on Berlin noise, and finally the enhanced pseudo-anomaly sample I is output aug The label is set to 1, and the output anomaly map M shows that the normal area is black and the local anomaly is white.

[0086] Further, in step S3, the construction method of the reconstruction sub-network includes the following steps:

[0087] S31 builds the basic structure of the reconstruction sub-network;

[0088] S32 builds the SE channel attention module;

[0089] S33 integrates the SE module into the reconstruction sub-network to form a reconstruction sub-network based on the SE channel attention mechanism.

[0090] Further, please refer to Figure 3 In step S31, the reconstruction sub-network is composed of an encoder-decoder pair, the encoder structure is similar to VGGNet, a deep neural network is built, the input sample is reduced in dimension and multi-level feature extraction is performed, and then multi-layer up-sampling is performed to realize anomaly-free reconstruction of the input sample.

[0091] The encoder structure is similar to VGGNet, a 14-layer deep neural network is successfully constructed by repeatedly stacking 3*3 small convolution kernels and 2*2 max pooling layers. Using multiple small convolution kernels in series instead of larger convolution kernels can not only ensure the receptive field and increase the feature learning ability, but also reduce the parameter amount of the convolution layer, and the addition of multiple activation layers also increases the non-linear transformation of the network. In addition, deeper networks and smaller convolution kernels also bring implicit regularization effects.

[0092] Further, please refer to Figure 4 In step S32, the SE channel attention module is composed of a global pooling layer, a dimension reduction fully connected layer, a ReLU activation layer, a dimension increasing fully connected layer, and a sigmoid activation layer. Through the three operations of Squeeze, Excitation, and Scale, the input feature map is enhanced in each channel, so that the neural network can focus on some feature channels, and the feature channels useful for the current task are enhanced and the feature channels not useful for the current task are suppressed.

[0093] Further, please refer to Figure 5 In step S33, the SE module is added to the multi-level features extracted by the convolution layer in the encoder part of the reconstruction sub-network, which can effectively enhance the normal features and ignore the abnormal features that affect the anomaly-free reconstruction.

[0094] Further, in step S4, the construction method of the image segmentation subnetwork includes the following steps:

[0095] S41 building the basic structure of mobileNetv1 network;

[0096] S42 building the basic structure of UNet image segmentation network;

[0097] S43 model fusion is realized, and the final mobileUNet image segmentation network is built.

[0098] Further, in step S41, mobileNetv1 uses a combination of multiple deep convolutional layers and pointwise convolutional layers to reduce the dimension of the picture and extract features, reduces the parameter amount through the form of deep separable convolution, and reduces the model size while ensuring the same input and output;

[0099] Further, please refer to Figure 6 In step S42, the left shrinkage path in the UNet network adopts a combination of 6 standard convolutional layers, BN layers, ReLU activation layers and the last large pooling layer for down-sampling processing, so as to realize feature extraction of the input picture. In the process of symmetric up-sampling to restore the image, the low-level and high-level features obtained by down-sampling in the left shrinkage path are transmitted to the right expansion path in a jump connection form, and the same size feature maps on both sides are cascaded and then continue to be up-sampled;

[0100] Further, please refer to Figure 7 In step S43, the structure of mobile-Netv1 is fused with the original UNet structure. In the original expansion path, the deep separable convolution kernel is used instead of a large number of convolutional layers in the original UNet network for deep feature extraction. The features obtained after deep separation convolution are also transmitted to the right network in a jump connection form. The same size feature maps on both sides are cascaded and then continue to be up-sampled in the same way as in the UNet expansion network, and finally restored to the size of the original input picture 256*256. Since there are two classes in anomaly detection, namely normal and abnormal classes, the number of feature layers output by the network is 2, that is, the size of the output two channels is 256*256 feature map.

[0101] Further, in step S5, the model starts training, and the loss function is back-propagated. The total loss function in the training stage is the sum of the loss of the reconstruction subnetwork without abnormal features and the loss of the accurate segmentation of the image segmentation subnetwork, which includes the following steps:

[0102] S51 reconstruction loss L of the subnetwork without abnormal features G-Net ;

[0103] S52 image segmentation subnetwork accurate segmentation loss L Seg-Net .

[0104] Further, in step S51, the function of the reconstruction subnetwork is to realize the anomaly-free feature reconstruction of the output sample obtained by the data augmentation network. The normal sample closest to it is the original sample input to the data augmentation network. Then the loss function L G-Net of the network is set as the reconstruction sample I rec . SSIM and the image pixel loss L con ;

[0105] L G-Net = a1L con (I, I rec ) + a2L SSIM (I, I rec )

[0106] L con (I, I rec ) = ‖I - I rec ‖2

[0107]

[0108]

[0109] Where a1 and a2 are the weights of the two losses, H and W are the height and width of the input image I, N p is the number of pixels in the input image I, SSIM(I, I r ) (i,j) is the structural similarity value of the input image I and the reconstructed image I rec at the same patch centering on the image coordinates (i, j). μ is the mean value, σ 2 is the variance, is the covariance, and c is a constant used to maintain stability.

[0110] Further, in step S52, the function of the image segmentation subnetwork is to realize the accurate segmentation of the input image, that is, to make the anomaly positioning map output by the network as similar as possible to the anomaly map (label map) corresponding to the sample output in the data augmentation stage. The difference between the network output prediction value and the true label is taken as the objective function, and then the loss function L Seg-Net of the network is set as the focal loss between the network output feature map M a and the label map M. The specific expression is as follows, where y i represents the label of sample i, and the positive class is 1 and the negative class is 0; p idenotes the probability that sample i is predicted to be positive class. γ is a tuning parameter for this loss function, k i is a balancing factor.

[0111] L Seg-Net = FL(M, M a )

[0112] FL = -k i (1-p i ) γ y i log(p i )-k i (p i ) γ (1-y i )log(1-p i )

[0113] Please refer to Figure 8 , taking a 2*2 feature map and a label map as an example, the calculation process of focal loss in the network is specifically shown. The output feature layer M a is first normalized from the channel dimension by the softmax function, and the numerical value corresponding to each pixel point on the feature layer is converted into a probability between 0 and 1. Taking the abnormal image M generated by the data augmentation network as the ground truth (GT), a label map M one-h corresponding to the output feature layer of the image segmentation subnetwork is generated according to the one-hot encoding method. Then the actual label value and the predicted probability value are calculated according to the focal loss calculation formula, and finally the loss sum of all pixel points is averaged to obtain the final loss, which is back propagated to update the network weight.

[0114] Further, please refer to Figure 9 , in step S8, the test sample and the output sample obtained by the test sample entering the reconstruction subnetwork are concatenated in the channel, and are sent into the image segmentation subnetwork to obtain a pixel-level abnormal positioning map, which specifically includes the following steps:

[0115] S81In the field of anomaly detection, pixel points are divided into normal and abnormal two categories, so the image segmentation subnetwork finally outputs two feature layers, including a normal layer and an abnormal layer;

[0116] S82After the two feature layers are normalized by the softmax activation function in the channel dimension, the abnormal layer is taken;

[0117] S83After converting the numerical value of the abnormal layer into the corresponding gray value, the abnormal positioning map corresponding to the test sample is obtained after the picture is grayed.

[0118] Further, in step S9, the instance-level anomaly score corresponding to the test sample is obtained from the anomaly localization map, specifically including the following steps:

[0119] S91 performs a two-dimensional average pooling operation on the anomaly layer, with a step length stride of 1, and performs a smoothing operation on the anomaly map by means of 0 padding without changing the output size, to aggregate local anomaly response information;

[0120] S92 takes the global maximum value of the smoothed anomaly map as the instance-level anomaly score corresponding to the test sample.

[0121] Please refer to Figure 10 In terms of instance-level anomaly detection, the area under the ROC curve AUC at the image level is selected as the performance indicator, and the performance of the present application is compared with that of two baseline algorithms. It is found that the detection AUC of the present application is as high as 0.9576, which is about 2.5% higher than the performance of the best baseline algorithm.

[0122] Please refer to Figure 11 In terms of pixel-level anomaly localization, for abnormal class pictures, the anomaly localization map generated based on the Grad-CAM visualization method is compared with the anomaly localization map generated based on image segmentation in the present application. It is observed that both localization methods accurately locate the local irregular anomaly in the test sample. However, when the local anomaly is very small or narrow, the Grad-CAM-based localization often locates a large area containing the small anomaly, which is a relatively coarse localization, while the image segmentation network-based localization is more accurate and tends to be pixel-level fine localization.

[0123] Please refer to Figure 12 , 13, 14, In order to further illustrate the advancement and effectiveness of the present application in the field of example level industrial image anomaly detection, and the role of each module proposed, an ablation experiment is designed, starting from the performance indicators of image level AUC, AP, pixel level AUC, AP, exploring the network performance difference before and after adding the SE module and mobileNet module, starting from the parameter quantity, exploring the network efficiency difference before and after adding the two modules. By comparing the present application with the network using only the ordinary UNet, it is found that the pixel level AUC of the present application is 0.9477, which is slightly lower than that of the network using only the ordinary UNet, which is 0.9483, but the image level AUC, image level AP and pixel level AP of the present application are 0.9576, 0.9834 and 0.5811 respectively, which are about 2.5%, 1.5% and 1% higher respectively in the above three indicators, so it can be seen that the use of the present application effectively improves the anomaly detection performance. By comparing the total parameter amount of the network using only the UNet and the network proposed by the present application, it is found that the network proposed by the present application realizes lighter anomaly detection, and in terms of comprehensive performance, it can be seen that the present application uses smaller parameters to realize higher detection performance, and takes into account the recognition accuracy and detection efficiency.

Claims

1. A self-supervised image segmentation based surface defect anomaly localization and detection method, characterized in that: The method comprises the following steps: The training sample input data is input into the data augmentation network to realize random enhancement of the input sample; The random enhanced sample output by the data augmentation network is sent into the reconstruction sub-network, and an SE channel attention module is added in the reconstruction sub-network; The random enhanced sample output by the data augmentation network and the output sample obtained by sending the random enhanced sample into the reconstruction sub-network are cascaded at the channel and sent into the image segmentation sub-network, wherein the image segmentation sub-network is a mobileUNet image segmentation network fused by a mobileNetv1 and a UNet; The model starts training, the loss function is back propagated, and the parameters of each layer of the reconstruction sub-network and the image segmentation sub-network are continuously updated until the training is completed; The test sample and the output sample obtained by sending the test sample into the trained reconstruction sub-network are cascaded at the channel and sent into the trained image segmentation sub-network to obtain a pixel-level anomaly localization map; An instance-level anomaly score corresponding to the test sample is obtained from the anomaly localization map, and whether the test sample is a normal class or an abnormal class is determined according to the score; The construction method of the image segmentation sub-network comprises the following steps: The basic structure of the mobileNetv1 network is built; The basic structure of the UNet image segmentation network is built; in the UNet network, the left shrinkage path adopts a combination of a plurality of standard convolution layers, a BN layer, a ReLU activation layer and a last large pooling layer for down-sampling processing to realize feature extraction of the input picture; in the process of symmetric up-sampling to restore the image, the low-level and high-level features obtained by down-sampling in the left shrinkage path are transmitted to the right expansion path in a jump connection mode, and the same size feature maps on both sides are cascaded and then continue to be up-sampled; The model fusion is realized, and the final mobileUNet image segmentation network is built; in the original expansion path, a deep separable convolution layer in the mobileNetv1 is used to replace a standard convolution layer in the original UNet network; each deep separable convolution comprises two steps of deep convolution and point-wise convolution, the deep convolution is firstly performed on each input channel to extract features, and then the point-wise convolution is performed on the output of the deep convolution and a 1*1 convolution kernel to adjust the number of channels and extract deep features, wherein a BN layer is added to increase the generalization, and a ReLU6 activation layer is added to introduce more nonlinearity; the features obtained after the deep separable convolution are also transmitted to the right side network in a jump connection mode, the same size feature maps on both sides are cascaded and then subjected to the same up-sampling as in the UNet expansion network, and finally restored to the size of the original input picture 256*256.

2. The method of claim 1, wherein: The training sample input data is input into the data augmentation network to realize random enhancement of the input sample, comprising the following steps: The data augmentation network is built to realize normal enhancement of the input normal sample or to introduce a local irregular anomaly to construct a pseudo abnormal sample; The enhanced normal sample or the constructed pseudo abnormal sample and their corresponding labels and pixel perfect anomaly maps are output and used for subsequent supervision.

3. The method of claim 2, wherein: The data augmentation network is built to realize normal enhancement of input normal samples or introduction of local irregular abnormality to construct pseudo abnormal samples, and the specific steps include: setting two random values with values in the interval of 0-1; When , only the normal sample is randomly enhanced, including rotation, bias, sharpness, sunlight, equalization, brightness change, color change, automatic contrast change, etc., without abnormal processing, and the enhanced normal sample , whose label is set to 0, is output, and the corresponding output abnormal graph is set to a black picture with the same size as the input picture; When , the normal sample is abnormally processed to introduce local irregular anomalies, when , a pseudo-anomaly sample is generated based on cutpaste enhancement, when , a pseudo-anomaly sample is generated based on Berlin noise, and finally the enhanced pseudo-anomaly sample , whose label is set to 1, is output, and the output anomaly map M shows that the normal area is black and the local anomaly is white.

4. The method of claim 1, wherein: The reconstruction sub-network comprises an encoder and a decoder, and an SE channel attention module is added after the features extracted by the convolution layer in the encoder; The SE channel attention module is composed of a global pooling layer, a dimension reduction full connection layer, a ReLU activation layer, a dimension increase full connection layer and a sigmoid activation layer, and through three operations of Squeeze, Excitation and Scale, the input feature map is subjected to feature enhancement on each channel.

5. The method of claim 1, wherein: The total loss function in the training stage is the loss of the reconstruction subnetwork for the reconstruction of the non-abnormal features and the loss of the image segmentation subnetwork for accurate segmentation; reconstruction sub-network without abnormal feature reconstruction loss , set to the network reconstruction sample and data augmentation network input sample structure similarity loss and image pixel loss ; Image segmentation subnetwork accurate segmentation loss , set to the focal loss between the network output feature map and the label map.

6. The method of claim 1, wherein: The test sample and the output sample obtained by entering the reconstruction subnetwork are concatenated at the channel, and are sent into the image segmentation subnetwork to obtain a pixel-level abnormality positioning map, which specifically includes the following steps: The two layers of features output by the image segmentation subnetwork are normalized by a softmax activation function in the channel dimension, and then the abnormal layer is taken; After the values of the abnormal layer are converted into corresponding gray values, the abnormality positioning map corresponding to the test sample is obtained.

7. The method of claim 1, wherein: An instance-level abnormality score corresponding to the test sample is obtained from the abnormality positioning map, which specifically includes the following steps: A two-dimensional average pooling operation is performed on the abnormal layer with a stride of 1, and a smoothing operation is performed on the abnormality map by means of 0 padding without changing the output size, so as to aggregate the local abnormal response information; The global maximum value of the smoothed abnormality map is taken as the instance-level abnormality score corresponding to the test sample, and a threshold is set according to the scores of all test samples, and the samples greater than the threshold are classified as abnormal, and the samples less than the threshold are classified as normal.

Images