A steel surface abnormal defect detection method based on semi-supervised contrast learning

Abstract

The application discloses a steel surface abnormal defect detection method based on semi-supervised contrast learning, which uses a defect detection network architecture based on semi-supervised contrast learning of "abnormal reconstruction + contrast discrimination", obtains pseudo abnormal samples through abnormal simulation of normal samples, and then realizes the reconstruction and recovery of the pseudo abnormal samples by using an abnormal reconstruction network. For the abnormal samples and the recovered samples obtained after the reconstruction, the information of the two images is compared and learned by a subsequent contrast discrimination network to optimize the segmentation effect. Secondly, a mask hole convolution module is used in combination with a Transformer module to better optimize the performance of the abnormal reconstruction network. Meanwhile, a self-attention mechanism is added to the contrast discrimination network to improve the contrast learning ability of the network in space and channels. Finally, an efficient semi-supervised contrast learning method for steel surface abnormal defect detection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial defect detection, semantic segmentation, anomaly detection, etc., and in particular to a steel surface abnormal defect detection method based on semi-supervised contrast learning. BACKGROUND

[0002] In the field of industrial defect detection, the importance of semantic segmentation algorithms is self-evident. Industrial defect detection refers to the detection of various products in the industrial field, including large aerospace vehicles and small electronic components at the micro-nano level. Certain industrial detection is required before and after the production process to ensure safety and quality. Therefore, industrial defect detection is an important link and one of the important technologies for industrial product quality assurance and production safety and stability.

[0003] However, the industrial defect detection technology based on semantic segmentation faces many difficulties, which are closely related to the real environment in the industrial field. For example, the lack of defect sample resources, the unclear and incomplete definition of defect types, the poor visibility of defects, and the variable shape of defects. This has led to a series of technical problems in real-world scenarios, hindering the development of industrial defect detection. The specific problems are: 1. Difficulty in data labeling in real industrial scenarios, with large human labeling errors; 2. Limited data labeling, with a large dependence of algorithm models on data; 3. There are a large number of normal samples in real industrial scenarios, while the number of abnormal samples is too small, and the positive and negative samples are imbalanced. SUMMARY

[0004] The present application aims to overcome the shortcomings of the existing semantic segmentation method for industrial defect detection, and provides a steel surface abnormal defect detection method based on semi-supervised contrast learning.

[0005] The purpose of the present application is achieved by the following technical solution: a steel surface abnormal defect detection method based on semi-supervised contrast learning, comprising the following steps:

[0006] (1) Obtain steel surface data in real industrial scenarios, select machine tool area range pictures taken from different angles, and select data with surface defects as abnormal samples and data without surface defects as normal samples, wherein the surface defects include edge cracks and folds;

[0007] (2) For normal samples, simulate anomalies to generate simulated industrial real scene abnormal Berlin noise, combine with normal samples scaled by the same proportion, and obtain simulated abnormal sample image I c :

[0008]

[0009] I rdenotes the scaled compressed image of normal sample image I, A is the texture pattern of random simulated anomaly, P t is the binarization image of random Berlin noise image P, β denotes the proportion coefficient, is the fusion proportion between normal sample image and simulated anomaly defect, ⊙ denotes the pixel-by-pixel point multiplication operation between image matrices, denotes the inverse value image of anomaly binarization image P t .

[0010] (3) An anomaly reconstruction network is constructed based on an encoder-decoder structure, which is used to learn an anomaly reconstruction network for restoring simulated anomaly samples to normal samples, the simulated anomaly samples in step (2) are used as network inputs, and a reconstructed sample is obtained, and a loss is calculated with a normal sample to train the anomaly reconstruction network.

[0011] (4) A contrastive discriminative network based on semantic segmentation is constructed; the anomaly sample in step (1) is input into the trained anomaly reconstruction network to obtain a reconstructed sample, and the reconstructed sample is merged with the input anomaly sample in the channel to be used as the input of the contrastive discriminative network, and the difference between the reconstructed sample and the input anomaly sample is obtained through contrastive learning, and the steel surface defect detection result is output.

[0012] Further, a mask dilated convolution module is embedded in the encoder of the anomaly reconstruction network to expand the receptive field of the anomaly reconstruction network, and a Transformer is used to replace the full connection integration operation of the mask dilated convolution module to realize feature aggregation.

[0013] Further, for the contrastive discriminative network, a self-attention mechanism module is used to obtain self-attention in the channel and space of the input of the contrastive discriminative network, and the steel surface defect detection result is optimized.

[0014] Further, for the input feature map of the contrastive discriminative network, an attention extraction method of channel first and space second is used, and after each extraction of attention, the original feature map is multiplied and fed back.

[0015] Further, the texture pattern of the random simulated anomaly is used to strengthen the diversity of the simulated defect through random data enhancement, and three kinds of combinations of rotation, affine transformation, image brightness, sharpness, equalization value, contrast and saturation are randomly selected and used.

[0016] Further, the compressed image I r and the anomaly binarization image P t are overlapped to obtain a simulated anomaly part containing the original image information, and the compressed image I r and the inverse value image of the anomaly binarization image P t are overlapped to obtain an image area part without simulated anomaly.

[0017] The advantages of the present application are:

[0018] 1. Based on the abnormal simulation of Berlin noise and the equal scaling of industrial images, more realistic industrial abnormal defect simulation and image defect feature reservation are realized, and a basis for training of an abnormal reconstruction network is provided.

[0019] 2. Compared with the application of a basic semantic segmentation method in industrial defect detection, an abnormal reconstruction+contrast discrimination semi-supervised contrast learning structure is adopted, information value of normal samples can be better utilized, and the effect of reducing model data dependence is achieved.

[0020] 3. A self-supervised learning module using mask hole convolution is used to enhance the image restoration and reconstruction capability and provide a convenient condition for contrast learning.

[0021] 4. A CNN-oriented spatial and channel level self-attention mechanism is used to autonomously learn the information of the contrast image and improve the final segmentation and detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is a training phase flow chart of the abnormal reconstruction network of the normal sample of the present application;

[0024] Figure 2 is a training phase flow chart of the contrast discrimination network of the abnormal sample of the present application;

[0025] Figure 3 is an abnormal defect simulation flow chart of the present application. DETAILED DESCRIPTION

[0026] The specific embodiments of the present application will be further described in detail below in combination with the drawings. In order to more clearly illustrate the structural flow characteristics and effects of the present application, the method of the present application will be divided into three parts and described in detail as follows:

[0027] 1. Abnormal data simulation

[0028] (1) Obtain steel surface data in a real industrial scene, select machine tool area range pictures taken from different angles to form a data set, take data with surface defects as an abnormal sample data set, and take data without surface defects as a normal sample data set, wherein the surface defects include edge cracks and fold marks.

[0029] (2) Randomly generate a noise image P, the noise image is generated by Berlin Perlin noise, Perlin noise can well simulate various random defects, and can simulate abnormal defects of different shapes and sizes. Since the Perlin noise is not a binary image at the beginning, it cannot be directly used as a defect abnormal image, and needs to be binarized by random uniform sampling threshold to obtain the P Figure 3 image shown in t .

[0030] (3) For the simulation of abnormalities, not only the P t binarized image is used to realize the randomization of the simulated defect position, but also the texture information of the defect needs to be randomized, and the present application randomly selects different textures as the texture of the defect, and A in Figure 3 is the selected random abnormal defect texture pattern. Based on this texture, random data enhancement is needed to enhance the diversity of the simulated defects, including rotation, affine transformation, random enhancement of image brightness, sharpness, equalization value, contrast and saturation, and the like, and three kinds of use are randomly selected. The enhanced A and the abnormal binary image P t are overlapped to obtain a random texture and enhanced simulated abnormal defect part.

[0031] (4) For the original image I, since the images in the data set are of a uniform resolution size of 1440x2560, considering that most of the defects are too small, if the image is forcibly compressed to 256x256 resolution to match the input resolution of the network, the information of the small defects will be greatly lost. Therefore, considering this problem, the present application selects to first perform 2560x2560 square image superposition on the original image, and then perform 256x256 resolution compression on the whole. The shape information of the small defects is preserved to a certain extent, which is helpful for subsequent detection, and finally the compressed image I Figure 3 shown in r is obtained. r I t and the abnormal binary image P r are overlapped to obtain a simulated abnormal part containing the original image information, and I t and the inverse value image of the abnormal binary image P c are overlapped to obtain a part of the image region without simulated abnormalities.

[0032] Finally, as shown in Figure 3 , the three parts are mixed to obtain the simulated abnormal image I c :

[0033]

[0034] Wherein, beta represents a proportional coefficient, is the proportional fusion between the original image and the simulated abnormal defect, so that a random noise abnormality is designed, and the information of the original image itself is combined, so that the information of the final simulated defect abnormality is closer to the pixel distribution characteristics of the original image. The image matrix represents the point-by-point point multiplication operation between the image matrix, The inverse value image of the abnormal binary image P t .

[0035] Since the actual range of the original image is considered, the area is cut for the simulated abnormal image I c and the simulated abnormal P t , and the final result I A and P A .

[0036] 2, construct an abnormal reconstruction network and a contrastive discrimination network

[0037] (1) A semi-supervised contrastive learning architecture of "abnormal reconstruction + contrastive discrimination" is constructed. For this network structure, the invention designs a two-stage semi-supervised learning training mode, that is, first training normal samples and reconstruction network, then training abnormal samples and discrimination network. The reconstruction network can be a specific GAN generator or a basic FCN network encoder-decoder structure model, and the basic feature down-sampling and up-sampling convolutional network is realized. The discrimination network can be a basic semantic segmentation network such as UNet, SegNet, DeepLab, etc. In this way, the information value of the previously unavailable normal samples can be fully utilized, and the reconstruction of the abnormal samples plays a key role.

[0038] (2) The first stage, as shown in Figure 1 , only the simulated abnormal sample of the normal sample is input for learning, the abnormal reconstruction network is learned, the "de-abnormalization" sample output is generated, the reconstructed recovery sample is obtained, and then the actual normal sample is calculated. Loss (Loss1 in Figure 1 ) to train the abnormal reconstruction network to obtain a stable abnormal reconstruction network. It should be noted that there is no annotation for normal samples, so the subsequent contrastive discrimination network can be selectively trained. The training loss (Loss2 in Figure 1 ) of the discrimination network is mainly reflected in the second stage. In addition, the simulated abnormal sample generated in the normal sample training stage is only a simulated defect abnormality, and cannot simulate the segmentation of the machine tool area positioning, so it will not involve the training and learning of the machine tool area positioning.

[0039] (3) The second stage, as shown in Figure 2As shown, only the abnormal sample is input into the pre-trained abnormal reconstruction network to generate a "de-abnormalized" industrial real image sample, which is then merged with the input abnormal sample in the channel and input into the comparative discrimination network to learn the output defect detection result and machine tool area positioning result. This stage will not make any training adjustment to the abnormal reconstruction network which has been trained in the first stage, and directly used for the restoration work of abnormal samples, providing good conditions for comparative learning.

[0040] 3. Optimization design of abnormal reconstruction network and comparative discrimination network

[0041] (1) For the abnormal reconstruction network, a mask hole convolution module (Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection) is used to expand the receptive field of the reconstruction network, realize the information aggregation ability around the pixel, and better predict the restoration result of the current pixel. At the same time, the Transformer is used instead of the basic fully connected integration operation in the feature fusion, which better realizes the aggregation of features.

[0042] The mask hole convolution needs to determine two parameters, one is the hole rate d, and the other is the convolution size k used. Thus, the convolution used is k x k x c, where c is the channel number of the convolution kernel.

[0043] For an h x w x c image, it first needs to be padded and expanded. The image is expanded by a distance of k+d around the four corners, thereby obtaining an expanded image of (h+2k+2d) x (w+2k+2d) x c.

[0044] For the expanded image, it is divided into four parts, namely the upper left corner, the lower left corner, the upper right corner, and the lower right corner of the four sub-image blocks, each of which is (h+k+2d) x (w+k+2d) x c in size. Thus, the four sub-image blocks are respectively convolved and fused and activated, which is equivalent to performing self-supervised learning on the four perspectives around each pixel of the original image to predict the actual value of the pixel, thereby realizing self-supervised learning.

[0045] For the feature map output by the mask hole convolution which represents the h x w size feature map obtained by c mask hole convolutions. First, the spatial average pooling is used to reduce the feature map to obtain the pooled feature map where h'≤h and w'≤w. Next, is reshaped according to the channel to obtain the flattened feature where n = h' x w', A represents the flattening features of different channels. Then, Tokens are extracted from A by a linear layer where d t represents the dimension of Tokens for each channel. Tokens will be added with position encoding to obtain the final Tokens Finally, Tokens are trained using a multi-head attention mechanism to obtain a feature map with self-attention, which optimizes the image restoration capability of the anomaly reconstruction network.

[0046] (2) Self-attention mechanism module of the contrastive discriminative network

[0047] For the two samples with subtle differences in abnormal defects as input of the contrastive discriminative network, the scheme of first channel attention module and then spatial attention module is adopted:

[0048] For the channel attention module, the method of compressing the feature map space is adopted, which is different from the commonly used method of only average pooling to obtain attention. CBAM uses both average pooling and maximum pooling to effectively calculate the spatial information statistics, collects the attention features of different channels, and improves the network representation ability.

[0049] For the spatial attention module, both average pooling and maximum pooling are used to compress the channel dimension and effectively calculate the information statistics on the channel, and finally obtain the attention features of different spaces.

[0050] Finally, combining the two attention modules, the channel and then the spatial attention extraction method is adopted for the feature map, and after each attention extraction, it is multiplied with the original feature map for feedback. Thus, the final attention mechanism module is obtained, which improves the final effect of industrial defect detection.

[0051] The steel surface defect dataset used in this embodiment contains 2387 training set pictures and 175 test set pictures. Among them, there are 338 normal samples and 2049 abnormal samples in the training set. For each type of sample, there are four different perspective images, and the normal samples are not labeled. In the test set, for the four different perspectives, a certain number of pictures are randomly selected as test data for each perspective image, and finally 175 test set images are obtained, including 26 normal samples and 149 abnormal samples for testing.

[0052] The abnormal samples of the training set and all samples of the test set are provided with two kinds of data labeling, which represent the data labeling for defect detection, including the defect labeling of creases and edge cracks; and the labeling for machine tool positioning, mainly using pixel-level labeling of machine tool range.

[0053] The final experimental results prove that for defect detection, the basic semantic segmentation method UNet is 74.5% in MIoU, compared with 86.3% in MIoU standard of the method designed by the application, and the method designed by the application still reaches 77.0% in MIoU standard in the case of using 50% of the abnormal sample data amount, proving the superiority of the method.

[0054] The above examples are used to explain and illustrate the application, but not to limit the application, and any modification and change made to the application within the spirit and protection scope of the claims of the application, falls into the protection scope of the application.

Claims

1. A method for detecting abnormal defects on the surface of steel based on semi-supervised contrastive learning, characterized in that, Includes the following steps: (1) Obtain steel surface data in real industrial scenarios, select machine tool area images taken from different angles, take data with surface defects as abnormal samples, and take data without surface defects as normal samples. The surface defects include edge cracks and creases. (2) For simulations of anomalies, it is not only necessary to use P t Binarizing the image to randomize the location of the simulated defect also requires randomizing the texture information of the defect, randomly selecting different textures as the texture of the defect. For normal samples, anomalies are simulated to generate anomalous Burmester noise that simulates a real industrial scenario. This noise is then combined with the proportionally scaled normal samples to obtain the simulated anomalous sample image I. c : Among them, I r This represents a compressed image that is proportionally scaled from a normal sample image I. The original image is first overlaid with square images, and then the overall resolution is compressed. A represents a randomly simulated anomalous texture pattern, and P... t This is the binarized image of the random Berlin noise image P, where β represents the scaling factor, which is the fusion ratio between the normal sample image and the simulated abnormal defect, and ⊙ represents the pixel-wise dot product operation between the image matrices. Represents the anomalous binarized image P t The inverse value image; compressed image I r With the anomalous binarized image P t Phase overlay yields the simulated anomaly portion containing information from the original image, while simultaneously compressing image I. r With the anomalous binarized image P t The reverse value image is overlaid to obtain the image region without simulated anomalies; (3) An anomaly reconstruction network is constructed based on the encoder-decoder structure to learn how to reconstruct anomaly samples into normal samples. The simulated anomaly samples in step (2) are used as network input to obtain reconstructed samples. The loss is calculated with normal samples to train the anomaly reconstruction network. A mask-hole convolution module is embedded in the encoder of the anomaly reconstruction network to expand the receptive field of the anomaly reconstruction network. The fully connected integration operation of the mask-hole convolution module is replaced by Transformer to realize feature aggregation. (4) Construct a contrastive discriminant network based on semantic segmentation; input the abnormal samples from step (1) into the trained abnormal reconstruction network to obtain reconstructed and restored samples, merge the channels with the input abnormal samples, and use them as input to the contrastive discriminant network. The difference between the reconstructed and restored samples and the input abnormal samples is obtained through contrastive learning. For the contrastive discriminant network, a self-attention mechanism module is used to obtain the channel and spatial self-attention of the input of the contrastive discriminant network. For the input feature map of the contrastive discriminant network, the channel-first and spatial attention extraction method is adopted. After each attention extraction, it is multiplied with the original feature map for feedback, and the steel surface defect detection results are optimized and output.

2. The method for detecting abnormal defects on the surface of steel based on semi-supervised contrastive learning according to claim 1, characterized in that, Randomly simulated abnormal texture patterns enhance the diversity of simulated defects through randomized data augmentation, randomly selecting three combinations from rotation, affine transformation, image brightness, sharpness, equalization, contrast, and saturation.

Images