Metal surface defect semantic segmentation network and corresponding strategy

Abstract

The application discloses a metal surface defect semantic segmentation network and a corresponding strategy to solve technical problems such as serious imbalance in quantity and distribution, low contrast, and weak boundary information in metal surface defect semantic segmentation, and belongs to the field of computer vision. The application comprises a multi-scale attention feature fusion module, a normalized mean square frequency category weight strategy, a bias weight training sampling strategy, and a category and boundary loss calculation strategy. The above method successively solves the above technical problems by fusing feature information of different scales through double attention, adjusting the category weight coefficient factor, increasing the attention to defect samples, and fusing boundary loss. Practice has proved that the method can improve the detail capturing capability for metal surface defects and meet the urgent needs of industry for metal surface defect detection.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation, and in particular to an image semantic segmentation method and strategy for detecting defects on metal surfaces. Background Technology

[0002] Image semantic segmentation is a crucial research area in computer vision, aiming to classify input images into pixel-level categories based on semantic information. In industrial production, the dimensional information of defects on metal surfaces is vital for product yield, and pixel-level semantic segmentation of metal surfaces is a prerequisite for achieving dimensional measurement.

[0003] Unlike natural image data, metal surface defect detection faces challenges due to the scarcity of defect samples in industrial production. Furthermore, defects are often small in scale, have low contrast, and indistinct boundaries. These issues frequently lead to false positives or false negatives during training, resulting in misclassification of pixels in metal surface images.

[0004] Currently, due to the aforementioned technical challenges, networks including SegNet, Unet, FCN, and DeepLabv3+ cannot effectively achieve semantic segmentation of defects on metal surfaces. Summary of the Invention

[0005] To address the aforementioned technical challenges, this invention provides a semantic segmentation network for metal surface defects and a corresponding strategy, which can significantly improve the accuracy of semantic segmentation of metal surface defects.

[0006] In the publicly available technical methods, the semantic segmentation network for metal surface defects includes an HRNet feature extractor. After the original image undergoes bilinear interpolation, the HRNet feature extractor generates multi-scale feature maps, with scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 relative to the original input image. Subsequently, the four feature maps at different scales are divided into two branches.

[0007] In the first branch, each multi-scale feature map passes through an independent spatial attention module. Next, two adjacent multi-scale information are fused through independent channel attention and then passed through another independent spatial attention module to obtain multi-scale feature maps containing rich semantic and texture information. Finally, the multi-scale feature maps are interpolated to the 1 / 4 scale level of the original image and then stitched together.

[0008] In the second branch, the multi-scale feature maps are directly upsampled, linearly interpolated to 1 / 4 scale of the original image, and then stitched together.

[0009] The feature information obtained from the above two branches is concatenated to obtain the third branch.

[0010] Each of the three feature branches is processed through a convolutional block to compress the channels to the number of categories. Then, it is upsampled to the original image scale and the loss is calculated by comparing it with the ground truth label.

[0011] The publicly disclosed strategies include a normalized mean square frequency class weight strategy, a biased weight training sampling strategy, and a loss calculation strategy that combines class and boundary operations.

[0012] The normalized mean square frequency category weighting strategy first analyzes all data samples to determine the pixel proportion of each category and the number of images containing each category, and then calculates the weight coefficient factor for each category based on the above information.

[0013] The biased weighted training sampling strategy first records the defect categories contained in the images in the dataset, forms a list of image addresses containing defects of each category, and randomly samples from the category list according to the different categories during training.

[0014] The loss calculation strategy employs parallel category and boundary loss calculations. During training, the system obtains the original image and label information of a sample. Boundary contours are extracted from the label information to obtain boundary information. This boundary information is then filtered to obtain boundary weights. In the loss calculation, the network parameters are adjusted by combining category loss and boundary loss.

[0015] Compared with existing solutions, this solution has the following technical advantages:

[0016] A semantic segmentation network for metal surface defects significantly expands the receptive field and enhances the network's ability to understand input information by fusing a dual attention mechanism and a multi-scale fusion approach. Furthermore, the introduced strategies only occur during the training phase and participate in computation during the testing phase. Therefore, without increasing computational cost or the number of parameters, this significantly improves the network's segmentation ability and greatly reduces the proportion of false positives and false negatives. Overall, this method significantly improves the detection performance for metal surface defects. Attached Figure Description

[0017] Figure 1 A schematic diagram of the overall structure of the metal surface defect segmentation network framework;

[0018] Figure 2 This is a schematic diagram of the spatial attention module of a metal surface defect segmentation network, corresponding to... Figure 1 The SA part;

[0019] Figure 3 This is a schematic diagram of the channel attention module of a metal surface defect segmentation network, corresponding to... Figure 1 The CA part;

[0020] Figure 4 Flowchart of the biased weighted training sampling strategy;

[0021] Figure 5 The figure shows the comparative experimental results of the method of the present invention on the SD900 dataset;

[0022] Figure 6 The image shows the comparative experimental results of the method of this invention on the Crack dataset. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In the technical method disclosed in this invention, the input image is an RGB image. The feature extractor is an HRNet network, trained using its pre-trained weights on a human keypoint detection dataset as initial parameters. After the original RGB image passes through the feature extractor, the output of the network's stage 4 part is used as the multi-scale feature information of the original input image.

[0025] Spatial attention module such as Figure 2 As shown, the network consists of a 1×1 convolution, a batch normalization operator, and a sigmoid operator. The convolution module compresses the input information into a feature map with 1 channel and performs a spatial positional multiplication operation with the original input image. Finally, it uses residuals to connect the features, introducing initial structural information to ensure that the network properties do not degrade.

[0026] Channel attention module, such as Figure 3 As shown, the algorithm consists of an average pooling operator, a 1×1 convolution, a batch normalization operator, and a sigmoid operator, accepting feature map information from two adjacent scales as input. The low-resolution feature map is compressed into a feature map with dimensions of 1 after the average pooling operator, preserving channel information. After a 1×1 convolution, the channels are aligned with the high-resolution feature map. Following batch normalization and the sigmoid operator, a spatial dot product is performed with the high-resolution feature map. Finally, residual connections are used to incorporate the original information from the high-resolution image, outputting the high-resolution feature map.

[0027] In multi-supervised training, the loss weight coefficients for the two side branches and the main prediction branch are set to 0.25:0.25:0.5, thereby ensuring that the prediction branch is prioritized during network training.

[0028] In the normalized mean square frequency class weighting strategy, assuming there are N classes in the dataset (including background), the frequency information of the i-th class is defined as follows:

[0029] f i =Γ i Λ i

[0030] Where i∈[1,2,..,N], Γ i Λ represents the number of pixels in the dataset belonging to the i-th class. i This represents the total number of pixels in an image containing pixels of class i.

[0031] The category weight coefficient W of the i-th pixel is calculated. i As shown below:

[0032]

[0033] Because pixels from multiple categories coexist, and the weight coefficients of different categories influence each other, a normalized category weight w is used. i As shown below:

[0034]

[0035] Thus, the normalized weights of each category in the network were obtained.

[0036] Partial weighted training sampling strategy, such as Figure 4 As shown. After obtaining the training samples and the number of categories N, the training set is first analyzed. Each image is sequentially assessed to determine whether it contains defects of different categories. When a defect of a corresponding category exists, the image address is recorded in the corresponding defect category set (considering computer memory limitations). If no defect category exists, the image does not contain any defects, and its address is recorded in the defect-free image address set. After implementing the above process, image address sets for different defect categories and defect-free image address sets are obtained sequentially. During training, samples are taken from different category sets with equal probability, and images are randomly selected for training within each category.

[0037] In the category and boundary parallel loss calculation strategy, during training, the Sobel operator is first used to extract the boundary contours of the label image to obtain the image's boundary contour information. Then, a Gaussian filter is used to filter the boundary contour image, thus assigning higher weight information to the boundary contour regions. The boundary weight coefficients are shown below:

[0038]

[0039] Where ψ xLet ψx be the boundary weight value at pixel x. ψ0 is a constant that ensures information not located at the label position receives some attention; in the experiment, it is set to 0.05. Θ x The distance from pixel x to the boundary is described by σ, which is the standard deviation of the Gaussian filter and is taken as 5.0 in the experiment.

[0040] The loss is calculated using cross-entropy, and the overall loss function Ψ can be expressed as:

[0041]

[0042] Where x represents the feature map information predicted by the network, y represents the label information, C represents the number of categories, w represents the category weight information, Ω represents the set of all pixels in the image, and λ represents the scaling factor, which is set to 1.0 in the experiment.

[0043] The method of this invention is compared with multiple methods on two datasets (mIou) as shown in Table 1. Experiments show that this method can significantly improve the accuracy of semantic segmentation of metal surface defects.

[0044] Table 1

[0045]

[0046] Furthermore, the effect diagram of this invention is as follows: Figure 5 , Figure 6 As shown, it can be seen that the method of the present invention is closest to the true value of the label compared with other methods, and can improve the accuracy of semantic segmentation of metal surface defects.

Claims

1. A corresponding strategy for a semantic segmentation network of metal surface defects, characterized in that, include: The input image is a metal surface image; multiple spatial attention is used to extract multi-scale feature information, and the input features are sent to the channel attention module. The channel attention fuses the features of adjacent scale information, inputs high-level semantic information into the low-level texture feature map, and then passes the information through the spatial attention module to fuse feature information again. Using a multi-supervised approach, the feature information after passing through the module is sequentially aggregated with the original output information of the HRNet feature extractor, and the network is trained separately to prevent gradient vanishing. The aggregated information is then concatenated for final category determination. The normalized mean square frequency category weight strategy is used to circumvent the calculation method of weight coefficient factors in natural datasets. By adjusting the weight factors in the training process of defective pixel categories, it suppresses the false positive classification phenomenon caused by the network training process getting stuck in local optima due to excessively large weight coefficients. By combining the proportion of pixel categories and their distribution proportions, the corresponding weight coefficients are calculated using the quadratic radical formula, and the weight factors for each category are determined together with all categories. The bias weight training sampling strategy is used to avoid random sampling methods in natural datasets. By combining the ratio of defective samples to normal samples, the network pays more attention to defective samples by increasing the proportion of defective samples during training, thereby reducing the uncertainty caused by the small number of defective samples. Unlike random sampling training, this method first analyzes and studies all data in the training dataset, and then randomly samples within the category for training. The category and boundary loss calculation strategy differs from traditional loss calculation methods that only calculate category loss. It extracts boundary contours from the label image and applies greater attention weight to defect boundaries, forcing the network to focus more on boundary information during training. The Sobel operator is used to extract contours in the boundary region, followed by spatial filtering using a Gaussian filter, which is then incorporated into the loss function. In this parallel category and boundary loss calculation strategy, during training, the Sobel operator is first used to extract boundary contours from the label image, obtaining the image's boundary contour information. Then, a Gaussian filter is used to filter the boundary contour image, resulting in higher weight information in the boundary contour region. The boundary weight coefficients are shown below: in Let x be the boundary weight value at pixel x. The constant is 0.05, which ensures that information not located at the label position still receives a certain level of attention. Describes the distance from pixel x to the boundary. is the standard deviation of the Gaussian filter, which is taken as 5.0 in the experiment; The loss is calculated using cross-entropy, and the overall loss function is... Represented as: Where x represents the feature map information predicted by the network, y represents the label information, C represents the number of categories, and w represents the category weight information. It is the set of all pixels in the image. This is the scaling factor, which was set to 1.0 in the experiment; Unlike traditional methods that use category loss for calculation, this method uses category loss and boundary loss in parallel, forcing the network to deepen its learning and attention to boundary information.

2. The corresponding strategy of the semantic segmentation network for metal surface defects according to claim 1, characterized in that: For the final convolutional network, to save memory, 1 is used.

1. Convolution.

Images