Training method of data-driven metal surface defect semantic segmentation network

Abstract

The application discloses a metal surface defect semantic segmentation network and a training method based on data driving, and belongs to the field of image processing. 2 The application comprises a dual-attention multi-scale residual aggregation network based on U The above network and training method sequentially solve the above technical problems by aggregating multi-scale information of an image and applying attention, changing a weight factor of a category, topologically migrating a defect pixel region to a defect-free image, and applying double attention to a ground truth boundary and a predicted result boundary. Practice has proved that the method improves the precision of dense classification of a metal surface defect data set, and meets the urgent needs of the manufacturing industry for metal surface quality monitoring.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a semantic segmentation method for monitoring the quality of metal surfaces, which is urgently needed in the industrial field. Background Technology

[0002] In industrial manufacturing, quality monitoring of metal surfaces is crucial for ensuring product quality. Defects such as cracks on metal surfaces not only affect aesthetics but also cause economic losses. Visual inspection typically employs defect identification techniques such as image classification, object detection, and semantic segmentation. In comparison, semantic segmentation, which performs pixel-level classification of images, is more accurate in extracting defect areas than the former two methods. It can clearly analyze metal damage caused during processing and is the core and key to metal surface quality monitoring.

[0003] Semantic segmentation technology originates from natural images, but defects on metal surfaces in industry are generated during processing. These defects typically present challenges such as small defect areas, a small proportion of the sample, low contrast, and difficulty in accurately fitting boundary contour details. Therefore, existing methods often result in false positives or false negatives, affecting subsequent analysis of product quality.

[0004] To date, due to the aforementioned technical issues, existing methods (including UNet, DANet, DeepLabv3+, PSPNet, etc.) are unable to achieve accurate localization and analysis of defect regions on metal surfaces. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a data-driven semantic segmentation network and training method for metal surface defects, which can improve the accuracy of dense classification of metal surface defects.

[0006] Among the publicly available technical methods, the dual-attention multi-scale residual aggregation network employs U... 2 Net extracts key feature information from the input image, which includes six scales, with a scale difference of 1 / 2 between adjacent sizes. Subsequently, each feature map information passes through a dual attention module.

[0007] The dual attention module consists of spatial attention and channel attention. It first passes through channel attention and then through spatial attention. Channel attention spans two scales of information, extracting the higher-level features with stronger semantic information into attention coefficient vectors along the channel direction and mapping them to the lower-level features with stronger texture information. After passing through channel attention, the spatial attention module performs self-attention operations on the feature map information at different scales.

[0008] The dual-attention multi-scale residual aggregation network uses the Res module to extract feature information of the input image at different scales and concatenates it with the feature map information after passing through the dual-attention module.

[0009] During training, the dual-attention multi-scale residual aggregation network is supervised by balancing the scale through bilinear interpolation on feature maps of different scales.

[0010] The feature map information at different scales is linearly interpolated and then concatenated with the low-level features extracted by a Res module in the channel direction. After passing through a convolution module, the predicted category information is output.

[0011] The publicly available training methods include methods for calculating defect category weights, defect transfer topology methods, and loss calculation methods for double-boundary concern.

[0012] The defect category weight calculation method analyzes all samples in the dataset to determine the number and distribution ratio of pixels in different categories. Based on the above information, the weight coefficient factor under different categories is calculated using the key coefficient calculation formula.

[0013] The defect transfer topology method analyzes the samples in the dataset sequentially, recording the addresses of images containing defective pixels and those without. During training, based on the corresponding address information, defective regions in defective images are transferred to defect-free samples with a certain probability.

[0014] The loss calculation method of dual boundary attention applies attention to the boundary information of the ground truth and the boundary information of the predicted image by using edge extraction operators and Gaussian image filtering, forming a boundary attention information matrix of the input image scale.

[0015] Compared with existing solutions, this solution has the following technical advantages:

[0016] This invention expands the spatial receptive field through a dual-attention multi-scale residual aggregation network, learning richer semantic information from the input image from both color and semantic perspectives. The introduction of defect category weight calculation methods, defect transfer topology methods, and dual-boundary attention loss calculation methods enables the network model to better reflect pixel category classification information in boundary regions, and these methods only participate in the network training process without affecting the network's prediction speed. Overall, this method improves the accuracy of semantic segmentation and addresses the urgent need in industrial applications for monitoring the quality of metal surfaces. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall process of training the network in this invention.

[0018] Figure 2 For U-based 2A schematic diagram of the overall structure of Net's dual-attention multi-scale residual aggregation network;

[0019] Figure 3 For U-based 2 A schematic diagram of the spatial attention module of the Net dual-attention multi-scale residual aggregation network, corresponding to Figure 1 The SA part;

[0020] Figure 4 For U-based 2 A schematic diagram of the channel attention module of the Net dual-attention multi-scale residual aggregation network, corresponding to... Figure 1 The CA part;

[0021] Figure 5 Here is a flowchart of the defect migration topology method;

[0022] Figure 6 Flowchart of the loss calculation method for dual-boundary concern; Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] In the technical method disclosed in this invention, the overall operation process is as follows: Figure 1 As shown, the weight factors for each category in the dataset are calculated using a defect category weight calculation method; the dataset is analyzed based on a defect migration topology method; and the model parameters are updated by predicting using a network model and backpropagating gradient information using a double-boundary attention method.

[0025] In the technical method disclosed in this invention, a dual-attention multi-scale residual aggregation network is proposed, the network structure of which is as follows: Figure 2 As shown. The network uses U 2 The Net network acts as a feature extractor, extracting its sidelists portion to obtain multi-scale features from the input RGB image (scales 3, H, W). These multi-scale features are then processed through cascaded dual attention modules, concatenated with the low-level information extracted by the Res module, linearly interpolated, and finally transformed into a predicted feature image (scales N, H, W) via a convolution module, where N represents the number of pixel categories in the dataset.

[0026] Dual attention modules include spatial attention (such as...) Figure 3The input feature map (as shown) consists of convolution operators, activation functions, etc. In spatial attention, the input feature map is processed by convolution, batch normalization, and sigmoid activation to obtain a single-channel spatial attention coefficient matrix. This matrix reflects the network's understanding of the core information of key spatial directions in the input image and applies different attention weights. To avoid underfitting, the above attention coefficient matrix is ​​multiplied by the input feature map after convolution, and residual connections are used to maintain gradient propagation stability.

[0027] Dual attention modules include channel attention (such as...) Figure 4 (As shown) It consists of average pooling operators, convolution, etc. Channel attention differs from spatial attention; it connects across two adjacent scales. It extracts feature map information with high semantic features through average pooling operators. After convolution, batch normalization, and sigmoid, it forms a channel attention coefficient vector with the same scale as the high-texture feature information and the same channel size as the high-texture feature information. This vector is then multiplied by the convolved high-texture feature information to obtain feature map information with channel attention coefficients. Finally, to prevent instability during training, residual connections are used to obtain the original information of the input feature map.

[0028] The semantic information of defects on metal surfaces is weak. After numerous convolutional and pooling operations, the low-level information of the input image becomes increasingly weak. Therefore, a Res module is introduced. The Res module consists of convolution, batch normalization, and a ReLU operator, and uses a residual structure to ensure training stability. The information acquired by the Res module does not undergo U... 2 The Net feature extractor directly obtains information from the underlying input image, but because U 2 The feature maps output by the Net feature extractor have different scales, so they need to undergo linear interpolation before entering the Res module.

[0029] The network training process uses a multi-supervised approach to transmit gradient information. Feature maps of different scales are concatenated with feature information extracted by the Res module and linear interpolation is performed. A feature map with scale (N, H, W) is obtained through convolution operators. This feature map is then concatenated with the feature information extracted by the Res module to obtain a feature map with scale (6×N+3, H, W). Finally, the prediction result is output through convolution.

[0030] The defect category weight calculation method defines the weight information w for the i-th category if there are N categories in the dataset. i As shown below:

[0031]

[0032] Where, τ i κ represents the percentage of pixels belonging to the i-th class in the dataset. iThis represents the proportion of images containing the i-th class of defects. Since the dataset contains multiple classes, the weights of each class can be combined into a vector. This method calculates the class weight vector using the following formula.

[0033] W = median(w) / w

[0034] Where median is the median operator. The weight factors for each category of the network can be obtained using the above formula.

[0035] Defect migration topology methods such as Figure 5 As shown. First, the samples in the dataset are analyzed sequentially to determine if they contain defective regions, and their addresses are recorded according to the defect category. After implementing the above method, a list of N addresses under N categories (including the normal category) in the dataset can be obtained. During the training process, normal training is performed when the input sample contains defective regions; when the input sample does not contain defective regions, topology of the defective regions is performed with a probability of 0.5. During the topology process, a sample is randomly selected from the list of samples containing defects, and the defective region is extracted based on the ground truth. After vertical and horizontal flipping, the defective region is topologically integrated into the defect-free sample, thereby obtaining a new image containing the defective region and ground truth information.

[0036] The loss calculation method for dual-boundary concern is as follows Figure 6 As shown, contour extraction is performed on the boundary between the ground truth and predicted information. Next, a Gaussian convolution operator is used to filter the image, resulting in greater attention information in the boundary regions. Based on both the ground truth and predicted boundaries, attention is introduced simultaneously, and the loss is calculated separately for the boundary information. The weighting factor matrix introduced by the boundary is shown below.

[0037]

[0038] Where ψ x Θ represents the boundary weight information at pixel x. l Θ represents the distance of pixel x from the ground truth boundary. p σ represents the distance of pixel x from the prediction boundary, σ is the standard deviation of the Gaussian filter, which is taken as 5.0 in the experiment. ψ0 is a constant, taken as 0.1 in the experiment, to ensure that other areas also have a certain degree of attention.

[0039] In the experiment, if double boundary attention is used directly for training, the predicted boundary loss is relatively large in the early stages of training, which will affect the stability of training. To solve the above problem, double boundary attention is not used in the first half of the training process; only class weights are used for calculation, and the loss is shown below:

[0040]

[0041] Where sup is the number of multi-supervised branches, Ω represents the set of all pixels in the image, C is the number of classes in the dataset, x is the predicted information of pixels in the image, y is the ground truth information, and w is the class weight, which is calculated by the class weight calculation method.

[0042] In the later stages of training, a double-boundary weighting method is used to calculate the loss, as shown below.

[0043]

[0044] Wherein, λ is the scaling factor constant, which is set to 1.0 in the experiment.

[0045] The method of this invention is compared with multiple methods on two datasets (mIou) as shown in Table 1. Experiments show that the method of this invention can significantly improve the accuracy of semantic segmentation of metal surface defects and has a positive effect on metal surface quality monitoring.

[0046] Table 1

[0047]

Claims

1. A training method for a data-driven semantic segmentation network for metal surface defects, characterized in that, Dual-attention multi-scale residual aggregation network with U 2 Net is an autoencoder that extracts six multi-scale features from the input image. It applies spatial and channel-level attention to the multi-scale feature map information through a cascaded dual attention module, and uses the Res branch module to introduce key boundary information of the input image. It uses feature map stitching and multi-supervised training to backpropagate and update the network model parameters. The steps of the method include: The defect category weight calculation method combines the proportion of pixel occurrences with their distribution ratio in the dataset to adjust the category weight coefficient factors during the network model learning process. It also combines the proportion of defective pixels with the proportion of images containing defects, employing a key coefficient calculation formula to determine the weight factors for each category in the metal surface defect dataset during training. If the dataset has N categories, the method defines the weight information w for the i-th category. i As shown below: in, This represents the percentage of pixels belonging to the i-th category in the dataset. This represents the proportion of images containing the i-th category of defects. Since the dataset contains multiple categories, the weights of each category can be combined into a vector. This method calculates the category weight vector using the following formula: in median As the median operator, the weight factors for each category of the network can be obtained through the above formula; The defect transfer topology method extracts defective regions from defective images and topologically transfers them to defect-free images. The ground truth values ​​of the defective regions in the defective images are recorded as ground truth values ​​of the data-augmented images and output. The dataset samples are analyzed to determine whether each sample contains defective regions. During training, the defective regions of defective samples are mirrored and flipped horizontally and vertically according to a certain probability, and topologically transferred to defect-free samples to form new sample images. The ground truth values ​​of the defective regions are retained as ground truth values ​​and output. The dual-boundary attention loss calculation method simultaneously extracts boundary contours from both the ground truth image and the prediction image, and applies attention to the boundary contour regions using a Gaussian filter to form a boundary attention information matrix. Dual-boundary attention includes boundary attention for both the ground truth image and the prediction image, approximating the ground truth information from two directions. The process employs the Sobel operator for contour extraction and a Gaussian filter for filtering, thereby forming a boundary attention information matrix of the input image scale near the boundary regions. In the first half of training, a single class weight loss is used. After the network parameters stabilize, a parallel loss calculation method combining class loss and dual-boundary attention is adopted in the later stages of training. The weight factor matrix introduced by the boundary is shown below: in This represents the boundary weight information at pixel x. This represents the distance of pixel x from the ground truth boundary. This represents the distance of pixel x from the predicted boundary. is the standard deviation of the Gaussian filter, which is taken as 5.0 in the experiment. It is a constant, and in the experiment it is taken as 0.1 to ensure that other areas are also given some attention; In the experiment, if double boundary attention is used for direct training, the predicted boundary loss is relatively large in the early stages of training, which will affect the stability of training. To solve the above problem, double boundary attention is not used in the first half of the training process; only class weights are used for calculation. The loss is shown below: Where sup represents the number of supervised branches. C represents the set of all pixels in the image, C is the number of classes in the dataset, x is the predicted information of pixels in the image, y is the ground truth information, and w is the class weight, which is calculated by the class weight calculation method. In the later stages of training, a double-boundary weighting method is used to calculate the loss, as shown below: in, is the scaling factor constant, which is set to 1.0 in the experiment.

2. The data-driven semantic segmentation method for metal surface defects according to claim 1, characterized in that: The dual attention module includes channel attention and spatial attention. Channel attention extracts key feature information from the high-level feature map, forms attention coefficients in the channel dimension, and introduces them into the low-level scale. Spatial attention adopts a self-attention structure. Both dual attention modules adopt a residual connection structure.

3. The data-driven semantic segmentation method for metal surface defects according to claim 1, characterized in that: The Res branch module is used to embed the key texture information of the input image. The size of feature maps at different scales is balanced by bilinear interpolation. The Res branch module consists of convolution, normalization operator, and nonlinear activation function, and is connected by residual structure.

4. The data-driven semantic segmentation method for metal surface defects according to claim 1, characterized in that: Linear interpolation is performed on feature map information at different scales, and the network is trained using a multi-supervised approach. The loss function is calculated for information at different scales, and then the loss function is concatenated and introduced into the Res module to extract low-level information. Finally, the convolution operator is used to perform dense classification prediction of the input image.

Images