A surface defect image segmentation method based on weight fusion and double encoders

Abstract

The application discloses a surface defect image segmentation method based on weight fusion and double encoders, and comprises the following steps: constructing a surface defect image semantic segmentation network, including a feature extraction module, a feature fusion module and a double encoding module; inputting an image set into the feature extraction module to obtain feature maps of different levels; inputting the feature maps of different levels into the feature fusion module to obtain enhanced feature maps of different levels; inputting the enhanced feature maps of different levels into the double encoding module to obtain segmentation maps of different levels; calculating a weighted cross-entropy loss and a weighted intersection-over-union loss, and performing weighted summation to obtain a total loss; and using a total loss function constraint to complete training of the surface defect image semantic segmentation network and perform a surface defect image segmentation task; and the application effectively improves the receptive field of each layer of features, and can greatly improve the problem of different scale defect prediction difficulty.

Description

Technical Field

[0001] This invention relates to the field of image segmentation, and more particularly to a method for segmenting surface defect images based on weight fusion and dual encoders. Background Technology

[0002] In the manufacturing process, surface defect detection is one of the key links to ensure product qualification. Surface defects not only damage the surface quality of the product, but may also damage the internal structure of the product, thus leading to serious production safety problems. Therefore, in order to ensure the reliability of the product, it is necessary to conduct product surface defect detection [1]. Surface defect detection refers to the detection of scratches, defects, foreign objects obstructing, color pollution, holes and other defects on the surface of the sample to be tested, so as to obtain a series of relevant information such as the type, outline, location and size of the surface defects of the sample to be tested. Unfortunately, the current quality inspection still relies heavily on visual inspection and manual judgment by workers. Manual inspection methods are prone to missed detection and false detection. While reducing product quality, it also consumes a lot of labor costs.

[0003] Machine vision methods are mainly divided into image preprocessing, feature construction, and feature classification. These methods heavily rely on expert experience and knowledge during feature construction, and involve numerous hyperparameters during model training, requiring significant time for parameter tuning. Furthermore, the features extracted by these methods (such as color, texture, and contrast) struggle to capture high-level semantic relationships and contextual information, making them unsuitable for complex scenarios. Currently, deep learning algorithms are widely used in surface defect detection, such as FCN, UNET, and deeplabv3+. These networks use convolutional neural networks (CNNs) to extract features. They can not only extract high-level semantic information and context but also combine semantic information at different levels, greatly enhancing the model's applicability in complex scenarios.

[0004] With the development of deep learning, many deep learning-based surface defect detection algorithms have emerged. In addition to these problems, surface defects themselves are also very complex. First, there are large differences between classes, that is, the morphological features of defects of different classes may be very different. Second, there is a large degree of ambiguity between categories, that is, the appearance features of defects of different categories have certain similarities, making it difficult to distinguish the type of defects, and therefore it is impossible to accurately determine the cause of defects. Third, the background is complex, that is, there are many interfering factors in the production scene, such as environment, lighting, production process, noise, etc., so it is difficult to completely separate defects from the background, and the defect features are unknown. Fortunately, with the development of deep learning, many deep learning-based surface defect detection algorithms have emerged. One of the main reasons for the above problems is that deepening the network layer and upsampling will lead to the loss of information. As we all know, high-dimensional feature maps carry more semantic information, but ignore many details. Low-dimensional feature maps have a lot of detailed information, but lack semantic information. Therefore, most methods use feature fusion to alleviate the above problems. However, these methods do not fully mine multi-scale feature information in feature fusion and ignore the information redundancy problem in the feature fusion process. Therefore, how to provide a feature fusion method based on weight fusion is an urgent problem to be solved in this field. Summary of the Invention

[0005] To address the issue that existing feature fusion methods fail to fully exploit multi-scale feature information and neglect information redundancy during the feature fusion process, this invention provides a surface defect image segmentation method based on weighted fusion and dual encoders. The method includes the following steps:

[0006] S1. Construct a semantic segmentation network for surface defect images; the semantic segmentation network for surface defect images includes: a feature extraction module, a feature fusion module, and a dual encoding module;

[0007] S2. Input the surface defect image set into the feature extraction module to obtain feature maps at different levels;

[0008] S3. Input the feature maps of different levels into the feature fusion module to obtain the enhanced feature maps of each level;

[0009] S4. Input the enhanced feature maps of each level into the dual encoding module to obtain segmentation maps of different levels;

[0010] S5. Calculate the weighted cross-entropy loss and weighted cross-union ratio loss for the labels in the different level segmentation maps and surface defect image sets, and sum the weighted cross-entropy loss and weighted cross-union ratio loss to obtain the total loss.

[0011] S6. Using the total loss function constraint, train the semantic segmentation network for surface defect images to obtain the trained network;

[0012] S7. Use the trained network to complete the surface defect image segmentation task.

[0013] The beneficial effects provided by this invention are as follows: A novel feature fusion method—weighted feature fusion—is proposed. Based on neighbor-to-neighbor fusion, a GATE module is added to assign a dynamic weight to each feature map to be fused, avoiding feature redundancy during the fusion process. Furthermore, the CFM module in the dual encoder can effectively suppress invalid information and enhance valid information. Finally, a structural loss function is employed, using each feature layer as a monitoring signal to optimize the entire network. This method effectively increases the receptive field of each feature layer, greatly improving the difficulty of predicting defects at different scales and making a significant contribution to the field of surface defect segmentation. Attached Figure Description

[0014] Figure 1 This is a flowchart of the method of the present invention;

[0015] Figure 2 This is a diagram of the semantic segmentation network structure for surface defect images in this invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0017] Please refer to Figure 1 , Figure 1 This is a flowchart of the method of the present invention. The present invention provides a surface defect image segmentation method based on weight fusion and dual encoder, comprising the following steps:

[0018] S1. Construct a semantic segmentation network for surface defect images; the semantic segmentation network for surface defect images includes: a feature extraction module, a feature fusion module, and a dual encoding module;

[0019] S2. Input the surface defect image set into the feature extraction module to obtain feature maps at different levels;

[0020] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the semantic segmentation network structure for surface defect images according to the present invention;

[0021] It should be noted that the feature extraction module adopts a ResNet50 network structure; see Figure 2 Blocks 1-5; the feature maps of different levels, from top to bottom, are feature map F1 output from the first layer, feature map F2 output from the second layer, feature map F3 output from the third layer, feature map F4 output from the fourth layer, and high-level semantic feature map F5.

[0022] S3. Input the feature maps of different levels into the feature fusion module to obtain the enhanced feature maps of each level;

[0023] Still referencing Figure 2 It should be noted that the feature fusion module in step S3 uses weighted fusion to enhance feature maps at different levels. The specific process is as follows:

[0024] S31. First, perform feature fusion on F2, and input F1, F2, and F3 into the Gate1 module to obtain a new enhanced feature map New_F2;

[0025] S32. Input F2, F3, and F4 into the Gate2 module to obtain a new enhanced feature map New_F3;

[0026] S33. Input F3, F4, and F5 into the Gate3 module to obtain a new enhanced feature map New_F4.

[0027] By performing feature fusion enhancement in the above manner, it is ensured that the feature maps of the three middle layers are all enhanced once, reducing the semantic gap between feature fusion at different levels, and allowing each layer to include both low-level and high-level semantic information.

[0028] It should be noted that the Gate module is essentially a convolution function, and as the model is updated, the weights are continuously adjusted for feature fusion.

[0029] The following section uses the enhanced feature map F2 as an example to explain the working principle of the Gate module.

[0030] First, feature maps F1, F2, and F3 are scaled and have their channel count adjusted. F1 is subjected to a 3x3 convolution with a stride of 2, followed by a BN layer and a ReLU activation layer. F3 is upsampled by a factor of 2, followed by a 1x1 convolution. This ensures that the channel count and size of F1 and F3 are consistent with F2. These three feature maps are then concatenated along their channel dimensions to obtain a concatenated feature map. This concatenated feature map is input into the Gate1 module to obtain three dynamic weights ω1, ω2, and ω3. These weights are multiplied by their corresponding feature maps, and the results are summed to obtain the enhanced feature map New_F2. Taking F2 fusion as an example, the calculation formula is as follows:

[0031] ω1,ω2,ω3=Gate1(F1,F2,F3) (1)

[0032] New_F2=ω1×F1+ω2×F2+ω3×F3 (2)

[0033] Attention mechanisms can focus on salient information in feature maps, which can help improve the accuracy of the model. Here, the attention mechanism used is the CBAM attention mechanism. It can aggregate information from channel AND and spatial AND. New_F2, New_F3, and New_F4 are the results after feature extraction through one attention step.

[0034] For the process of other feature fusion enhancement, refer to the process of F2 above, and it will not be repeated here.

[0035] S4. Input the enhanced feature maps of each level into the dual encoding module to obtain segmentation maps of different levels;

[0036] It should be noted that the enhanced feature maps of each level and the high-level semantic feature map are input into the dual encoder, and finally two segmentation maps Predict1 and Predict2 and four segmentation maps out1, out2, out3 and out4 corresponding to different levels are obtained.

[0037] The process is briefly explained below:

[0038] The high-level semantic feature map F5 is bilinearly upsampled twice to obtain F′5, and then the sampled feature map is input into the CFM module together with New_F4.

[0039] The CFM module will perform feature fusion between F5 and New_F4. The output will be the enhanced feature NEW_F. 4H NEW_F 4L NEW_F 4H Next-stage fusion for dual encoders. NEW_F 4L This is used for the next upsampling. After three CFM fusions, a final prediction result, predict1, is output. To prevent information loss during the above process, the last feature map, NEW_F, is used. 2L Compensation is applied to the feature maps at other levels. That is, NEW_F 2L With NEW_F 2H The fusion yields NEW_F2'. 2L With NEW_F 3H The fusion yields NEW_F3', NEW_F 2L With NEW_F 4H The fusion yields NEW_F4', NEW_F 2L F"5 is obtained by fusing with F′5, resulting in the feature map. The CFM fusion process described above is as follows:

[0040] f L =f L +M l (Gl (f L )*G h (f H (3)

[0041] f H =f H +M h (G l (f L )*G h (f H (4)

[0042] In the formula, each M h (·), M l (·), G h (·), G l (·) represents the combination of the convolutional batch norm and ReLU. After obtaining the refined features, a 3x3 convolution is used to restore the original dimension. The "*" and "+" symbols represent the multiplication and addition of two matrices, respectively.

[0043] After the above steps, a second encoding process will be performed. This process is consistent with the first stage encoding process. Finally, a predicted segmentation map predict2 and four predicted segmentation maps at different levels out1, out2, out3, and out4 will be output.

[0044] In this application, step S4, the detailed data flow, and the corresponding fusion process are also described in detail below, still referring to... Figure 2 Step S4 includes two encoding processes, specifically:

[0045] First coding:

[0046] S41. Perform bilinear upsampling on the high-level semantic feature map F5 by a factor of two to obtain the sampled feature map F5';

[0047] S42. Input the sampled feature map F5' and the enhanced feature map New_F4 together into the first CFM module of the dual coding module for enhancement, to obtain the enhanced feature New_F 4H New_F 4L ;

[0048] S43, Enhance feature New_F 4L The enhanced feature map New_F3 is input into the second CFM module for enhancement, resulting in the enhanced feature New_F. 3H New_F 3L ;

[0049] S44, Enhance feature New_F 3LThe enhanced feature map New_F2 is input into the third CFM module for enhancement, resulting in the enhanced feature New_F. 2H New_F 2L ; and New_F 2L As a prediction output result, predict1;

[0050] Second encoding:

[0051] S45. Perform bilinear upsampling on the sampled feature map F5' twice to obtain the secondary sampled feature map F5''.

[0052] S46, Enhance feature New_F 2H New_F 2L By performing fusion feedback compensation, the enhanced feature New_F is obtained. 2` ; will enhance feature New_F 2L New_F 3H By performing fusion feedback compensation, the enhanced feature New_F is obtained. 3` ; will enhance feature New_F 2L New_F 4H By performing fusion feedback compensation, the enhanced feature New_F is obtained. 4` ;

[0053] S47. Combine the secondary sampling feature map F5`` with the enhanced feature New_F 4` Together, they are input into the fourth CFM module of the dual-encoding module for enhancement, resulting in the enhanced feature New_F. 4H` New_F 4L` ;

[0054] S48, Enhance feature New_F 4L` Compared with the enhanced feature map New_F 3` The input is fed into the fifth CFM module for enhancement, resulting in the enhanced feature New_F. 3H` New_F 3L` ;

[0055] S49, Enhance feature New_F 3L` Compared with the enhanced feature map New_F 2` The input is fed into the fifth CFM module for enhancement, resulting in the enhanced feature New_F. 2H` New_F 2L` ;where New_F 2L` As another prediction output result, predict2;

[0056] S410, Enhance feature New_F 2H` New_F 2L`Perform fusion feedback compensation to obtain the first output result out1; then enhance the feature New_F 2L` New_F 3H` Perform fusion feedback compensation to obtain the second output result out2; then enhance the feature New_F 2L` New_F 4H` Perform fusion feedback compensation to obtain the third output result out3; enhance the feature New_F 2L` The secondary sampled feature map F5`` is fused and compensated to obtain the fourth output result out4; the different level segmentation map is composed of the predicted output result predict1, the predicted output result predict2, the first output result out1, the second output result out2, the third output result out3, and the fourth output result out4.

[0057] S5. Calculate the weighted cross-entropy loss and weighted cross-union ratio loss for the labels in the different level segmentation maps and surface defect image sets, and sum the weighted cross-entropy loss and weighted cross-union ratio loss to obtain the total loss.

[0058] Four different levels of predicted segmentation maps are used, and loss functions are calculated for each of these two predicted maps and the labels. A single loss function consists of the weighted cross-entropy loss and its weighted intersection-union (IoU) loss. The formula for a single loss is as follows:

[0059] L = L ωbce +L ωiou (5)

[0060] The overall loss function is L total , where L i L represents the loss calculated from the two predicted graph segmentation plots. j The loss is calculated for the prediction graphs representing four different levels.

[0061]

[0062] S6. Using the total loss function constraint, train the semantic segmentation network for surface defect images to obtain the trained network;

[0063] S7. Use the trained network to complete the surface defect image segmentation task.

[0064] To better explain the content of this invention, three specific implementation examples using this invention are provided below. These examples were tested on the publicly available NEU, DAGM, and MT datasets. The proposed WFD-NET is implemented in PyTorch and runs on an RTX 2080Ti GPU.

[0065] The NEU dataset, released by Northeastern University (NEU), is a surface defect database that collects six typical surface defects of hot-rolled steel strip. For the segmentation task, the dataset provides pixel-level labels for three types of defects: inclusions (In), patches (Pa), and scratches (Sc), totaling 900 images with 900 pixel-level labels. This experiment uses 90-degree rotation, horizontal flipping, vertical flipping, and color jitter to augment the dataset, resulting in a total of 4500 images, used in a 4:1 ratio for training and validation.

[0066] The DAGM dataset is a dataset of surface defect images with various textures. The data is artificially generated but resembles real-world problems. The first six of the ten datasets (referred to as the development datasets) should be used for algorithm development. The remaining four datasets (referred to as the competition datasets) can be used to evaluate performance. They have a resolution of 512×512. For ground truth, the defect region is roughly covered by an ellipse. There are 900 images available for segmentation in this dataset. We use a 4:1 ratio for training and validation.

[0067] The Magnetic-Tile Dataset (MT) was collected by Huang et al.

[22] for the saliency detection of surface defects. It includes six types of defects: wear, inhomogeneity, cracks, fractures, pores, and freeness. 392 images have pixel-level labels. To prevent overfitting, we used 90-degree rotation, horizontal flipping, vertical flipping, and color jitter to augment the dataset. Finally, we obtained 1960 images. After removing some poor data, we retained 1887 images for training and validation. The training-validation ratio was 4:1.

[0068] Table 1 shows the results of WFD-NET on the three datasets.

[0069] This method achieves relatively high detection performance. Other methods in the literature, such as (Pyramid sceneparsing network, PSPnet), (DeepLab: Semantic image segmentation with deep convolutional nets, Atrous convolution, and fully connected CRFs, DeepLab), (RefifineNet: Multi-path refifinement networks for high-resolution semanticsegmentation, RefineNet), (Fully convolutional networks for semanticsegmentation, FCN), (Pyramid attention network for semantic segmentation, PAN), (U-Net: Convolutional networks for biomedical image segmentation, Unet), (Apixel-level segmentation convolutional neural network based on deep feature fusion for surface defect detection, PS-CNN), and (Pyramid feature fusion and global context attention network for automated surface defect detection, PGANet), all achieve lower average cross-union ratios than WFD-NET. This demonstrates that this method performs well on surface defect datasets.

[0070] In summary, the beneficial effects of this invention are as follows: It proposes a novel feature fusion method—weighted feature fusion—which, based on neighbor-to-neighbor fusion, adds a GATE module to assign a dynamic weight to each feature map to be fused, thus avoiding feature redundancy during the fusion process. Furthermore, the CFM module in the dual encoder can effectively suppress invalid information and enhance valid information. Finally, a structural loss function is employed, using each feature layer as a monitoring signal to optimize the entire network. This method effectively increases the receptive field of each feature layer, greatly improving the difficulty of predicting defects at different scales and making a significant contribution to the field of surface defect segmentation.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A surface defect image segmentation method based on weight fusion and dual encoder, characterized in that: Includes the following steps: S1. Construct a semantic segmentation network for surface defect images; the semantic segmentation network for surface defect images includes: a feature extraction module, a feature fusion module, and a dual encoding module; S2. Input the surface defect image set into the feature extraction module to obtain feature maps at different levels; the feature extraction module adopts the ResNet50 network structure; the feature maps at different levels, from top to bottom, are feature map F1 output by the first layer, feature map F2 output by the second layer, feature map F3 output by the third layer, feature map F4 output by the fourth layer, and high-level semantic feature map F5. S3. Enhance the feature maps at different levels using weighted fusion to obtain enhanced feature maps at each level; the specific process is as follows: S31. First, feature fusion is performed on F2. F1, F2, and F3 are input into the Gate1 module and the CBAM attention mechanism, and then a new enhanced feature map New_F2 is obtained. S32. Input F2, F3, and F4 into the Gate2 module and the CBAM attention mechanism to obtain a new enhanced feature map New_F3; S33. Input F3, F4, and F5 into the Gate3 module and the CBAM attention mechanism to obtain a new enhanced feature map New_F4; The Gate1, Gate2, and Gate3 modules are all convolutional functions that are updated during the training of the surface defect image semantic segmentation network, and their weights are continuously adjusted for feature fusion. S4. Input the enhanced feature maps of each level and the high-level semantic feature map F5 into the dual encoding module to obtain segmentation maps of different levels; S5. Calculate the weighted cross-entropy loss and weighted cross-union ratio loss for the labels in the different level segmentation maps and surface defect image sets, and sum the weighted cross-entropy loss and weighted cross-union ratio loss to obtain the total loss. S6. Using the total loss function constraint, train the semantic segmentation network for surface defect images to obtain the trained network; S7. Use the trained network to complete the surface defect image segmentation task.

2. The surface defect image segmentation method based on weight fusion and dual encoder as described in claim 1, characterized in that: Step S4 includes two encoding processes, specifically: First coding: S41. The high-level semantic feature map F5 is upsampled twice by bilinearly to obtain the sampled feature map F5ˋ. S42. Input the sampled feature map F5ˋ and the enhanced feature map New_F4 together into the first CFM module of the dual encoding module for enhancement, to obtain the enhanced feature New_F. 4H New_F 4L ; S43, Enhance feature New_F 4L The enhanced feature map New_F3 is input into the second CFM module for enhancement, resulting in the enhanced feature New_F. 3H New_F 3L ; S44, Enhance feature New_F 3L The enhanced feature map New_F2 is input into the third CFM module for enhancement, resulting in the enhanced feature New_F. 2H New_F 2L ; and New_F 2L As a prediction output result, predict1; The second coding: S45. Perform bilinear upsampling on the sampled feature map F5ˋ again to obtain the secondary sampled feature map F5ˋˋ. S46, Enhance feature New_F 2H New_F 2L By performing fusion feedback compensation, the enhanced feature New_F is obtained. 2ˋ ; will enhance feature New_F 2L New_F 3H By performing fusion feedback compensation, the enhanced feature New_F is obtained. 3ˋ ; will enhance feature New_F 2L New_F 4H By performing fusion feedback compensation, the enhanced feature New_F is obtained. 4ˋ ; S47. Combine the secondary sampling feature map F5ˋˋ with the enhanced feature New_F 4ˋ Together, they are input into the fourth CFM module of the dual-encoding module for enhancement, resulting in the enhanced feature New_F. 4Hˋ New_F 4Lˋ ; S48, Enhance feature New_F 4L` Compared with the enhanced feature map New_F 3` The input is fed into the fifth CFM module for enhancement, resulting in the enhanced feature New_F. 3H` New_F 3L` ; S49, Enhance feature New_F 3L` Compared with the enhanced feature map New_F 2` The input is fed into the fifth CFM module for enhancement, resulting in the enhanced feature New_F. 2H` New_F 2L` ;where New_F 2L` As another prediction output result, predict2; S410, Enhance feature New_F 2H` New_F 2L` Perform fusion feedback compensation to obtain the first output result out1; then enhance the feature New_F 2L` New_F 3H` Perform fusion feedback compensation to obtain the second output result out2; then enhance the feature New_F 2L` New_F 4H` Perform fusion feedback compensation to obtain the third output result out3; enhance the feature New_F 2L` The secondary sampled feature map F5`` is fused and compensated to obtain the fourth output result out4; the different level segmentation map is composed of the predicted output result predict1, the predicted output result predict2, the first output result out1, the second output result out2, the third output result out3, and the fourth output result out4.

3. The surface defect image segmentation method based on weight fusion and dual encoder as described in claim 1, characterized in that: The formula for calculating the total loss function in step S6 is as follows: Where L i L represents the loss calculated from two prediction outputs, predict1 and predict2. j The loss is calculated for the first output result out1, the second output result out2, the third output result out3, and the fourth output result out4, representing four different levels.

Images