Surface defect segmentation method and system based on dual-branch distinguishable features

Abstract

The present invention discloses a surface defect segmentation method and system based on dual-branch distinguishable features, the method comprising: step 1: constructing a dual-branch distinguishable feature cross-attention network; step 2: binarizing the training set labels to obtain the binary labels of the training set images; step 3: obtaining a dual-branch distinguishable feature cross-attention network model; step 4: using the semantic segmentation branch of the dual-branch distinguishable feature cross-attention network model to obtain a semantic segmentation mask for pixel-level classification; introducing a dual-branch decoder, integrating the defect semantic information from the binary segmentation branch and the semantic segmentation branch through distinguishable feature cross-attention, constructing the potential interaction between the two branches and capturing cross-domain information interaction, and using the defect features in the binary segmentation to increase the inter-class distance between defects of different categories in the semantic segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation, and in particular relates to a surface defect segmentation method and system based on dual-branch distinguishable features. Background Art

[0002] In industrial production, products inevitably exhibit various surface defects, such as fabric blemishes, board scratches, and magnetic contamination. Because worker efficiency is easily influenced by subjective judgment, missed or incorrect detections can occur, negatively impacting product quality and safety. Therefore, transitioning surface defect detection from manual inspection to automated and intelligent detection has become a critical task. Computer vision-based image segmentation can obtain the precise shape and location of defects, enabling reliable judgment of product surface quality.

[0003] Early computer vision techniques initially designed representations such as Haar and HOG to extract features, which were then used to train classifiers such as AdaBoost and SVM for defect classification. However, manual feature extraction is limited by human cognitive limitations, so key features are often overlooked. Furthermore, feature extraction and classifier training are performed separately, resulting in suboptimal performance in defect detection. Recently, computer vision defect detection using deep learning has achieved significant success. Compared to early AOI methods, deep learning can autonomously learn and extract effective features from images, resulting in good and robust detection results. These deep learning-based defect detection methods can be categorized into two main networks: detection networks and segmentation networks. Detection networks, including those from the YOLO family, RCNN, and DETR families, generate bounding boxes to locate and classify defects. In contrast, segmentation networks classify pixels in an image, enabling accurate defect localization and classification, as well as estimation of defect areas. Consequently, segmentation has been widely used in defect detection.

[0004] However, many current algorithms in semantic segmentation face the following problems. Surface defects often come in many types. Even within the same category, some defects may show similarities in local texture and color due to the influence of different environmental factors (such as temperature, humidity, light, pollution, etc.) during the manufacturing or use of materials. For example, certain types of cracks may be difficult to distinguish from scratches in specific areas, and the same type of defects may also show different characteristics depending on their location. Therefore, when identifying and segmenting these defects, misclassification or missegmentation often occurs. In addition, some surface defects have fuzzy boundaries because their surrounding areas are very similar to normal material surfaces, although the central area of ​​the defect may be easier to identify. The wide variation in defect size and shape also increases the difficulty of accurate detection. All of the above problems pose huge challenges to the semantic segmentation of surface defects. Summary of the Invention

[0005] The technical solution of the present invention addresses the problem that candidate regions and target classifications are run in series, resulting in slow running speed. A surface defect segmentation method and system based on dual-branch distinguishable features are proposed. Accurate surface defect segmentation is achieved by adopting a block-level multi-scale method in the ASPP module to enhance the receptive field, adding a binary segmentation branch to guide the semantic segmentation branch, and designing a distinguishable feature cross-attention module to construct information interaction between the two branches.

[0006] On the one hand, the technical solution of the present invention provides a surface defect segmentation method based on dual-branch distinguishable features, comprising:

[0007] Step 1: Construct a two-branch distinguishable feature cross-attention network;

[0008] The dual-branch distinguishable feature cross attention network includes a backbone network for extracting visual information, an ASPP module, and a parallel dual-branch decoder network connected in series;

[0009] The parallel dual-branch decoder network includes a distinguishable feature cross-attention module, a binary segmentation branch decoder network, and a semantic segmentation branch decoder network. The binary segmentation branch decoder network and the semantic segmentation branch decoder network run in parallel. The distinguishable feature cross-attention module is inserted into the middle part of the parallel binary segmentation branch decoder network and the semantic segmentation branch decoder network to link the binary segmentation branch decoder network and the semantic segmentation branch decoder network.

[0010] Step 2: Use the original labels of the training set images to perform binarization processing on the training set images. That is, the pixel values ​​of all defect areas in the original labels are uniformly modified to "1" to obtain the binary labels corresponding to the training set images;

[0011] Step 3: Use the original labels and binary labels of the training set images as supervision information to train a two-branch distinguishable feature cross-attention network to obtain the optimal two-branch distinguishable feature cross-attention network model;

[0012] Step 4: Input the test image to be segmented into the optimal dual-branch distinguishable feature cross-attention network model obtained in step 3 to obtain the semantic segmentation result image of the test image.

[0013] The backbone network refers to a network in which the resolution of image features decreases in stages;

[0014] Furthermore, the distinguishable feature cross-attention module sequentially performs patch embedding processing, QKV linear transformation, multi-head cross-attention calculation, and MLP feature extraction and conversion; utilizes defect features in binary segmentation in the binary segmentation branch decoder network to expand the inter-class distance in semantic segmentation in the semantic segmentation branch decoder network, thereby enhancing the distinguishability between defect categories and between defects and non-defects;

[0015] The binary segmentation branch decoder network first restores the size of the feature map by gradually upsampling, then introduces a skip connection to fuse the backbone network feature map and the binary segmentation branch decoder feature map of the same resolution, perform convolution operations, perform feature extraction and conversion, and finally generate segmentation result maps of defects and non-defects;

[0016] The semantic segmentation branch decoder network is obtained by gradually upsampling to restore the feature map size, introducing skip connections to fuse the binary segmentation branch decoder feature map and the semantic segmentation branch decoder feature map of the same resolution, and performing convolution operations to further extract and transform features, ultimately generating segmentation result maps of multiple different categories of defects.

[0017] Furthermore, the working process of the distinguishable cross attention module is as follows:

[0018] Step B1: Input features for Patch Embedding processing;

[0019] The input feature X of the binary segmentation branch decoder network bi ∈R C×H×W and the input features X of the semantic segmentation branch decoder network se ∈R C×H×W Patch Embedding is performed separately, where C, H, and W represent the batch size, number of channels, height, and width, respectively, and R represents a multidimensional tensor;

[0020] Step B2: QKV linear transformation;

[0021] The input feature X of the semantic segmentation branch decoder network se , generate query vector through linear transformation and dimensional transformation The input feature X of the binary segmentation branch decoder network bi Used to generate key vector K and numerical vector V, X bi Generated by linear transformation X bi Generated by linear transformation and dimensional transformation To achieve the goal of enhancing the semantic segmentation effect by utilizing binary segmentation information; where N represents the number of attention heads, R represents a multidimensional tensor, and C emb represents the length of the embedding vector, E represents the intermediate variable, E=H e ×W e , H e =H / patch_size, W e =W / patch_size, where patch_size represents the patch size and its value is set based on experience;

[0022] Step B3: multi-head cross attention calculation;

[0023] Cross-attention is defined as:

[0024]

[0025] Among them, "*" is the matrix multiplication operation, and "+" is the matrix addition operation; Indicates the attention feature map obtained after multi-head cross attention calculation;

[0026] Step B4: MLP feature extraction and conversion;

[0027] Transform the dimension of Att(Q,K,V) obtained in step B3 to obtain the feature map Attention∈ The feature map is used as the input of the MLP, and then the output features are obtained by applying serial convolution, activation and drop operations as follows

[0028] y=Drop(GELU(Conv1(Attention)))

[0029] z=BatchNorm2d(GELU(Conv2(y)+Attention))

[0030] output=Drop(Conv3(z))

[0031] Among them, Conv1 and Conv3 represent convolution operations with a convolution kernel size of 1×1. The former is used to adjust the dimension of the feature, and the latter is used to restore the dimension of the feature; GELU represents the Gaussian error linear unit activation function; Conv2 represents a convolution operation with a convolution kernel size of 3×3, which is mainly used for feature extraction; Drop represents the DropPath regularization operation, y and z both represent intermediate variables, and the probability value of randomly deleting the sub-path of the multi-branch structure of the model is set to 0.5.

[0032] Furthermore, the specific process of performing Patch Embedding processing on the input features is as follows:

[0033] The input features are first convolved with a kernel size and a step size of patch_size, and the number of channels, height, and width of the image are modified to C respectively. emb 、H e and W e ,get and Among them, C emb is the length of the embedding vector, H e =H / patch_size, W e =W / patch_size, where patch_size is set to 16; then, bi and X se Flatten the last two dimensions of and Then X bi and X se By exchanging the two dimensions of and Where E=H e ×W e .

[0034] Feature X from the decoder layer of the binary segmentation branch bi and the feature X in the semantic segmentation branch decoder network layer se , after Patch Embedding processing, it is effectively converted into an embedding vector of fixed size;

[0035] Furthermore, the training process of the dual-branch distinguishable feature cross-attention network is as follows:

[0036] Taking the known training set images as input, the original labels of the training set images are used to perform binarization processing on them, that is, the pixel values ​​of all defect areas in the original labels are uniformly modified to "1", and the binary labels corresponding to the training set images are obtained. The original labels and binary labels of the training set images are used as supervision information, and a joint loss function is set to train the dual-branch distinguishable feature cross-attention network to obtain the trained dual-branch distinguishable feature cross-attention network model.

[0037] Furthermore, the joint loss function L total It is composed of the binary cross entropy function of the binary segmentation branch decoder network of the parallel two-branch decoder network and the cross entropy loss function of the semantic segmentation branch decoder network, as follows:

[0038] L total =L seg +L b

[0039] Among them, L seg Represents the cross entropy loss function of the semantic segmentation branch decoder network:

[0040]

[0041] L b Represents the binary cross entropy function of the binary segmentation branch decoder network:

[0042]

[0043] Among them, y i and p i denote the label and predicted probability of the i-th image, n denotes the batch size, and k denotes the number of defect categories.

[0044] Furthermore, the parallel feature blocks in the ASPP module are combined in series and parallel to form a block-level multi-scale MSASPP module, and the MSASPP module is used to replace the ASPP module in the dual-branch distinguishable feature cross-attention network; the working process of the MSASPP module is as follows:

[0045] Step A1: input image features;

[0046] The final deep image feature x∈R with the most abstract semantic information extracted from the backbone network C×H×W Input to the MSASPP module, where R represents a multidimensional tensor, C, H, and W represent the number of channels, height, and width of the deep image features, respectively;

[0047] Step A2: 1×1 convolution;

[0048] Perform 1×1 convolution on the deep image feature x to achieve feature channel dimensionality reduction and obtain the output feature map f1∈R of the first operation 256×H×W ;

[0049] f1=σ(Conv 1×1 (x))

[0050] Where, Conv 1×1 represents 1×1 convolution operation, σ represents BN and ReLU activation operations;

[0051] Step A3: Parallel dilated convolutions and introducing block-level multi-scale between adjacent convolution operations;

[0052] The output feature map f1 obtained in step A2 is channel-concatenated with the deep image feature x, and then a 3×3 convolution with an expansion rate of 6 is performed to obtain the output feature map f2∈R 256×H×W ; The output feature map f2 is channel-concatenated with the deep image feature x, and then a 3×3 convolution with an expansion rate of 12 is performed to obtain the output feature map f3∈R of the third operation 256×H×W ; Concatenate the output feature map f3 with the input image feature x, and then perform a 3×3 convolution with an expansion rate of 18 to obtain the output feature map f4∈R of the fourth operation 256×H×W ;

[0053]

[0054] Where, DConv 3×3,d represents a 3×3 dilated convolution with a dilation rate of d, represents the channel splicing operation, d = (n-1)*6;

[0055] Step A4: global average pooling;

[0056] Perform a global average pooling operation on the deep image feature x, and then perform 1×1 convolution, BN, ReLU activation and upsampling operations in sequence to obtain the output feature map f5∈R of the fifth operation 256×H×W ;

[0057] f5=up(σ(Conv 1×1 (pool 1×1 (x))))

[0058] Where, pool 1×1 represents global average pooling, and up represents upsampling;

[0059] Step A5: feature splicing and fusion;

[0060] The output feature maps f1, f2, f3, f4 and f5 of the above steps are concatenated and fused in the channel dimension to obtain the fused feature map f out ∈R B×1280×H×W ;

[0061]

[0062] Step A6: output image features;

[0063] For the fusion feature map f out Perform 1×1 convolution, BN and ReLU activation operations in sequence to obtain the final output image feature f out ∈R 256×H×W ;

[0064] f out =σ(Conv 1×1 (f out )).

[0065] The serial structure means that the output results of the upper-level branches are sequentially used as the input of the next-level branches until the end of the last layer of branches. The parallel structure means that the branches of each layer are processed in parallel and the final output results are summarized. The block-level multi-scale idea is used to improve the original ASPP module to capture finer-grained multi-scale features, enhance the receptive field, and solve the multi-scale variation problem of surface defects.

[0066] Furthermore, the binary segmentation branch decoder network works as follows:

[0067] The ASPP module output feature block is convolved and double up-sampled to obtain feature block b1; then the feature block generated by the backbone network stage 4 is merged with b1 to complete the convolution operation to obtain the distinguishable feature cross attention input X bi , after the distinguishable feature cross attention module, the output Y is obtained bi ; Y bi After double upsampling, the feature block b2 is merged with the feature block generated in the third stage of the backbone network to complete the convolution operation to obtain the feature block b2, and the feature block b2 is transmitted to the feature layer of the same resolution in the semantic segmentation branch decoder network; the feature block b2 is double upsampling and merged with the feature block generated in the second stage of the backbone network to complete the convolution operation to obtain the feature block b3, and the feature block b3 is transmitted to the feature layer of the same resolution in the semantic segmentation branch decoder network; the feature block b3 is double upsampling to obtain the feature block b4, and then the feature block b4 is subjected to convolution operation and double upsampling to complete the binary segmentation prediction result with the same resolution as the input image.

[0068] Binary segmentation mainly focuses on the distinction between defects and non-defects, and therefore can provide semantic segmentation with stronger prior information such as defect activation and pre-positioning of position information, thereby improving the accuracy of semantic segmentation.

[0069] The backbone network can adopt any one of the backbone networks in the ResNet series, Swin-Transformer series and MobileNet series; the ResNet series network gradually reduces the feature map size through the convolution operation in the residual block, combined with the convolution with a stride of 2, from the original input to 1 / 32 resolution, with a total of five stages, and the feature map size is halved at the end of each stage; the Swin-Transformer series network uses a shift window mechanism and a self-attention mechanism to capture local and global information, and goes through five stages (stage one, stage two, stage three, stage four, stage five), each time the spatial resolution is halved through Patch Merging, and finally reaches 1 / 32 of the input resolution; the MobileNet series networks all use downsampling operations in specific stages to reduce the spatial dimension of the feature map with a stride of 2, and go through five stages, and the feature map size is halved at the end of each stage.

[0070] Furthermore, the semantic segmentation branch decoder network works as follows:

[0071] The ASPP module output feature block is convolved and double up-sampled to obtain data s1; then the data s1 is convolved to obtain the distinguishable feature cross attention input X se , get the distinguishable feature cross attention module to get the output Y se ; Y se After double upsampling, the data is merged with the feature block b2 generated by the binary segmentation branch decoder network to complete the convolution operation to obtain data s2; after double upsampling, the data s2 is merged with the feature block b3 generated by the binary segmentation branch decoder network to complete the convolution operation to obtain data s3; the data s3 features are double upsampled to obtain data s4. Through convolution operation and double upsampling, the semantic segmentation prediction result with the same resolution as the input image is completed.

[0072] Binary segmentation primarily focuses on distinguishing defects from non-defects, thus providing stronger defect activation and location information for semantic segmentation. Semantic segmentation can fully exploit these discriminative features of defect categories by obtaining prior information from binary segmentation. The generated regions and targets enhance inter-class separability, reduce misclassification, and alleviate overfitting to a certain extent.

[0073] In a second aspect, a surface defect segmentation system with dual-branch distinguishable features based on the above method comprises:

[0074] Network construction module: used to construct a dual-branch distinguishable feature cross-attention network, which includes a backbone network, an ASPP module and a parallel dual-branch decoder network connected in series;

[0075] Cross-attention is used to integrate the defect semantic information from the binary segmentation branch and the semantic segmentation branch, constructing the potential interaction between the two branches and capturing cross-domain information interaction. The defect features in the binary segmentation are used to increase the inter-class distance between different categories of defects in the semantic segmentation, thereby obtaining enhanced semantic segmentation defect features.

[0076] Training module: Use the original labels and binary labels of the training set images as supervision information to train a two-branch distinguishable feature cross-attention network to obtain the optimal two-branch distinguishable feature cross-attention network model;

[0077] Segmentation module: The test image to be segmented is input into the obtained optimal dual-branch distinguishable feature cross-attention network model, and the semantic segmentation result image of the test image is obtained through the semantic segmentation branch in the model.

[0078] Beneficial effects

[0079] Compared with the existing method, the advantages of the present invention are:

[0080] The dual-branch distinguishable feature cross-attention network in the technical solution of the present invention can simultaneously and simply and efficiently complete the tasks of binary segmentation and semantic segmentation. The network uses the MSASPP module to capture finer-grained multi-scale features and enhance the receptive field. The network introduces a dual-branch decoder and uses the binary segmentation branch to guide the semantic segmentation branch to extract finer defect boundaries. The network integrates the defect semantic information from the binary segmentation branch and the semantic segmentation branch through distinguishable feature cross-attention, constructs potential interactions between the two branches, captures cross-domain information interactions, and uses defect features in binary segmentation to increase the inter-class distance between different types of defects in semantic segmentation.

[0081] The technical solution of the present invention also uses a joint loss to train a dual-branch distinguishable feature cross-attention network. The loss used by the semantic segmentation branch is a cross-entropy loss function, and the loss used by the binary segmentation branch is a binary cross-entropy loss function.

[0082] The details are as follows:

[0083] 1. Compared to a single-branch semantic segmentation network, the dual-branch distinguishable feature cross-attention network proposed in this technical solution, benefiting from the guidance mechanism of the dual-branch decoder, can effectively use the binary segmentation branch to guide the semantic segmentation branch, extracting finer defect boundaries and resolving the problem of blurred surface defect boundaries. In addition, compared to a two-stage semantic segmentation network, the two branches of the dual-branch distinguishable feature cross-attention network run in parallel, resulting in higher segmentation efficiency.

[0084] 2. Introducing block-level multi-scale into the ASPP module captures finer-grained multi-scale features, thereby enhancing the receptive field and helping to solve the multi-scale variation problem of surface defects.

[0085] 3. A distinguishable feature cross-attention module is proposed. Cross-attention is used to integrate the defect semantic information from the binary segmentation branch and the semantic segmentation branch, construct potential interactions between the two branches and capture cross-domain information interactions. The defect features in binary segmentation are used to increase the inter-class distance between different categories of defects in semantic segmentation, which is conducive to the accurate classification of surface defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 Schematic diagram of the architecture of the surface defect segmentation method based on dual-branch distinguishable features according to an embodiment of the present invention;

[0087] Figure 2 Schematic diagrams of the structures of the original ASPP module and the MSASPP module in the embodiment of the present invention respectively;

[0088] Figure 3 Schematic diagram of a distinguishable feature cross-attention module in an embodiment of the present invention;

[0089] Figure 4 Schematic diagram showing the comparison of four semantic segmentation evaluation indicators of the method of the present invention on the pine wood dataset;

[0090] Figure 5 Schematic diagram showing the comparison of four semantic segmentation evaluation indicators of the method of the present invention on the rubber wood dataset;

[0091] Figure 6 This is a schematic diagram of the visualization results of the method of the present invention on the pine wood dataset;

[0092] Figure 7 This is a schematic diagram of the visualization results of the method of the present invention on the rubber wood dataset;

[0093] Figure 8 Schematic diagram showing the comparison of the three semantic segmentation evaluation indicators of the method of the present invention on the road crack and steel datasets, where NEU-Seg is the steel dataset and Crack500 is the road crack dataset;

[0094] Figure 9 Schematic diagram of the visualization results of the method of the present invention on the road cracking and steel datasets, (a) is the input image, (b) is the label, (c) is U-Net, (d) is DeepLabV3+, (e) SegFormer, (f) is HRNet, (g) is HRFormer, (h) is Swin-T, and (i) is the present invention; red in the first and second rows indicates crack defects from the road cracking dataset; red in the third row indicates inclusion defects in the steel dataset, and green in the fourth row indicates plaque defects in the steel dataset. DETAILED DESCRIPTION

[0095] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0096] Example 1

[0097] The embodiment of the present invention provides a surface defect segmentation method based on dual-branch distinguishable features, which adopts the architecture as follows: Figure 1 shown; in Figure 1 In

[15] , the Swin Transformer series is used as the backbone network to extract image features; MSASPP represents the introduction of the block-level multi-scale ASPP module.

[0098] The specific process is as follows:

[0099] Step 1: Construct a two-branch distinguishable feature cross-attention network;

[0100] The dual-branch distinguishable feature cross attention network includes a backbone network for extracting visual information, an MSASPP module, and a parallel dual-branch decoder network connected in series;

[0101] The MSASPP module is obtained by introducing block-level multi-scale into the ASPP module, such as Figure 2 As shown;

[0102] The general steps of the ASPP module are: input image features, parallel dilated convolution layers with different expansion rates, global average pooling layer, feature splicing and fusion, 1×1 convolution, to obtain output image features; the newly introduced block-level multi-scale is set between adjacent dilated convolution layers with different expansion rates;

[0103] The parallel dual-branch decoder network includes a distinguishable feature cross-attention module, a binary segmentation branch decoder network and a semantic segmentation branch decoder network;

[0104] The distinguishable feature cross attention module is obtained by introducing the binary segmentation branch input feature and the semantic segmentation branch input feature to the input of the cross attention module, and introducing the semantic segmentation branch input feature to the output of the cross attention module, as shown in Figure 3 As shown, the distinguishable feature cross-attention module is located in the middle of the binary segmentation branch and the semantic segmentation branch that are parallel to each other;

[0105] The general steps of the cross-attention module are: calculating the similarity between the query vector Q and the key vector K, dynamically generating the weight coefficient of the numerical vector V, performing weighted summation on V, and outputting the final attention value; the newly added introduction of the binary segmentation branch input features and the semantic segmentation branch input features to the input of the cross-attention module are set before calculating the similarity between Q and K; and the newly added introduction of the semantic segmentation branch input features to the output of the cross-attention module are set after outputting the final attention value;

[0106] The binary segmentation branch decoder network first restores the size of the feature map by gradually upsampling, then introduces a skip connection to fuse the backbone network feature map and the binary segmentation branch decoder feature map of the same resolution, perform convolution operations, perform feature extraction and conversion, and finally generate segmentation result maps of defects and non-defects;

[0107] The semantic segmentation branch decoder network is obtained by gradually upsampling to restore the feature map size, introducing skip connections to fuse the binary segmentation branch decoder feature map and the semantic segmentation branch decoder feature map of the same resolution, and performing convolution operations to further extract and transform features, ultimately generating segmentation result maps of multiple different categories of defects.

[0108] Taking Swin Transformer tiny as an example, the input known training set image size is preprocessed to 512×512, and five feature layers of different scales are extracted from the backbone network, including shallow layer, shallow layer, middle layer, sub-deep layer and deep layer, with scales of 256×256, 128×128, 64×64, 32×32 and 16×16 respectively.

[0109] The working process of the MSASPP module is:

[0110] The parallel feature blocks in the ASPP module are combined in series and parallel to form a block-level multi-scale MSASPP module, and the MSASPP module is used to replace the ASPP module in the dual-branch distinguishable feature cross-attention network. The working process of the MSASPP module is as follows:

[0111] Step A1: input image features;

[0112] The final deep image feature x∈R with the most abstract semantic information extracted from the backbone network C×H×W Input to the MSASPP module, where R represents a multidimensional tensor, C, H, and W represent the number of channels, height, and width of the deep image features, respectively. In this example, the values ​​of C, H, and W are 768, 16, and 16, respectively;

[0113] Step A2: 1×1 convolution;

[0114] Perform 1×1 convolution on the deep image feature x to achieve feature channel dimensionality reduction and obtain the output feature map f1∈R of the first operation 256×H×W ;

[0115] f1=σ(Conv 1×1 (x))

[0116] Where, Conv 1×1 represents 1×1 convolution operation, σ represents BN and ReLU activation operations;

[0117] Step A3: Parallel dilated convolutions and introducing block-level multi-scale between adjacent convolution operations;

[0118] The output feature map f1 obtained in step A2 is channel-concatenated with the deep image feature x, and then a 3×3 convolution with an expansion rate of 6 is performed to obtain the output feature map f2∈R 256×H×W ; The output feature map f2 is channel-concatenated with the deep image feature x, and then a 3×3 convolution with an expansion rate of 12 is performed to obtain the output feature map f3∈R of the third operation 256×H×W ; Concatenate the output feature map f3 with the input image feature x, and then perform a 3×3 convolution with an expansion rate of 18 to obtain the output feature map f4∈R of the fourth operation 256×H×W ;

[0119]

[0120] Where, DConv 3×3,d represents a 3×3 dilated convolution with a dilation rate of d, represents the channel splicing operation, d = (n-1)*6;

[0121] Step A4: global average pooling;

[0122] Perform a global average pooling operation on the deep image feature x, and then perform 1×1 convolution, BN, ReLU activation and upsampling operations in sequence to obtain the output feature map f5∈R of the fifth operation 256×H×W ;

[0123] f5=up(σ(Conv 1×1 (pool1×1 (x))))

[0124] Where, pool 1×1 represents global average pooling, and up represents upsampling;

[0125] Step A5: feature splicing and fusion;

[0126] The output feature maps f1, f2, f3, f4 and f5 of the above steps are concatenated and fused in the channel dimension to obtain the fused feature map f out ∈R B×1280×H×W ;

[0127]

[0128] Step A6: output image features;

[0129] For the fusion feature map f out Perform 1×1 convolution, BN and ReLU activation operations in sequence to obtain the final output image feature f out ∈R 256×H×W ;

[0130] f out =σ(Conv 1×1 (f out )).

[0131] The serial structure means that the output results of the upper-level branches are sequentially used as the input of the next-level branches until the end of the last layer of branches. The parallel structure means that the branches of each layer are processed in parallel and the final output results are summarized. The block-level multi-scale idea is used to improve the original ASPP module to capture finer-grained multi-scale features, enhance the receptive field, and solve the multi-scale variation problem of surface defects.

[0132] The working process of the distinguishable crisscross attention module is as follows:

[0133] Step B1: Input features for Patch Embedding processing;

[0134] The input feature X of the binary segmentation branch decoder network bi ∈R C×H×W and the input features X of the semantic segmentation branch decoder network se ∈R C×H×W Patch Embedding is performed separately, where C, H, and W represent the batch size, number of channels, height, and width, respectively. The values ​​of C, H, and W are 256, 32, and 32, respectively.

[0135] The specific process of Patch Embedding processing of the input features is as follows:

[0136] The input features are first modified to C by a convolution operation with kernel size and kernel step size of patch_size. emb 、H e and W e ,get and Among them, C emb is the length of the embedding vector, H e =H / patch_size, W e =W / patch_size, patch_size indicates the patch size, here patch_size is set to 16; then, X bi and X se Flatten the last two dimensions of and Then X bi and X se By exchanging the two dimensions of and Where E=H e ×W e . C emb In the embodiment of the present invention, the value is 256;

[0137] Feature X from the decoder layer of the binary segmentation branch bi and the feature X in the semantic segmentation branch decoder network layer se , after Patch Embedding processing, it is effectively converted into an embedding vector of fixed size;

[0138] Step B2: QKV linear transformation;

[0139] The input feature X of the semantic segmentation branch decoder network se , generate query vector through linear transformation and dimensional transformation The input feature X of the binary segmentation branch decoder network bi Used to generate key vector K and numerical vector V, X bi Generated by linear transformation X bi Generated by linear transformation and dimensional transformation To achieve the goal of enhancing the semantic segmentation effect by utilizing binary segmentation information; where N represents the number of attention heads, R represents a multidimensional tensor, and C emb represents the length of the embedding vector, E represents the intermediate variable, E=H e ×W e ;

[0140] In the embodiment of the present invention, N is set to 16;

[0141] Step B3: multi-head cross attention calculation;

[0142] Cross-attention is defined as:

[0143]

[0144] Among them, "*" is the matrix multiplication operation, and "+" is the matrix addition operation; Indicates the attention feature map obtained after multi-head cross attention calculation;

[0145] Step B4: MLP feature extraction and conversion;

[0146] Transform the dimension of Att(Q,K,V) obtained in step B3 to obtain the feature map Attention∈ The feature map is used as the input of the MLP, and then the output features are obtained by applying serial convolution, activation and drop operations as follows

[0147] y=Drop(GELU(Conv1(Attention)))

[0148] z=BatchNorm2d(GELU(Conv2(y)+Attention))

[0149] output=Drop(Conv3(z))

[0150] Among them, Conv1 and Conv3 represent convolution operations with a convolution kernel size of 1×1. The former is used to adjust the dimension of the feature, and the latter is used to restore the dimension of the feature; GELU represents the Gaussian error linear unit activation function; Conv2 represents a convolution operation with a convolution kernel size of 3×3, which is mainly used for feature extraction; Drop represents the DropPath regularization operation, y and z both represent intermediate variables, and the probability value of randomly deleting the sub-path of the multi-branch structure of the model is set to 0.5.

[0151] The binary segmentation branch decoder network works as follows:

[0152] The ASPP module output feature block is convolved and double up-sampled to obtain feature block b1; then the feature block generated by the backbone network stage 4 is merged with b1 to complete the convolution operation to obtain the distinguishable feature cross attention input X bi , after the distinguishable feature cross attention module, the output Y is obtained bi ; Y biAfter double upsampling, the feature block b2 is merged with the feature block generated in the third stage of the backbone network to complete the convolution operation to obtain the feature block b2, and the feature block b2 is transmitted to the feature layer of the same resolution in the semantic segmentation branch decoder network; the feature block b2 is double upsampling and merged with the feature block generated in the second stage of the backbone network to complete the convolution operation to obtain the feature block b3, and the feature block b3 is transmitted to the feature layer of the same resolution in the semantic segmentation branch decoder network; the feature block b3 is double upsampling to obtain the feature block b4, and then the feature block b4 is subjected to convolution operation and double upsampling to complete the binary segmentation prediction result with the same resolution as the input image.

[0153] Binary segmentation mainly focuses on the distinction between defects and non-defects, and therefore can provide semantic segmentation with stronger prior information such as defect activation and pre-positioning of position information, thereby improving the accuracy of semantic segmentation.

[0154] The backbone network can adopt any one of the backbone networks in the ResNet series, Swin-Transformer series and MobileNet series; the ResNet series network gradually reduces the feature map size through the convolution operation in the residual block, combined with the convolution with a stride of 2, from the original input to 1 / 32 resolution, with a total of five stages, and the feature map size is halved at the end of each stage; the Swin-Transformer series network uses a shift window mechanism and a self-attention mechanism to capture local and global information, and goes through five stages, each time the spatial resolution is halved through Patch Merging, eventually reaching 1 / 32 of the input resolution; the MobileNet series networks all use downsampling operations in specific stages to reduce the spatial dimension of the feature map with a stride of 2, and go through five stages, and the feature map size is halved at the end of each stage.

[0155] The semantic segmentation branch decoder network works as follows:

[0156] The ASPP module output feature block is convolved and double up-sampled to obtain data s1; then the data s1 is convolved to obtain the distinguishable feature cross attention input X se , get the distinguishable feature cross attention module to get the output Y se ; Y se After double upsampling, the data is merged with the feature block b2 generated by the binary segmentation branch decoder network to complete the convolution operation to obtain data s2; after double upsampling, the data s2 is merged with the feature block b3 generated by the binary segmentation branch decoder network to complete the convolution operation to obtain data s3; the data s3 features are double upsampled to obtain data s4. Through convolution operation and double upsampling, the semantic segmentation prediction result with the same resolution as the input image is completed.

[0157] Binary segmentation primarily focuses on distinguishing defects from non-defects, thus providing stronger defect activation and location information for semantic segmentation. Semantic segmentation can fully exploit these discriminative features of defect categories by obtaining prior information from binary segmentation. The generated regions and targets enhance inter-class separability, reduce misclassification, and alleviate overfitting to a certain extent.

[0158] Step 2: Use the original labels of the training set images to perform binarization processing on the training set images. That is, the pixel values ​​of all defect areas in the original labels are uniformly modified to "1" to obtain the binary labels corresponding to the training set images;

[0159] Step 3: Use the original labels and binary labels of the training set images as supervision information to train a two-branch distinguishable feature cross-attention network to obtain the optimal two-branch distinguishable feature cross-attention network model;

[0160] The training process of the two-branch distinguishable feature cross-attention network is as follows:

[0161] Taking the known training set images as input, the original labels and binary labels of the training set images obtained in step 2 as supervision information, setting the joint loss function, training the dual-branch distinguishable feature cross-attention network, and obtaining the trained dual-branch distinguishable feature cross-attention network model.

[0162] The joint loss function L total It is composed of the cross entropy loss function of the semantic segmentation branch and the binary cross entropy loss function of the binary segmentation branch. The formula is:

[0163] L total =L seg +L b

[0164] Among them, L seg Represents the cross entropy loss function:

[0165]

[0166] Among them, y i and p i They represent the label and predicted probability of the i-th image, n represents the batch size, and k represents the number of defect categories;

[0167] L b Represents the binary cross entropy loss function:

[0168]

[0169] Step 4: Input the test image to be segmented into the optimal dual-branch distinguishable feature cross-attention network model obtained in step 3, and obtain the semantic segmentation result image of the test image through the semantic segmentation branch in the model.

[0170] like Figure 4 、 5 As shown in the figure, the method described in the embodiment of the present invention achieved the best results in the two semantic segmentation evaluation indicators mIoU and mACC on the pine wood dataset and the rubber wood dataset, and the segmentation accuracy exceeded the existing methods, verifying the effectiveness of the technical solution of the present invention in the surface defect segmentation task.

[0171] like Figure 6 、 7 As shown, the visualization results of the method described in the embodiment of the present invention on two datasets, a pine wood dataset and a rubber wood dataset, show that the quality of the prediction mask and classification of the present invention exceeds that of the existing methods, verifying the effectiveness of the present invention.

[0172] like Figure 8 As shown, a schematic diagram of the comparison of the evaluation indicators of three semantic segmentation of the method described in an embodiment of the present invention on two datasets, road cracking and steel datasets, is provided. The effectiveness of the technical solution of the present invention in various surface defect segmentation tasks verifies the effectiveness of the present invention.

[0173] like Figure 9 As shown, the visualization results of the method described in the embodiment of the present invention on two datasets, road cracking and steel dataset, show that the quality of the prediction mask and classification of the present invention exceeds that of the existing methods, verifying the effectiveness of the present invention.

[0174] Example 2

[0175] A surface defect segmentation system with dual-branch distinguishable features based on the above method comprises:

[0176] Network construction module: used to construct a dual-branch distinguishable feature cross-attention network, which includes a backbone network, an ASPP module and a parallel dual-branch decoder network connected in series;

[0177] Cross-attention is used to integrate the defect semantic information from the binary segmentation branch and the semantic segmentation branch, constructing the potential interaction between the two branches and capturing cross-domain information interaction. The defect features in the binary segmentation are used to increase the inter-class distance between different categories of defects in the semantic segmentation, thereby obtaining enhanced semantic segmentation defect features.

[0178] Training module: Use the original labels and binary labels of the training set images as supervision information to train a two-branch distinguishable feature cross-attention network to obtain the optimal two-branch distinguishable feature cross-attention network model;

[0179] Segmentation module: The test image to be segmented is input into the obtained optimal dual-branch distinguishable feature cross-attention network model, and the semantic segmentation result image of the test image is obtained through the semantic segmentation branch in the model.

[0180] It should be understood that the implementation process of each module can refer to the description of the aforementioned method. The above division of functional modules is merely a division of logical functions. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0181] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0182] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A surface defect segmentation method based on dual-branch distinguishable features, characterized in that: include: Step 1: Construct a two-branch distinguishable feature cross-attention network; The dual-branch distinguishable feature cross attention network includes a backbone network for extracting visual information, an ASPP module, and a parallel dual-branch decoder network connected in series; The parallel dual-branch decoder network includes a distinguishable feature cross-attention module, a binary segmentation branch decoder network, and a semantic segmentation branch decoder network. The binary segmentation branch decoder network and the semantic segmentation branch decoder network run in parallel. The distinguishable feature cross-attention module is inserted into the middle part of the parallel binary segmentation branch decoder network and the semantic segmentation branch decoder network to link the binary segmentation branch decoder network and the semantic segmentation branch decoder network. Step 2: Use the original labels of the training set images to perform binarization processing on the training set images. That is, the pixel values ​​of all defect areas in the original labels are uniformly changed to "1" to obtain the binary labels corresponding to the training set images; Step 3: Use the original labels and binary labels of the training set images as supervision information to train a two-branch distinguishable feature cross-attention network to obtain the optimal two-branch distinguishable feature cross-attention network model; Step 4: Input the test image to be segmented into the optimal dual-branch distinguishable feature cross-attention network model obtained in step 3 to obtain the semantic segmentation result image of the test image.

2. The method according to claim 1, characterized in that The distinguishable feature cross attention module sequentially performs Patch Embedding processing, QKV linear transformation, multi-head cross attention calculation, and MLP feature extraction and conversion; The binary segmentation branch decoder network first restores the size of the feature map by gradually upsampling, then introduces a skip connection to fuse the backbone network feature map and the binary segmentation branch decoder feature map of the same resolution, perform convolution operations, perform feature extraction and conversion, and finally generate segmentation result maps of defects and non-defects; The semantic segmentation branch decoder network is obtained by gradually upsampling to restore the feature map size, introducing skip connections to fuse the binary segmentation branch decoder feature map and the semantic segmentation branch decoder feature map of the same resolution, and performing convolution operations to further extract and transform features, ultimately generating segmentation result maps of multiple different categories of defects.

3. The method according to claim 2, characterized in that The working process of the distinguishable feature cross attention module is as follows: Step B1: Input features for Patch Embedding processing; The input feature X of the binary segmentation branch decoder network bi ∈R C×H×W and the input features X of the semantic segmentation branch decoder network se ∈R C×H×W Patch Embedding is performed separately, where C, H, and W represent the batch size, number of channels, height, and width, respectively, and R represents a multidimensional tensor; Step B2: QKV linear transformation; The input feature X of the semantic segmentation branch decoder network se , generate query vector through linear transformation and dimensional transformation The input feature X of the binary segmentation branch decoder network bi Used to generate key vector K and numerical vector V, X bi Generated by linear transformation X bi Generated by linear transformation and dimensional transformation To achieve the goal of enhancing the semantic segmentation effect by utilizing binary segmentation information; where N represents the number of attention heads, R represents a multidimensional tensor, and C emb represents the length of the embedding vector, E represents the intermediate variable, E=H e ×W e , H e =H / patch_size, W e =W / patch_size, where patch_size represents the patch size and its value is set based on experience; Step B3: multi-head cross attention calculation; Cross-attention is defined as: Among them, "*" is the matrix multiplication operation, and "+" is the matrix addition operation; Indicates the attention feature map obtained after multi-head cross attention calculation; Step B4: MLP feature extraction and conversion; Transform the dimension of Att(Q,K,V) obtained in step B3 to obtain the feature map The feature map is used as the input of the MLP, and then the output features are obtained by applying serial convolution, activation and drop operations as follows y=Drop(GELU(Conv1(Attention))) z=BatchNorm2d(GELU(Conv2(y)+Attention)) output=Drop(Conv3(z)) Among them, Conv1 and Conv3 represent convolution operations with a convolution kernel size of 1×1. The former is used to adjust the dimension of the feature, and the latter is used to restore the dimension of the feature; GELU represents the Gaussian error linear unit activation function; Conv2 represents a convolution operation with a convolution kernel size of 3×3, which is used for feature extraction; Drop represents the DropPath regularization operation, and y and z both represent intermediate variables.

4. The method according to claim 3, characterized in that The specific process of Patch Embedding processing of the input features is as follows: The input features are first convolved with a kernel size and a step size of patch_size, and the number of channels, height, and width of the image are modified to C respectively. emb 、H e and W e ,get and Next, X bi and X se Flatten the last two dimensions of and Then X bi and X se By exchanging the two dimensions of and 5. The method according to claim 1, wherein The training process of the dual-branch distinguishable feature cross-attention network is as follows: Taking the known training set images as input, the original labels of the training set images are used to perform binarization processing on them, that is, the pixel values ​​of all defect areas in the original labels are uniformly modified to "1", and the binary labels corresponding to the training set images are obtained. The original labels and binary labels of the training set images are used as supervision information, and a joint loss function is set to train the dual-branch distinguishable feature cross-attention network, and the trained dual-branch distinguishable feature cross-attention network model is obtained.

6. The method according to claim 5, characterized in that The joint loss function L total It is composed of the binary cross entropy function of the binary segmentation branch decoder network of the parallel two-branch decoder network and the cross entropy loss function of the semantic segmentation branch decoder network, as follows: L total =L seg +L b Among them, L seg Represents the cross entropy loss function of the semantic segmentation branch decoder network: L b Represents the binary cross entropy function of the binary segmentation branch decoder network: Among them, y i and p i denote the label and predicted probability of the i-th image, n denotes the batch size, and k denotes the number of defect categories.

7. The method according to claim 2, characterized in that The parallel feature blocks in the ASPP module are combined in series and parallel to form a block-level multi-scale MSASPP module, and the MSASPP module is used to replace the ASPP module in the dual-branch distinguishable feature cross-attention network. The working process of the MSASPP module is as follows: Step A1: input image features; The final deep image feature x∈R with the most abstract semantic information extracted from the backbone network C×H×W Input to the MSASPP module, where R represents the real number domain, C, H, and W represent the number of channels, height, and width of the deep image features respectively; Step A2: 1×1 convolution; Perform 1×1 convolution on the deep image feature x to achieve feature channel dimensionality reduction and obtain the output feature map f1∈R of the first operation 256×H×W ; f1=σ(Conv 1×1 (x)) Where, Conv 1×1 represents a 1×1 convolution operation, and σ represents a BN and ReLU activation operation; Step A3: Parallel dilated convolutions and introducing block-level multi-scale between adjacent convolution operations; The output feature map f1 obtained in step A2 is channel-concatenated with the deep image feature x, and then a 3×3 convolution with an expansion rate of 6 is performed to obtain the output feature map f2∈R 256×H×W ; The output feature map f2 is channel-concatenated with the deep image feature x, and then a 3×3 convolution with an expansion rate of 12 is performed to obtain the output feature map f3∈R of the third operation 256 ×H×W ; Concatenate the output feature map f3 with the input image feature x, and then perform a 3×3 convolution with an expansion rate of 18 to obtain the output feature map f4∈R of the fourth operation 256×H×W ; Where, DConv 3×3,d represents a 3×3 dilated convolution with a dilation rate of d, represents the channel splicing operation, d = (n-1)*6; Step A4: global average pooling; Perform a global average pooling operation on the deep image feature x, and then perform 1×1 convolution, BN, ReLU activation and upsampling operations in sequence to obtain the output feature map f5∈R of the fifth operation 256×H×W ; f5=up(σ(Conv 1×1 (pool 1×1 (x)))) Where, pool 1×1 represents global average pooling, and up represents upsampling; Step A5: feature splicing and fusion; The output feature maps f1, f2, f3, f4 and f5 of the above steps are concatenated and fused in the channel dimension to obtain the fused feature map f out ∈R B×1280×H×W ; Step A6: output image features; For the fusion feature map f out Perform 1×1 convolution, BN and ReLU activation operations in sequence to obtain the final output image feature f out ∈R 256×H×W ; f out =σ(Conv 1×1 (f out ))。 8. The method according to claim 2, characterized in that The binary segmentation branch decoder network works as follows: The ASPP module output feature block is convolved and double up-sampled to obtain feature block b1; then the feature block generated by the backbone network stage 4 is merged with b1 to complete the convolution operation to obtain the distinguishable feature cross attention input X bi , after the distinguishable feature cross attention module, the output Y is obtained bi ; Y bi After double upsampling, the feature block b2 is merged with the feature block generated in the third stage of the backbone network to complete the convolution operation to obtain the feature block b2, and the feature block b2 is transmitted to the feature layer of the semantic segmentation branch decoder network with the same resolution; after double upsampling, the feature block b2 is merged with the feature block generated in the second stage of the backbone network to complete the convolution operation to obtain the feature block b3, and the feature block b3 is transmitted to the feature layer of the semantic segmentation branch decoder network with the same resolution; The feature block b3 is upsampled twice to obtain the feature block b4, and then the feature block b4 is subjected to convolution operation and upsampled twice to complete the binary segmentation prediction result with the same resolution as the input image.

9. The method according to claim 8, characterized in that The semantic segmentation branch decoder network works as follows: The ASPP module output feature block is convolved and double up-sampled to obtain data s1; then the data s1 is convolved to obtain the distinguishable feature cross attention input X se , get the distinguishable feature cross attention module to get the output Y se ; Y se After double upsampling, the data is merged with the feature block b2 generated by the binary segmentation branch decoder network to complete the convolution operation to obtain data s2; after double upsampling, the data s2 is merged with the feature block b3 generated by the binary segmentation branch decoder network to complete the convolution operation to obtain data s3; the data s3 features are double upsampled to obtain data s4. Through convolution operation and double upsampling, the semantic segmentation prediction result with the same resolution as the input image is completed.

10. A surface defect segmentation system based on the dual-branch distinguishable features of the method according to any one of claims 1 to 9, characterized in that: include: Network construction module: used to construct a dual-branch distinguishable feature cross-attention network, which includes a backbone network, an ASPP module and a parallel dual-branch decoder network connected in series; Cross-attention is used to integrate the defect semantic information from the binary segmentation branch and the semantic segmentation branch, constructing the potential interaction between the two branches and capturing cross-domain information interaction. The defect features in the binary segmentation are used to increase the inter-class distance between different categories of defects in the semantic segmentation, thereby obtaining enhanced semantic segmentation defect features. Training module: Use the original labels and binary labels of the training set images as supervision information to train a two-branch distinguishable feature cross-attention network to obtain the optimal two-branch distinguishable feature cross-attention network model; Segmentation module: The test image to be segmented is input into the obtained optimal dual-branch distinguishable feature cross-attention network model, and the semantic segmentation result image of the test image is obtained through the semantic segmentation branch in the model.

Images