Surface defect segmentation method and system based on dual-branch distinguishable features

Abstract

The invention discloses a surface defect segmentation method and system based on double-branch distinguishable features. The method comprises the following steps: step 1, constructing a double-branch distinguishable feature cross attention network; 2, binarization processing is carried out on the training set labels, and binary labels of the training set images are obtained; step 3, obtaining a double-branch distinguishable feature cross attention network model; step 4, using semantic segmentation branches of the double-branch distinguishable feature cross attention network model to obtain semantic segmentation masks of pixel-level classification; a double-branch decoder is introduced, defect semantic information from a binary segmentation branch and a semantic segmentation branch is integrated through distinguishable feature cross attention, potential interaction between double branches is constructed, cross-domain information interaction is captured, and the inter-class distance between different classes of defects in semantic segmentation is increased by utilizing defect features in binary segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation, and in particular relates to a surface defect segmentation method and system based on dual-branch distinguishable features. Background Art

[0002] In industrial production, products inevitably have various surface defects, such as fabric flaws, board scratches, magnetic contamination, etc. Since workers' efficiency is easily affected by subjective judgment, problems such as missed inspections and wrong inspections may occur, which will have a negative impact on product quality and safety. Therefore, it has become a key task to transform surface defect detection from manual inspection to automation and intelligence. Image segmentation tasks based on computer vision can obtain the precise shape and location information of defects, so as to make reliable judgments on product surface quality.

[0003] Early computer vision techniques initially designed representations, such as Haar, HOG, etc., to extract features, which were then used to train classifiers such as AdaBoost and SVM for defect classification. However, manual feature extraction is limited by human cognitive limitations, so key features are often overlooked. In addition, feature extraction and classifier training are performed separately, so these methods often lead to suboptimal performance in defect detection. Recently, computer vision detection of defects through deep learning has achieved significant success. Compared with early AOI methods, deep learning can autonomously learn and extract effective features from images, thereby obtaining good and robust detection results. These defect detection methods based on deep learning methods can be divided into two main networks: detection network and segmentation network. The detection network generates bounding boxes to locate and classify defects, including the YOLO series, RCNN series, and DETR series. In contrast, the segmentation network classifies pixels in the image, which can achieve accurate defect location and classification, and estimate the defect area. Therefore, segmentation has been widely used in defect detection.

[0004] However, many current algorithms in semantic segmentation face the following problems. Surface defects often come in many types. Even within the same category, some different defects may show similarities in local texture and color due to the influence of different environmental factors (such as temperature, humidity, light, pollution, etc.) during the manufacturing or use of materials. For example, some types of cracks may be difficult to distinguish from scratches in specific areas, and the same type of defects may also show different characteristics depending on the location. Therefore, when identifying and segmenting these defects, misclassification or missegmentation often occurs. In addition, some surface defects have fuzzy boundaries because their surrounding areas are very similar to normal material surfaces, although the central area of ​​the defect may be easier to identify. The wide variation in size and shape of defects also increases the difficulty of accurate detection. All of the above problems have brought great challenges to the semantic segmentation of surface defects. Summary of the invention

[0005] The technical solution of the present invention aims at the problem that candidate regions and target classifications are run in series, resulting in a slow running speed. A surface defect segmentation method and system based on dual-branch distinguishable features are proposed. The block-level multi-scale method is adopted in the ASPP module to enhance the receptive field, a binary segmentation branch is added to guide the semantic segmentation branch, and a distinguishable feature cross-attention module is designed to construct information interaction between the dual branches, thereby achieving accurate surface defect segmentation.

[0006] On the one hand, the technical solution of the present invention provides a surface defect segmentation method based on dual-branch distinguishable features, comprising:

[0007] Step 1: Construct a dual-branch distinguishable feature cross-attention network;

[0008] The dual-branch distinguishable feature cross-attention network includes a backbone network for extracting visual information, an ASPP module, and a parallel dual-branch decoder network connected in series;

[0009] The parallel dual-branch decoder network includes a distinguishable feature cross-attention module, a binary segmentation branch decoder network and a semantic segmentation branch decoder network, the binary segmentation branch decoder network and the semantic segmentation branch decoder network run in parallel, and the distinguishable feature cross-attention module is inserted into the middle part of the parallel binary segmentation branch decoder network and the semantic segmentation branch decoder network to link the binary segmentation branch decoder network and the semantic segmentation branch decoder network;

[0010] Step 2: Use the original labels of the training set images to perform binary processing on the training set images, that is, uniformly modify the pixel values ​​of all defective areas in the original labels to "1" to obtain the binary labels corresponding to the training set images;

[0011] Step 3: Use the original labels and binary labels of the training set images as supervision information at the same time, train the dual-branch distinguishable feature cross-attention network, and obtain the optimal dual-branch distinguishable feature cross-attention network model;

[0012] Step 4: Input the test image to be segmented into the optimal dual-branch distinguishable feature cross-attention network model obtained in step 3 to obtain the semantic segmentation result image of the test image.

[0013] The backbone network refers to a network in which the resolution of image features decreases in stages;

[0014] Furthermore, the distinguishable feature cross attention module sequentially performs Patch Embedding processing, QKV linear transformation, multi-head cross attention calculation, and MLP feature extraction and conversion; utilizes defect features in binary segmentation in the binary segmentation branch decoder network to expand the inter-class distance in semantic segmentation in the semantic segmentation branch decoder network, thereby enhancing the distinguishability between defect categories and between defects and non-defects;

[0015] The binary segmentation branch decoder network first restores the feature map size by gradually upsampling, then introduces a skip connection to fuse and convolve the backbone network feature map and the binary segmentation branch decoder feature map of the same resolution, extracts and converts features, and finally generates segmentation result maps of defects and non-defects;

[0016] The semantic segmentation branch decoder network is obtained by gradually upsampling to restore the feature map size, introducing jump connections to fuse the binary segmentation branch decoder feature map and the semantic segmentation branch decoder feature map of the same resolution, and performing convolution operations to further extract and transform features, ultimately generating segmentation result maps of multiple different categories of defects.

[0017] Furthermore, the working process of the distinguishable cross attention module is:

[0018] Step B1: Input features for Patch Embedding processing;

[0019] The input feature X of the binary segmentation branch decoder network bi ∈R C×H×W and the input features X of the semantic segmentation branch decoder network se ∈R C×H×W Patch Embedding is performed separately, where C, H, and W represent the batch size, number of channels, height, and width, respectively, and R represents a multidimensional tensor;

[0020] Step B2: QKV linear transformation;

[0021] The input feature X of the semantic segmentation branch decoder network se , generate query vector through linear transformation and dimensional transformation Input feature X of the binary segmentation branch decoder network bi Used to generate key vector K and numerical vector V, X bi Generated by linear transformation X bi Generated by linear transformation and dimensional transformation In order to achieve the goal of enhancing the semantic segmentation effect by utilizing binary segmentation information; where N represents the number of attention heads, R represents a multidimensional tensor, and C emb represents the length of the embedding vector, E represents the intermediate variable, E=H e ×W e , H e =H / patch_size, W e =W / patch_size, patch_size indicates the patch size, and its value is set based on experience;

[0022] Step B3: multi-head cross attention calculation;

[0023] Cross-attention is defined as:

[0024]

[0025] Among them, "*" is the matrix multiplication operation, and "+" is the matrix addition operation; It indicates that the attention feature map is obtained after multi-head cross attention calculation;

[0026] Step B4: MLP feature extraction and conversion;

[0027] Transform the dimension of Att(Q,K,V) obtained in step B3 to obtain the feature map Attention∈ The feature map is used as the input of the MLP, and then the output feature is obtained by applying serial convolution, activation and drop operations as follows

[0028] y=Drop(GELU(Conv1(Attention)))

[0029] z=BatchNorm2d(GELU(Conv2(y)+Attention))

[0030] output = Drop(Conv3(z))

[0031] Among them, Conv1 and Conv3 represent convolution operations with a convolution kernel size of 1×1. The former is used to adjust the dimension of the feature, and the latter is used to restore the dimension of the feature; GELU represents the Gaussian error linear unit activation function; Conv2 represents a convolution operation with a convolution kernel size of 3×3, which is mainly used for feature extraction; Drop represents the DropPath regularization operation, y and z both represent intermediate variables, and the probability value of randomly deleting the sub-path of the multi-branch structure of the model is set to 0.5.

[0032] Furthermore, the specific process of performing Patch Embedding processing on the input features is as follows:

[0033] The input features are first convolved with a kernel size and a step size of patch_size, and the number of channels, height, and width of the image are modified to C respectively. emb , H e and W e ,get and Among them, C emb is the length of the embedding vector, H e =H / patch_size, W e =W / patch_size, where patch_size is set to 16; then, bi and X se Flatten the last two dimensions of and Then X bi and X se By exchanging the two dimensions of and Where E = H e ×W e .

[0034] Feature X from the decoder layer of the binary segmentation branch bi and the feature X in the decoder network layer of the semantic segmentation branch se , after Patch Embedding processing, it is effectively converted into an embedding vector of fixed size;

[0035] Furthermore, the training process of the dual-branch distinguishable feature cross-attention network is as follows:

[0036] Taking the known training set images as input, the original labels of the training set images are used to perform binarization processing on them, that is, the pixel values ​​of all defect areas in the original labels are uniformly modified to "1", and the binary labels corresponding to the training set images are obtained. The original labels and binary labels of the training set images are used as supervision information, and a joint loss function is set to train a dual-branch distinguishable feature cross-attention network to obtain a trained dual-branch distinguishable feature cross-attention network model.

[0037] Furthermore, the joint loss function L total It is composed of the binary classification cross entropy function of the binary segmentation branch decoder network of the parallel two-branch decoder network and the cross entropy loss function of the semantic segmentation branch decoder network, as follows:

[0038] L total =L seg +L b

[0039] Among them, L seg Represents the cross entropy loss function of the semantic segmentation branch decoder network:

[0040]

[0041] L b Represents the binary cross entropy function of the binary segmentation branch decoder network:

[0042]

[0043] Among them, y i and p i They represent the label and predicted probability of the i-th image, n represents the batch size, and k represents the number of defect categories.

[0044] Furthermore, the parallel feature blocks in the ASPP module are combined in series and in parallel to form a block-level multi-scale MSASPP module, and the MSASPP module is used to replace the ASPP module in the dual-branch distinguishable feature cross-attention network; the working process of the MSASPP module is as follows:

[0045] Step A1: input image features;

[0046] The final deep image feature x∈R with the most abstract semantic information extracted from the backbone network C×H×W Input to the MSASPP module, where R represents a multidimensional tensor, and C, H, and W represent the number of channels, height, and width of the deep image features, respectively;

[0047] Step A2: 1×1 convolution;

[0048] Perform 1×1 convolution on the deep image feature x to achieve feature channel dimensionality reduction and obtain the output feature map f1∈R of the first operation 256×H×W ;

[0049] f1=σ(Conv 1×1 (x)

[0050] In the formula, Conv 1×1 represents 1×1 convolution operation, σ represents BN and ReLU activation operations;

[0051] Step A3: parallel dilated convolutions and introduce block-level multi-scale between adjacent convolution operations;

[0052] The output feature map f1 obtained in step A2 is concatenated with the deep image feature x through channels, and then a 3×3 convolution with a dilation rate of 6 is performed to obtain the output feature map f2∈R 256×H×W ; Concatenate the output feature map f2 with the deep image feature x, and then perform a 3×3 convolution with a dilation rate of 12 to obtain the output feature map f3∈R of the third operation 256×H×W ; Concatenate the output feature map f3 with the input image feature x, and then perform a 3×3 convolution with a dilation rate of 18 to obtain the output feature map f4∈R of the fourth operation 256×H×W ;

[0053]

[0054] In the formula, DConv 3×3,d represents a 3×3 dilated convolution with a dilation rate of d, represents the channel splicing operation, d = (n-1)*6;

[0055] Step A4: global average pooling;

[0056] Perform a global average pooling operation on the deep image feature x, and then perform 1×1 convolution, BN, ReLU activation and upsampling operations in sequence to obtain the output feature map f5∈R of the fifth operation 256×H×W ;

[0057] f5=up(σ(Conv 1×1 (pool 1×1 (x))))

[0058] In the formula, pool 1×1 represents global average pooling, and up represents upsampling;

[0059] Step A5: feature splicing and fusion;

[0060] The output feature maps f1, f2, f3, f4 and f5 of the above steps are concatenated and fused in the channel dimension to obtain the fused feature map f out ∈R B×1280×H×W ;

[0061]

[0062] Step A6: output image features;

[0063] For the fusion feature map f out Perform 1×1 convolution, BN and ReLU activation operations in sequence to obtain the final output image feature f out ∈R 256×H×W ;

[0064] f out =σ(Conv 1×1 (f out )).

[0065] The serial structure means that the output results of the upper-level branches are sequentially used as the input of the next-level branches until the end of the last level of branches. The parallel structure means that each level of branches is processed in parallel and the final output results are summarized. The block-level multi-scale idea is used to improve the original ASPP module to capture finer-grained multi-scale features, enhance the receptive field, and solve the multi-scale variation problem of surface defects.

[0066] Furthermore, the binary segmentation branch decoder network works as follows:

[0067] The output feature block of the ASPP module is convolved and double up-sampled to obtain the feature block b1; next, the feature block generated in the fourth stage of the backbone network is merged with b1 to complete the convolution operation to obtain the distinguishable feature cross attention input X bi , after the distinguishable feature cross attention module, the output Y is obtained bi ; Y bi After double upsampling, the feature block b2 is merged with the feature block generated in the third stage of the backbone network to complete the convolution operation to obtain the feature block b2, and the feature block b2 is transmitted to the feature layer of the semantic segmentation branch decoder network with the same resolution; the feature block b2 is double upsampled and merged with the feature block generated in the second stage of the backbone network to complete the convolution operation to obtain the feature block b3, and the feature block b3 is transmitted to the feature layer of the semantic segmentation branch decoder network with the same resolution; the feature block b3 is double upsampled to obtain the feature block b4, and then the feature block b4 is convolutionally and double upsampled to complete the binary segmentation prediction result with the same resolution as the input image.

[0068] Binary segmentation mainly focuses on the distinction between defects and non-defects, and can therefore provide stronger prior information such as defect activation and pre-positioning of location information for semantic segmentation, improving the accuracy of semantic segmentation.

[0069] The backbone network can adopt any backbone network of the ResNet series, the Swin-Transformer series and the MobileNet series; the ResNet series network gradually reduces the feature map size through the convolution operation in the residual block combined with the convolution with a step size of 2, from the original input to 1 / 32 resolution, with a total of five stages, and the feature map size is halved at the end of each stage; the Swin-Transformer series network uses a shift window mechanism and a self-attention mechanism to capture local and global information, and goes through five stages (stage one, stage two, stage three, stage four, stage five), each time the spatial resolution is halved through Patch Merging, and finally reaches 1 / 32 of the input resolution; the MobileNet series networks all use downsampling operations in specific stages to reduce the spatial dimension of the feature map in a step size of 2, and go through five stages, and the feature map size is halved at the end of each stage.

[0070] Furthermore, the semantic segmentation branch decoder network works as follows:

[0071] The output feature block of the ASPP module is convolved and double up-sampled to obtain data s1; then the data s1 is convolved to obtain the distinguishable feature cross-attention input X se , get the distinguishable feature cross attention module to get the output Y se ; Y se After double upsampling, the data is merged with the feature block b2 generated by the binary segmentation branch decoder network to complete the convolution operation and obtain data s2; after double upsampling, the data s2 is merged with the feature block b3 generated by the binary segmentation branch decoder network to complete the convolution operation and obtain data s3; the data s3 feature is double upsampled to obtain data s4, and the semantic segmentation prediction result with the same resolution as the input image is completed through convolution operation and double upsampling.

[0072] Binary segmentation mainly focuses on the distinction between defects and non-defects, so it can provide stronger defect activation and location information pre-positioning for semantic segmentation. Semantic segmentation can fully extract these discriminative features of defect categories by obtaining prior information of binary segmentation, enhance the separability between classes through the generated regions and targets, reduce misclassification, and alleviate the overfitting problem to a certain extent.

[0073] In a second aspect, a surface defect segmentation system with dual-branch distinguishable features based on the above method comprises:

[0074] Network construction module: used to construct a dual-branch distinguishable feature cross-attention network, wherein the dual-branch distinguishable feature cross-attention network includes a backbone network, an ASPP module, and a parallel dual-branch decoder network connected in series in sequence;

[0075] Cross-attention is used to integrate the semantic information of defects from the binary segmentation branch and the semantic segmentation branch, to build potential interactions between the two branches and capture cross-domain information interactions. The defect features in the binary segmentation are used to increase the inter-class distance between defects of different categories in the semantic segmentation, and to obtain enhanced semantic segmentation defect features.

[0076] Training module: The original labels and binary labels of the training set images are used as supervisory information to train a dual-branch distinguishable feature cross-attention network to obtain the optimal dual-branch distinguishable feature cross-attention network model;

[0077] Segmentation module: The test image to be segmented is input into the obtained optimal dual-branch distinguishable feature cross-attention network model, and the semantic segmentation result image of the test image is obtained through the semantic segmentation branch in the model.

[0078] Beneficial Effects

[0079] Compared with the existing method, the advantages of the present invention are:

[0080] The dual-branch distinguishable feature cross-attention network in the technical solution of the present invention can simultaneously and simply and efficiently complete the tasks of binary segmentation and semantic segmentation; the network uses the MSASPP module to capture more fine-grained multi-scale features and enhance the receptive field; the network introduces a dual-branch decoder, uses the binary segmentation branch to guide the semantic segmentation branch, and extracts finer defect boundaries; the network integrates the defect semantic information from the binary segmentation branch and the semantic segmentation branch through distinguishable feature cross-attention, constructs potential interactions between the two branches and captures cross-domain information interactions, and uses the defect features in the binary segmentation to increase the inter-class distance between defects of different categories in the semantic segmentation;

[0081] The technical solution of the present invention also uses a joint loss to train a dual-branch distinguishable feature cross-attention network, the loss used by the semantic segmentation branch is a cross entropy loss function, and the loss used by the binary segmentation branch is a binary classification cross entropy loss function.

[0082] The details are as follows:

[0083] 1. Compared with the single-branch semantic segmentation network, benefiting from the guidance mechanism of the dual-branch decoder, the dual-branch distinguishable feature cross-attention network proposed in the technical solution of the present invention can effectively use the binary segmentation branch to guide the semantic segmentation branch, extract finer defect boundaries, and solve the problem of blurred surface defect boundaries. In addition, compared with the two-stage semantic segmentation network, the two branches of the dual-branch distinguishable feature cross-attention network run in parallel, and the segmentation efficiency is higher.

[0084] 2. Introducing block-level multi-scale into the ASPP module to capture more fine-grained multi-scale features, thereby enhancing the receptive field, which is conducive to solving the problem of multi-scale changes in surface defects.

[0085] 3. A distinguishable feature cross-attention module is proposed. Cross-attention is used to integrate the defect semantic information from the binary segmentation branch and the semantic segmentation branch, construct potential interactions between the two branches and capture cross-domain information interactions. The defect features in binary segmentation are used to increase the inter-class distance between defects of different categories in semantic segmentation, which is conducive to the accurate classification of surface defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a schematic diagram of the architecture of the surface defect segmentation method based on dual-branch distinguishable features described in an embodiment of the present invention;

[0087] Figure 2 They are schematic diagrams of the structures of the original ASPP module and the MSASPP module in the embodiment of the present invention respectively;

[0088] Figure 3 Schematic diagram of a distinguishable feature cross-attention module in an embodiment of the present invention;

[0089] Figure 4 It is a schematic diagram comparing the evaluation indicators of four semantic segmentation of the method of the present invention on the pinewood dataset;

[0090] Figure 5 It is a schematic diagram comparing the evaluation indicators of four semantic segmentation of the method of the present invention on the rubber wood dataset;

[0091] Figure 6 This is a schematic diagram of the visualization results of the method of the present invention on the pine wood dataset;

[0092] Figure 7 This is a schematic diagram of the visualization results of the method of the present invention on a rubber wood dataset;

[0093] Figure 8 It is a schematic diagram of the comparison of the evaluation indicators of three semantic segmentation of the method of the present invention on the road crack and steel data sets, where NEU-Seg is the steel data set and Crack500 is the road crack data set;

[0094] Fig. 9 Schematic diagram of the visualization results of the method of the present invention on the road cracking and steel datasets, (a) is the input image, (b) is the label, (c) is U-Net, (d) is DeepLabV3+, (e) SegFormer, (f) is HRNet, (g) is HRFormer, (h) is Swin-T, and (i) is the present invention; in the first and second rows, red indicates crack defects from the road cracking dataset; in the third row, red indicates inclusion defects in the steel dataset, and in the fourth row, green indicates plaque defects in the steel dataset. DETAILED DESCRIPTION

[0095] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0096] Example 1

[0097] The embodiment of the present invention provides a surface defect segmentation method based on dual-branch distinguishable features, the architecture adopted is as follows Figure 1 shown in Figure 1 In the paper, the Swin Transformer series is used as the backbone network to extract image features; MSASPP represents the introduction of the block-level multi-scale ASPP module.

[0098] The specific process is as follows:

[0099] Step 1: Construct a dual-branch distinguishable feature cross-attention network;

[0100] The dual-branch distinguishable feature cross-attention network includes a backbone network for extracting visual information, an MSASPP module, and a parallel dual-branch decoder network connected in series;

[0101] The MSASPP module is obtained by introducing block-level multi-scale into the ASPP module, such as Figure 2 As shown;

[0102] The general steps of the ASPP module are: input image features, perform parallel dilated convolution layers with different expansion rates, global average pooling layers, feature splicing and fusion, 1×1 convolution, and obtain output image features; and the newly introduced block-level multi-scale is set between adjacent dilated convolution layers with different expansion rates;

[0103] The parallel dual-branch decoder network includes a distinguishable feature cross-attention module, a binary segmentation branch decoder network, and a semantic segmentation branch decoder network;

[0104] The distinguishable feature cross-attention module is obtained by introducing the binary segmentation branch input feature and the semantic segmentation branch input feature to the input of the cross-attention module, and introducing the semantic segmentation branch input feature to the output of the cross-attention module, such as Figure 3 As shown, the distinguishable feature cross-attention module is located in the middle of the binary segmentation branch and the semantic segmentation branch that are parallel to each other;

[0105] The general steps of the cross-attention module are: calculate the similarity between the query vector Q and the key vector K, dynamically generate the weight coefficient of the numerical vector V, perform weighted summation on V, and output the final attention value; the newly added introduction of the binary segmentation branch input features and the semantic segmentation branch input features to the input of the cross-attention module is set before calculating the similarity between Q and K; and the newly added introduction of the semantic segmentation branch input features to the output of the cross-attention module is set after outputting the final attention value;

[0106] The binary segmentation branch decoder network first restores the feature map size by gradually upsampling, then introduces a skip connection to fuse and convolve the backbone network feature map and the binary segmentation branch decoder feature map of the same resolution, extracts and converts features, and finally generates segmentation result maps of defects and non-defects;

[0107] The semantic segmentation branch decoder network is obtained by gradually upsampling to restore the feature map size, introducing jump connections to fuse the binary segmentation branch decoder feature map and the semantic segmentation branch decoder feature map of the same resolution, and performing convolution operations to further extract and transform features, ultimately generating segmentation result maps of multiple different categories of defects.

[0108] Taking Swin Transformer tiny as an example, the input known training set image size is preprocessed to 512×512, and five feature layers of different scales are extracted from the backbone network, including shallow layer, shallow layer, middle layer, sub-deep layer and deep layer, with scales of 256×256, 128×128, 64×64, 32×32 and 16×16 respectively.

[0109] The working process of the MSASPP module is:

[0110] The parallel feature blocks in the ASPP module are combined in series and in parallel to form a block-level multi-scale MSASPP module, and the MSASPP module is used to replace the ASPP module in the dual-branch distinguishable feature cross-attention network; the working process of the MSASPP module is as follows:

[0111] Step A1: input image features;

[0112] The final deep image feature x∈R with the most abstract semantic information extracted from the backbone network C×H×W Input to the MSASPP module, where R represents a multidimensional tensor, C, H, and W represent the number of channels, height, and width of the deep image features, respectively. In this example, the values ​​of C, H, and W are 768, 16, and 16, respectively;

[0113] Step A2: 1×1 convolution;

[0114] Perform 1×1 convolution on the deep image feature x to achieve feature channel dimensionality reduction and obtain the output feature map f1∈R of the first operation 256×H×W ;

[0115] f1=σ(Conv 1×1 (x)

[0116] In the formula, Conv 1×1 represents 1×1 convolution operation, σ represents BN and ReLU activation operations;

[0117] Step A3: parallel dilated convolutions and introduce block-level multi-scale between adjacent convolution operations;

[0118] The output feature map f1 obtained in step A2 is concatenated with the deep image feature x through channels, and then a 3×3 convolution with a dilation rate of 6 is performed to obtain the output feature map f2∈R 256×H×W ; Concatenate the output feature map f2 with the deep image feature x, and then perform a 3×3 convolution with a dilation rate of 12 to obtain the output feature map f3∈R of the third operation 256×H×W ; Concatenate the output feature map f3 with the input image feature x, and then perform a 3×3 convolution with a dilation rate of 18 to obtain the output feature map f4∈R of the fourth operation 256×H×W ;

[0119]

[0120] In the formula, DConv 3×3,d represents a 3×3 dilated convolution with a dilation rate of d, represents the channel splicing operation, d = (n-1)*6;

[0121] Step A4: global average pooling;

[0122] Perform a global average pooling operation on the deep image feature x, and then perform 1×1 convolution, BN, ReLU activation and upsampling operations in sequence to obtain the output feature map f5∈R of the fifth operation 256×H×W ;

[0123] f5=up(σ(Conv 1×1 (pool1×1 (x))))

[0124] In the formula, pool 1×1 represents global average pooling, and up represents upsampling;

[0125] Step A5: feature splicing and fusion;

[0126] The output feature maps f1, f2, f3, f4 and f5 of the above steps are concatenated and fused in the channel dimension to obtain the fused feature map f out ∈R B×1280×H×W ;

[0127]

[0128] Step A6: output image features;

[0129] For the fusion feature map f out Perform 1×1 convolution, BN and ReLU activation operations in sequence to obtain the final output image feature f out ∈R 256×H×W ;

[0130] f out =σ(Conv 1×1 (f out )).

[0131] The serial structure means that the output results of the upper-level branches are sequentially used as the input of the next-level branches until the end of the last level of branches. The parallel structure means that each level of branches is processed in parallel and the final output results are summarized. The block-level multi-scale idea is used to improve the original ASPP module to capture finer-grained multi-scale features, enhance the receptive field, and solve the multi-scale variation problem of surface defects.

[0132] The working process of the distinguishable cross attention module is:

[0133] Step B1: Input features for Patch Embedding processing;

[0134] The input feature X of the binary segmentation branch decoder network bi ∈R C×H×W and the input features X of the semantic segmentation branch decoder network se ∈R C×H×W Patch Embedding is performed separately, where C, H, and W represent the batch size, number of channels, height, and width, respectively. The values ​​of C, H, and W are 256, 32, and 32, respectively.

[0135] The specific process of performing Patch Embedding processing on the input features is as follows:

[0136] The input features are first modified to C by a convolution operation with a kernel size and a kernel step size of patch_size. emb , H e and W e ,get and Among them, C emb is the length of the embedding vector, H e =H / patch_size, W e =W / patch_size, where patch_size represents the patch size. Here, patch_size is set to 16. Next, set X bi and X se Flattening the last two dimensions of and Then X bi and X se By exchanging the two dimensions of and Where E=H e ×W e . C emb In the embodiment of the present invention, the value is 256;

[0137] Feature X from the decoder layer of the binary segmentation branch bi and the feature X in the decoder network layer of the semantic segmentation branch se , after Patch Embedding processing, it is effectively converted into an embedding vector of fixed size;

[0138] Step B2: QKV linear transformation;

[0139] The input feature X of the semantic segmentation branch decoder network se , generate query vector through linear transformation and dimensional transformation Input feature X of the binary segmentation branch decoder network bi Used to generate key vector K and numerical vector V, X bi Generated by linear transformation X bi Generated by linear transformation and dimensional transformation In order to achieve the goal of enhancing the semantic segmentation effect by utilizing binary segmentation information; where N represents the number of attention heads, R represents a multidimensional tensor, and C emb represents the length of the embedding vector, E represents the intermediate variable, E=H e ×W e ;

[0140] In the embodiment of the present invention, N is set to 16;

[0141] Step B3: multi-head cross attention calculation;

[0142] Cross-attention is defined as:

[0143]

[0144] Among them, "*" is the matrix multiplication operation, and "+" is the matrix addition operation; It indicates that the attention feature map is obtained after multi-head cross attention calculation;

[0145] Step B4: MLP feature extraction and conversion;

[0146] Transform the dimension of Att(Q,K,V) obtained in step B3 to obtain the feature map Attention∈ The feature map is used as the input of the MLP, and then the output feature is obtained by applying serial convolution, activation and drop operations as follows

[0147] y=Drop(GELU(Conv1(Attention)))

[0148] z=BatchNorm2d(GELU(Conv2(y)+Attention))

[0149] output = Drop(Conv3(z))

[0150] Among them, Conv1 and Conv3 represent convolution operations with a convolution kernel size of 1×1. The former is used to adjust the dimension of the feature, and the latter is used to restore the dimension of the feature; GELU represents the Gaussian error linear unit activation function; Conv2 represents a convolution operation with a convolution kernel size of 3×3, which is mainly used for feature extraction; Drop represents the DropPath regularization operation, y and z both represent intermediate variables, and the probability value of randomly deleting the sub-path of the multi-branch structure of the model is set to 0.5.

[0151] The binary segmentation branch decoder network works as follows:

[0152] The output feature block of the ASPP module is convolved and double up-sampled to obtain the feature block b1; next, the feature block generated in the fourth stage of the backbone network is merged with b1 to complete the convolution operation to obtain the distinguishable feature cross attention input X bi , after the distinguishable feature cross attention module, the output Y is obtained bi ; Y biAfter double upsampling, the feature block b2 is merged with the feature block generated in the third stage of the backbone network to complete the convolution operation to obtain the feature block b2, and the feature block b2 is transmitted to the feature layer of the semantic segmentation branch decoder network with the same resolution; the feature block b2 is double upsampled and merged with the feature block generated in the second stage of the backbone network to complete the convolution operation to obtain the feature block b3, and the feature block b3 is transmitted to the feature layer of the semantic segmentation branch decoder network with the same resolution; the feature block b3 is double upsampled to obtain the feature block b4, and then the feature block b4 is convolutionally and double upsampled to complete the binary segmentation prediction result with the same resolution as the input image.

[0153] Binary segmentation mainly focuses on the distinction between defects and non-defects, and can therefore provide stronger prior information such as defect activation and pre-positioning of location information for semantic segmentation, improving the accuracy of semantic segmentation.

[0154] The backbone network can adopt any backbone network of the ResNet series, the Swin-Transformer series and the MobileNet series; the ResNet series network gradually reduces the feature map size through the convolution operation in the residual block combined with the convolution with a step size of 2, from the original input to 1 / 32 resolution, with a total of five stages, and the feature map size is halved at the end of each stage; the Swin-Transformer series network uses a shift window mechanism and a self-attention mechanism to capture local and global information, and goes through five stages, each time the spatial resolution is halved through Patch Merging, and finally reaches 1 / 32 of the input resolution; the MobileNet series networks all use downsampling operations in specific stages to reduce the spatial dimension of the feature map with a step size of 2, and go through five stages, and the feature map size is halved at the end of each stage.

[0155] The semantic segmentation branch decoder network works as follows:

[0156] The output feature block of the ASPP module is convolved and double up-sampled to obtain data s1; then the data s1 is convolved to obtain the distinguishable feature cross-attention input X se , get the distinguishable feature cross attention module to get the output Y se ; Y se After double upsampling, the data is merged with the feature block b2 generated by the binary segmentation branch decoder network to complete the convolution operation and obtain data s2; after double upsampling, the data s2 is merged with the feature block b3 generated by the binary segmentation branch decoder network to complete the convolution operation and obtain data s3; the data s3 feature is double upsampled to obtain data s4, and the semantic segmentation prediction result with the same resolution as the input image is completed through convolution operation and double upsampling.

[0157] Binary segmentation mainly focuses on the distinction between defects and non-defects, so it can provide stronger defect activation and location information pre-positioning for semantic segmentation. Semantic segmentation can fully extract these discriminative features of defect categories by obtaining prior information of binary segmentation, enhance the separability between classes through the generated regions and targets, reduce misclassification, and alleviate the overfitting problem to a certain extent.

[0158] Step 2: Use the original labels of the training set images to perform binary processing on the training set images, that is, uniformly modify the pixel values ​​of all defective areas in the original labels to "1" to obtain the binary labels corresponding to the training set images;

[0159] Step 3: Use the original labels and binary labels of the training set images as supervision information at the same time, train the dual-branch distinguishable feature cross-attention network, and obtain the optimal dual-branch distinguishable feature cross-attention network model;

[0160] The training process of the two-branch distinguishable feature cross-attention network is as follows:

[0161] Taking the known training set images as input, the original labels and binary labels of the training set images obtained in step 2 as supervision information, setting the joint loss function, training the dual-branch distinguishable feature cross-attention network, and obtaining the trained dual-branch distinguishable feature cross-attention network model.

[0162] The joint loss function L total It is composed of the cross entropy loss function of the semantic segmentation branch and the binary cross entropy loss function of the binary segmentation branch. The formula is:

[0163] L total =L seg +L b

[0164] Among them, L seg Represents the cross entropy loss function:

[0165]

[0166] Among them, y i and p i They represent the label and predicted probability of the i-th image, n represents the batch size, and k represents the number of defect categories;

[0167] L b Represents the binary cross entropy loss function:

[0168]

[0169] Step 4: Input the test image to be segmented into the optimal dual-branch distinguishable feature cross-attention network model obtained in step 3, and obtain the semantic segmentation result image of the test image through the semantic segmentation branch in the model.

[0170] like Figure 4 , 5 As shown, the method described in the embodiment of the present invention achieved the best results in two semantic segmentation evaluation indicators mIoU and mACC on the pine wood dataset and the rubber wood dataset, and the segmentation accuracy exceeded the existing methods, verifying the effectiveness of the technical solution of the present invention in the surface defect segmentation task.

[0171] like Figure 6 , 7 As shown, the visualization results of the method described in the embodiment of the present invention on two data sets, a pine wood data set and a rubber wood data set, the quality of the prediction mask and classification of the present invention exceeds the existing methods, which verifies the effectiveness of the present invention.

[0172] like Figure 8 As shown, a schematic diagram comparing the evaluation indicators of three semantic segmentation of the method described in the embodiment of the present invention on two datasets, road cracking and steel datasets, is shown. The effectiveness of the technical solution of the present invention in various surface defect segmentation tasks verifies the effectiveness of the present invention.

[0173] like Fig. 9 As shown, the visualization results of the method described in the embodiment of the present invention on two datasets, road cracking and steel dataset, the quality of the prediction mask and classification of the present invention exceeds the existing methods, which verifies the effectiveness of the present invention.

[0174] Example 2

[0175] A surface defect segmentation system with dual-branch distinguishable features based on the above method comprises:

[0176] Network construction module: used to construct a dual-branch distinguishable feature cross-attention network, wherein the dual-branch distinguishable feature cross-attention network includes a backbone network, an ASPP module, and a parallel dual-branch decoder network connected in series in sequence;

[0177] Cross-attention is used to integrate the semantic information of defects from the binary segmentation branch and the semantic segmentation branch, to build potential interactions between the two branches and capture cross-domain information interactions. The defect features in the binary segmentation are used to increase the inter-class distance between defects of different categories in the semantic segmentation, and to obtain enhanced semantic segmentation defect features.

[0178] Training module: The original labels and binary labels of the training set images are used as supervisory information to train a dual-branch distinguishable feature cross-attention network to obtain the optimal dual-branch distinguishable feature cross-attention network model;

[0179] Segmentation module: The test image to be segmented is input into the obtained optimal dual-branch distinguishable feature cross-attention network model, and the semantic segmentation result image of the test image is obtained through the semantic segmentation branch in the model.

[0180] It should be understood that the implementation process of each module can refer to the content of the aforementioned method. The division of the above functional modules is only a division of logical functions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated units can be implemented in the form of hardware or software functional units.

[0181] Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk, etc. Various media that can store program codes.

[0182] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A surface defect segmentation method based on dual-branch distinguishable features, characterized in that: include: Step 1: Construct a dual-branch distinguishable feature cross-attention network; The dual-branch distinguishable feature cross-attention network includes a backbone network for extracting visual information, an ASPP module, and a parallel dual-branch decoder network connected in series; The parallel dual-branch decoder network includes a distinguishable feature cross-attention module, a binary segmentation branch decoder network and a semantic segmentation branch decoder network, the binary segmentation branch decoder network and the semantic segmentation branch decoder network run in parallel, and the distinguishable feature cross-attention module is inserted into the middle part of the parallel binary segmentation branch decoder network and the semantic segmentation branch decoder network to link the binary segmentation branch decoder network and the semantic segmentation branch decoder network; Step 2: Use the original labels of the training set images to perform binary processing on the training set images, that is, uniformly modify the pixel values ​​of all defective areas in the original labels to "1" to obtain the binary labels corresponding to the training set images; Step 3: Use the original labels and binary labels of the training set images as supervision information at the same time, train the dual-branch distinguishable feature cross-attention network, and obtain the optimal dual-branch distinguishable feature cross-attention network model; Step 4: Input the test image to be segmented into the optimal dual-branch distinguishable feature cross-attention network model obtained in step 3 to obtain the semantic segmentation result image of the test image.

2. The method according to claim 1, characterized in that The distinguishable feature cross attention module sequentially performs Patch Embedding processing, QKV linear transformation, multi-head cross attention calculation, and MLP feature extraction and conversion; The binary segmentation branch decoder network first restores the feature map size by gradually upsampling, then introduces a skip connection to fuse and convolve the backbone network feature map and the binary segmentation branch decoder feature map of the same resolution, extracts and converts features, and finally generates segmentation result maps of defects and non-defects; The semantic segmentation branch decoder network is obtained by gradually upsampling to restore the feature map size, introducing jump connections to fuse the binary segmentation branch decoder feature map and the semantic segmentation branch decoder feature map of the same resolution, and performing convolution operations to further extract and transform features, ultimately generating segmentation result maps of multiple different categories of defects.

3. The method according to claim 2, characterized in that The working process of the distinguishable cross attention module is: Step B1: Input features for Patch Embedding processing; The input feature X of the binary segmentation branch decoder network bi ∈R C×H×W and the input features X of the semantic segmentation branch decoder network se ∈R C×H×W Patch Embedding is performed separately, where C, H, and W represent the batch size, number of channels, height, and width, respectively, and R represents a multidimensional tensor; Step B2: QKV linear transformation; The input feature X of the semantic segmentation branch decoder network se , generate query vector through linear transformation and dimensional transformation Input feature X of the binary segmentation branch decoder network bi Used to generate key vector K and numerical vector V, X bi Generated by linear transformation X bi Generated by linear transformation and dimensional transformation In order to achieve the goal of enhancing the semantic segmentation effect by utilizing binary segmentation information; where N represents the number of attention heads, R represents a multidimensional tensor, and C emb represents the length of the embedding vector, E represents the intermediate variable, E=H e ×W e , H e =H / patch_size, W e =W / patch_size, patch_size indicates the patch size, and its value is set based on experience; Step B3: multi-head cross attention calculation; Cross-attention is defined as: Among them, "*" is a matrix multiplication operation, and "+" is a matrix addition operation; It indicates that the attention feature map is obtained after multi-head cross attention calculation; Step B4: MLP feature extraction and conversion; Transform the dimension of Att(Q,K,V) obtained in step B3 to obtain the feature map The feature map is used as the input of the MLP, and then the output feature is obtained by applying serial convolution, activation and drop operations as follows y=Drop(GELU(Conv1(Attention))) z=BatchNorm2d(GELU(Conv2(y)+Attention)) output = Drop(Conv3(z)) Among them, Conv1 and Conv3 represent convolution operations with a convolution kernel size of 1×1. The former is used to adjust the dimension of the feature, and the latter is used to restore the dimension of the feature; GELU represents the Gaussian error linear unit activation function; Conv2 represents a convolution operation with a convolution kernel size of 3×3, which is used for feature extraction; Drop represents the DropPath regularization operation, and y and z both represent intermediate variables.

4. The method according to claim 3, characterized in that The specific process of performing Patch Embedding processing on the input features is as follows: The input features are first convolved with a kernel size and a step size of patch_size, and the number of channels, height, and width of the image are modified to C respectively. emb , H e and W e ,get and Next, X bi and X se Flatten the last two dimensions of and Then X bi and X se By exchanging the two dimensions of and 5. The method according to claim 1, characterized in that The training process of the dual-branch distinguishable feature cross-attention network is as follows: Taking the known training set images as input, the original labels of the training set images are used to perform binarization processing, that is, the pixel values ​​of all defect areas in the original labels are uniformly modified to "1", and the binary labels corresponding to the training set images are obtained. The original labels and binary labels of the training set images are used as supervision information, and a joint loss function is set to train a dual-branch distinguishable feature cross-attention network, and a trained dual-branch distinguishable feature cross-attention network model is obtained.

6. The method according to claim 5, characterized in that The joint loss function L total It is composed of the binary classification cross entropy function of the binary segmentation branch decoder network of the parallel two-branch decoder network and the cross entropy loss function of the semantic segmentation branch decoder network, as follows: L total =L seg +L b Among them, L seg Represents the cross entropy loss function of the semantic segmentation branch decoder network: L b Represents the binary cross entropy function of the binary segmentation branch decoder network: Among them, y i and p i They represent the label and predicted probability of the i-th image, n represents the batch size, and k represents the number of defect categories.

7. The method according to claim 2, characterized in that The parallel feature blocks in the ASPP module are combined in series and in parallel to form a block-level multi-scale MSASPP module, and the MSASPP module is used to replace the ASPP module in the dual-branch distinguishable feature cross-attention network; the working process of the MSASPP module is as follows: Step A1: input image features; The final deep image feature x∈R with the most abstract semantic information extracted from the backbone network C×H×W Input to the MSASPP module, where R represents the real number domain, C, H, and W represent the number of channels, height, and width of the deep image features, respectively; Step A2: 1×1 convolution; Perform 1×1 convolution on the deep image feature x to achieve feature channel dimensionality reduction and obtain the output feature map f1∈R of the first operation 256×H×W ; f1=σ(Conv 1×1 (x)) In the formula, Conv 1×1 represents 1×1 convolution operation, σ represents BN and ReLU activation operations; Step A3: parallel dilated convolutions and introduce block-level multi-scale between adjacent convolution operations; The output feature map f1 obtained in step A2 is concatenated with the deep image feature x through channels, and then a 3×3 convolution with a dilation rate of 6 is performed to obtain the output feature map f2∈R 256×H×W ; Concatenate the output feature map f2 with the deep image feature x, and then perform a 3×3 convolution with a dilation rate of 12 to obtain the output feature map f3∈R of the third operation 256 ×H×W ; Concatenate the output feature map f3 with the input image feature x, and then perform a 3×3 convolution with a dilation rate of 18 to obtain the output feature map f4∈R of the fourth operation 256×H×W ; In the formula, DConv 3×3,d represents a 3×3 dilated convolution with a dilation rate of d, represents the channel splicing operation, d = (n-1)*6; Step A4: global average pooling; Perform a global average pooling operation on the deep image feature x, and then perform 1×1 convolution, BN, ReLU activation and upsampling operations in sequence to obtain the output feature map f5∈R of the fifth operation 256×H×W ; f5=up(σ(Conv 1×1 (pool 1×1 (x)))) In the formula, pool 1×1 represents global average pooling, and up represents upsampling; Step A5: feature splicing and fusion; The output feature maps f1, f2, f3, f4 and f5 of the above steps are concatenated and fused in the channel dimension to obtain the fused feature map f out ∈R B×1280×H×W ; Step A6: output image features; For the fusion feature map f out Perform 1×1 convolution, BN and ReLU activation operations in sequence to obtain the final output image feature f out ∈R 256×H×W ; f out =σ(Conv 1×1 (f out ))。 8. The method according to claim 2, characterized in that: The binary segmentation branch decoder network works as follows: The output feature block of the ASPP module is convolved and double up-sampled to obtain the feature block b1; next, the feature block generated in the fourth stage of the backbone network is merged with b1 to complete the convolution operation to obtain the distinguishable feature cross attention input X bi , after the distinguishable feature cross attention module, the output Y is obtained bi ; Y bi After double upsampling, the feature block b2 is merged with the feature block generated in the third stage of the backbone network to complete the convolution operation to obtain the feature block b2, and the feature block b2 is transmitted to the feature layer of the same resolution in the semantic segmentation branch decoder network; after double upsampling, the feature block b2 is merged with the feature block generated in the second stage of the backbone network to complete the convolution operation to obtain the feature block b3, and the feature block b3 is transmitted to the feature layer of the same resolution in the semantic segmentation branch decoder network; Feature block b3 is upsampled twice to obtain feature block b4, and then feature block b4 is convolved and upsampled twice to complete the binary segmentation prediction result with the same resolution as the input image.

9. The method according to claim 8, characterized in that The semantic segmentation branch decoder network works as follows: The output feature block of the ASPP module is convolved and double up-sampled to obtain data s1; then the data s1 is convolved to obtain the distinguishable feature cross-attention input X se , get the distinguishable feature cross attention module to get the output Y se ; Y se After double upsampling, the data is merged with the feature block b2 generated by the binary segmentation branch decoder network to complete the convolution operation and obtain data s2; after double upsampling, the data s2 is merged with the feature block b3 generated by the binary segmentation branch decoder network to complete the convolution operation and obtain data s3; the data s3 feature is double upsampled to obtain data s4, and the semantic segmentation prediction result with the same resolution as the input image is completed through convolution operation and double upsampling.

10. A surface defect segmentation system with dual-branch distinguishable features based on the method according to any one of claims 1 to 9, characterized in that: include: Network construction module: used to construct a dual-branch distinguishable feature cross-attention network, wherein the dual-branch distinguishable feature cross-attention network includes a backbone network, an ASPP module, and a parallel dual-branch decoder network connected in series in sequence; Cross-attention is used to integrate the semantic information of defects from the binary segmentation branch and the semantic segmentation branch, to build potential interactions between the two branches and capture cross-domain information interactions. The defect features in the binary segmentation are used to increase the inter-class distance between defects of different categories in the semantic segmentation, and to obtain enhanced semantic segmentation defect features. Training module: The original labels and binary labels of the training set images are used as supervisory information to train a dual-branch distinguishable feature cross-attention network to obtain the optimal dual-branch distinguishable feature cross-attention network model; Segmentation module: The test image to be segmented is input into the obtained optimal dual-branch distinguishable feature cross-attention network model, and the semantic segmentation result image of the test image is obtained through the semantic segmentation branch in the model.

Images