Two-stage coding and multi-decoding strip steel surface defect segmentation method

Abstract

The invention provides a two-stage coding and multi-decoding strip steel surface defect segmentation method, which belongs to the technical field of image processing, and comprises the following steps: carrying out segmentation by adopting a pre-trained two-stage coding and multi-decoding model; the model comprises an initial defect detection stage and a class-based fine stage; the initial defect detection stage is used for coding the initial segmentation image by adopting a U-shaped coding and decoding structure and carrying out up-sampling by adopting a global and local up-sampling module; and the class-based fine stage is used for carrying out separation according to classes by adopting a class separation module, carrying out coding and decoding again by adopting a specific defect fine module, and carrying out fusion by adopting a prior information fusion module to obtain a segmentation result. The defect detection accuracy under the low contrast condition is improved, and the segmentation performance on the tail category under the category imbalance condition is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a two-stage encoding and multi-decoding strip surface defect segmentation method. Background Art

[0002] Hot-rolled strip is an indispensable product in the steel industry and is widely used in daily life. However, in the actual production process of hot-rolled strip, defects may appear on the surface due to various factors. The defect edge shows a high noise ratio, low clarity and low contrast between the defect and the background. In addition, in actual strip production, long-tail distribution problems are often encountered. This distribution problem greatly reduces the detection accuracy of the tail category, which in turn significantly affects the qualified rate of strip products. Solving these challenges can not only improve product quality and reliability, but also effective defect detection can help reduce material waste, optimize production efficiency and reduce environmental impact, thereby supporting the sustainable development of the steel manufacturing industry.

[0003] Traditional defect segmentation methods, such as threshold algorithms and clustering algorithms, achieve segmentation by classifying each pixel in the image. However, these methods require determining the optimal threshold during algorithm design, which is challenging and results in significant differences between the upper and lower limits. Traditional defect segmentation methods have low accuracy in identifying defects in environments with high noise ratios, low defect edge clarity, and low defect-background contrast.

[0004] Although current deep learning-based methods have performed well in defect localization, they may incorrectly segment the background as a defect or miss the defect completely when the contrast between the defect and the background is low, which limits the accuracy of defect segmentation. In addition, previous methods usually conduct simulation experiments on balanced datasets, while defects in actual industrial production often have a long-tail distribution. There is a significant difference in segmentation performance between the head and tail categories. Summary of the invention

[0005] In view of the shortcomings of the prior art, this application proposes a two-stage encoding and multi-decoding method for strip surface defect segmentation. This application aims to integrate the actual conditions in the production environment to improve the defect detection accuracy under low-contrast conditions and improve the segmentation performance of tail categories in the case of class imbalance.

[0006] In the first aspect, the present application proposes a two-stage encoding and multi-decoding strip surface defect segmentation method, comprising:

[0007] Obtain a picture of the hot-rolled steel strip to be inspected;

[0008] The hot-rolled strip image to be detected is input into a pre-trained two-stage encoding and multi-decoding model to obtain a strip surface defect segmentation result; the pre-trained two-stage encoding and multi-decoding model includes: an initial defect detection stage and a class-based refinement stage; the initial defect detection stage is used to encode the hot-rolled strip image to be detected using a U-type encoding and decoding structure to obtain a first encoding feature map, and the first encoding feature map is up-sampled using a global local upsampling module to obtain an initial defect segmentation map; the class-based refinement stage is used to separate the initial defect segmentation map by category using a category separation module to obtain a specific category feature map, and the specific category feature map is encoded again using a specific defect refinement module to obtain a second encoding feature map, and the second encoding feature map is decoded using a frozen parameter GT decoder to obtain a first decoding feature map, and the second encoding feature map is decoded using an unfrozen parameter GT decoder to obtain a second decoding feature map, and the first decoding feature map is fused with the second decoding feature map using a priori information fusion module to obtain a strip surface defect segmentation result.

[0009] The training process of the GT decoder with frozen parameters includes: using four depth-separable convolutions to encode the truth map of the historical hot-rolled steel strip, each depth-separable convolution reduces the size of the truth map of the historical hot-rolled steel strip by two times, and increases the number of channels by two times; using four global local upsampling modules to decode the encoding result, each global local upsampling module enlarges the size of the truth map of the historical hot-rolled steel strip by two times, and reduces the number of channels by two times; and using the GT decoder after the decoding parameters are frozen as the GT decoder with frozen parameters;

[0010] The GT decoder without freezing the decoding parameters uses four global-local upsampling modules to implement decoding.

[0011] The initial defect detection stage includes: a first coding layer, a second coding layer, a third coding layer, a fourth coding layer, a fifth coding layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first global local upsampling module, a second global local upsampling module, a third global local upsampling module and a fourth global local upsampling module;

[0012] The last four layers of the pre-trained ResNet34 are used as the first, second, third, and fourth coding layers respectively; a 3×3 depthwise separable convolution and a batch normalization layer are used as the fifth coding layer;

[0013] The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer are all 1×1 convolutions;

[0014] Input the hot-rolled steel strip image to be detected into the first coding layer, input the output of the first coding layer into the second coding layer and the first convolution layer, and obtain the first abstract feature map; input the output of the second coding layer into the third coding layer and the second convolution layer, and obtain the second abstract feature map; input the output of the third coding layer into the fourth coding layer and the third convolution layer, and obtain the third abstract feature map; input the output of the fourth coding layer into the fifth coding layer and the fourth convolution layer, and obtain the fourth abstract feature map; use the output of the fifth coding layer as the fifth abstract feature map; use the first abstract feature map, the second abstract feature map, the third abstract feature map, the fourth abstract feature map, and the fifth abstract feature map as the first coding feature map;

[0015] The fifth abstract feature map is input into the fourth global local upsampling module, the output of the fourth global local upsampling module and the sum of the fourth abstract feature map are input into the third global local upsampling module, the output of the third global local upsampling module and the sum of the third abstract feature map are input into the second global local upsampling module, the output of the second global local upsampling module and the sum of the second abstract feature map are input into the first global local upsampling module, and the output of the first global local upsampling module and the sum of the first abstract feature map are used as the initial defect segmentation map.

[0016] The first coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 64, and the channel size remains unchanged; the second coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 128, and the channel size is reduced to 112; the third coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 256, and the channel size is reduced to 56; the fourth coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 512, and the channel size is reduced to 28; the fifth coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 1024, and the channel size is reduced to 14.

[0017] The global local upsampling module includes: a first deconvolution, a second deconvolution, a third deconvolution, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer and a ninth convolution layer; the first deconvolution, the second deconvolution and the third deconvolution have different expansion rates; the sixth convolution layer, the seventh convolution layer and the eighth convolution layer are all residual convolutions;

[0018] Performing upsampling on the feature map to be upsampled by the first deconvolution, the second deconvolution, and the third deconvolution, respectively, to obtain a first upsampling result, a second upsampling result, and a third upsampling result, respectively;

[0019] Respectively adding the first upsampling result, the second upsampling result and the third upsampling result in pairs to obtain a first addition result, a second addition result and a third addition result;

[0020] Input the first addition result into the sixth convolution layer, input the second addition result into the seventh convolution layer, and input the third up-sampling result into the eighth convolution layer;

[0021] The output results of the sixth convolutional layer, the seventh convolutional layer, and the eighth convolutional layer are added, the sum is input to the ninth convolutional layer, and the output of the ninth convolutional layer is used as the upsampling result.

[0022] The category separation module includes: a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a mean vector, a first fully connected layer, a second fully connected layer, and a category index; the tenth convolutional layer, the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the mean vector, the first fully connected layer, the second fully connected layer, and the category index are connected in sequence;

[0023] The initial defect segmentation map is input into the tenth convolutional layer, and passes through the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the mean vector, the first fully connected layer, the second fully connected layer and the category index in sequence, and is separated according to specific defect categories to obtain a specific category feature map.

[0024] The specific defect refinement module includes: a fifteenth convolution layer, a sixteenth convolution layer, a seventeenth convolution layer, an eighteenth convolution layer, a fifth global local upsampling module, a sixth global local upsampling module, a seventh global local upsampling module, an eighth global local upsampling module and a priori information fusion module; the fifteenth convolution layer, the sixteenth convolution layer, the seventeenth convolution layer and the eighteenth convolution layer are depth-separable convolutions;

[0025] Input the specific category feature map into the fifteenth convolution layer for encoding, and input the output of the fifteenth convolution layer into the sixteenth convolution layer;

[0026] Input the output of the sixteenth convolutional layer to the seventeenth convolutional layer, input the output of the seventeenth convolutional layer to the eighteenth convolutional layer, input the output of the eighteenth convolutional layer to the eighth global local upsampling module, add the output of the eighth global local upsampling module to the output of the seventeenth convolutional layer, and input the addition result to the seventh global local upsampling module, add the output of the seventh global local upsampling module to the output of the sixteenth convolutional layer, and input the addition result to the sixth global local upsampling module, add the output of the sixth global local upsampling module to the output of the fifteenth convolutional layer, and input the addition result to the fifth global local upsampling module, and use the output of the fifth global local upsampling module as the first decoding feature map to input into the prior information fusion module;

[0027] The second decoded feature map is input into the prior information fusion module, and the output of the prior information fusion module is used as the strip surface defect segmentation result.

[0028] The prior information fusion module includes: a first channel attention unit, a second channel attention unit, a first spatial attention unit, a second spatial attention unit, a channel splicing unit and a fifth convolutional layer;

[0029] The first channel attention unit is used to generate a first channel attention unit weight, multiply the first decoded feature map by the first channel attention weight channel by channel to obtain a first multiplication result, and multiply the second decoded feature map by the first channel attention weight channel by channel to obtain a second multiplication result;

[0030] The second channel attention unit is used to generate a second channel attention weight, multiply the first decoded feature map by the second channel attention weight channel by channel to obtain a third multiplication result, and multiply the second decoded feature map by the second channel attention weight channel by channel to obtain a fourth multiplication result;

[0031] The first spatial attention unit is used to generate a first spatial attention weight, perform a corresponding element multiplication operation on the first multiplication result and the first spatial attention weight to obtain a fifth multiplication result, and perform a corresponding element multiplication operation on the third multiplication result and the first spatial attention weight to obtain a sixth multiplication result;

[0032] The second spatial attention unit is used to generate a second spatial attention weight, perform a corresponding element multiplication operation on the second multiplication result and the second spatial attention weight to obtain a seventh multiplication result, and perform a corresponding element multiplication operation on the fourth multiplication result and the second spatial attention weight to obtain an eighth multiplication result;

[0033] The channel splicing unit is used to splice the fifth multiplication result, the sixth multiplication result, the seventh multiplication result and the eighth multiplication result to obtain a splicing result;

[0034] The fifth convolutional layer is used to compress the splicing result to obtain the strip surface defect segmentation result.

[0035] In a second aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the two-stage encoding and multi-decoding strip surface defect segmentation method.

[0036] In a third aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to execute the two-stage encoding and multi-decoding method for strip surface defect segmentation.

[0037] Beneficial effects:

[0038] The present application proposes a two-stage encoding and multi-decoding strip surface defect segmentation method, which effectively improves the accuracy of defect detection, especially tail defects. In this method, all decoders use the global local upsampling module proposed in this application to upsample the feature map to obtain high-quality abstract feature maps. At the same time, the present application adopts a two-stage method to detect defects, and uses a category separation module to separate the feature map by category, and uses specific parameters in the second stage to refine the defect detection for each category. The strip defect detection method proposed in this application can not only detect strip surface defects under low contrast conditions, but also improve the accuracy of tail category defects when facing an unbalanced category distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Flow chart of the strip surface defect segmentation method with two-stage encoding and multiple decoding in the embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a strip surface defect segmentation method with two-stage encoding and multiple decoding according to an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of the global local upsampling module in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of the category separation module in an embodiment of the present application;

[0043] Figure 5 Schematic diagram of the structure of a specific defect fine module in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of a priori information fusion module in an embodiment of the present application;

[0045] Figure 7 Unbalanced defect distribution diagram of the embodiment of the present application;

[0046] Figure 8 Visual comparison diagram of the segmentation results of the embodiment of the present application. DETAILED DESCRIPTION

[0047] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0048] Embodiment 1:

[0049] This application proposes a two-stage encoding and multi-decoding strip surface defect segmentation method, such as Figure 1 , Figure 2 As shown, including:

[0050] Step S1: obtaining a picture of the hot-rolled steel strip to be inspected;

[0051] Step S2: input the hot-rolled strip image to be detected into a pre-trained two-stage encoding and multi-decoding model to obtain a strip surface defect segmentation result; the pre-trained two-stage encoding and multi-decoding model includes: an initial defect detection stage and a class-based refinement stage; the initial defect detection stage is used to encode the hot-rolled strip image to be detected using a U-type encoding and decoding structure to obtain a first encoding feature map, and upsample the first encoding feature map using a global local upsampling module to obtain an initial defect segmentation map; the class-based refinement stage is used to separate the initial defect segmentation map by category using a category separation module to obtain a specific category feature map, and use a specific defect refinement module to encode the specific category feature map again to obtain a second encoding feature map, and use a frozen parameter GT decoder to decode the second encoding feature map to obtain a first decoding feature map, and use an unfrozen parameter GT decoder to decode the second encoding feature map to obtain a second decoding feature map, and use a priori information fusion module to fuse the first decoding feature map with the second decoding feature map to obtain a strip surface defect segmentation result.

[0052] In this embodiment, it is first necessary to construct a GT self-encoder-decoder and a two-stage encoding and multi-decoding model (TSEMD model, Two-Stage Encoder Multi-Decoder). The TSEMD model includes two stages: the initial defect detection stage and the class-based refinement stage. In the initial defect stage, a U-type encoding and decoding structure is adopted, and the feature map is upsampled by the global local upsampling module (GLUM) during the upsampling process. In the class-based refinement stage, the feature map is first separated by the category separation module (CSM, Category Separation Module), and then the specific defect refinement module (SDRM, Specific Defect Refinement Module) is used to encode and decode the specific category feature map again, and finally the prior information fusion module (PIFM, Prior Information Fusion Module) is used to fuse the parameter-frozen GT decoder and the main decoder (i.e., the GT decoder with unfrozen parameters) to generate the final segmentation map.

[0053] This embodiment needs to apply the strip steel defect dataset from the real world and the industrial unbalanced dataset from the actual project to train the model. At the same time, the trained TSEMDNet model is tested using the above two datasets to verify its segmentation performance. The datasets used in this embodiment are SD-saliency-900 and the industrial unbalanced dataset X-SDD collected in the actual project, of which 60% are training sets and 40% are test sets. The training sequence of this embodiment is as follows: first, the defect self-encoder decoders of different categories are trained, and the parameters of the defect decoder are frozen for use in the second stage decoding. Then the TSEMDNet proposed in this article is trained. The experimental environment of this article is the PyTorch1.7.0 open source deep learning framework and GeForce RTX 3080GPU (10GB). During training, the batch size is set to 32, and the defect image size is set to 224*224, the number of training rounds is 100 rounds, the optimizer is Adam, the weight decay rate is 0.00001, the learning rate is 0.001, and the loss function is the binary cross entropy function (BCE Loss).

[0054] Detailed description of the implementation process:

[0055] (1) Initial defect detection stage:

[0056] In the initial defect detection stage, the abstract feature map is upsampled using a global local upsampling module on the encoding and decoding architecture. The purpose of this stage is to roughly detect the location of the defect, remove noise, and provide a clearer defect feature map for the second stage. In this embodiment, the last four layers of ResNet34 pre-trained on ImageNet are used as the first four encoders, denoted as: the first encoding layer, the second encoding layer, the third encoding layer, and the fourth encoding layer. Subsequently, a 3×3 depth-separable convolution and a batch normalization are used as the fifth encoding layer. The first encoding layer increases the number of feature channels to 64, and the size remains unchanged. The second encoding layer to the fifth encoding layer increase the number of feature map channels to {128, 256, 512, 1024} and reduce the size to {112, 56, 28, 14} respectively. When decoding the abstract feature map of the defect image, first use 1×

[0057] 1 Convolution performs channel compression on the five encoded feature maps to obtain abstract feature maps E1-E5 (i.e., the first abstract feature map to the fifth abstract feature map), and then uses four GLUMs to upsample and decode the feature maps. After each upsampling, the abstract feature map of the corresponding size is added to the upsampled feature map to finally obtain the initial defect segmentation map.

[0058] The initial defect detection stage, such as Figure 2As shown, it includes: a first coding layer, a second coding layer, a third coding layer, a fourth coding layer, a fifth coding layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first global local upsampling module, a second global local upsampling module, a third global local upsampling module and a fourth global local upsampling module;

[0059] The last four layers of the pre-trained ResNet34 are used as the first, second, third, and fourth coding layers respectively; a 3×3 depthwise separable convolution and a batch normalization layer are used as the fifth coding layer;

[0060] The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer are all 1×1 convolutions;

[0061] Input the hot-rolled steel strip image to be detected into the first coding layer, input the output of the first coding layer into the second coding layer and the first convolution layer, and obtain the first abstract feature map; input the output of the second coding layer into the third coding layer and the second convolution layer, and obtain the second abstract feature map; input the output of the third coding layer into the fourth coding layer and the third convolution layer, and obtain the third abstract feature map; input the output of the fourth coding layer into the fifth coding layer and the fourth convolution layer, and obtain the fourth abstract feature map; use the output of the fifth coding layer as the fifth abstract feature map; use the first abstract feature map, the second abstract feature map, the third abstract feature map, the fourth abstract feature map, and the fifth abstract feature map as the first coding feature map;

[0062] The fifth abstract feature map is input into the fourth global local upsampling module, the output of the fourth global local upsampling module and the sum of the fourth abstract feature map are input into the third global local upsampling module, the output of the third global local upsampling module and the sum of the third abstract feature map are input into the second global local upsampling module, the output of the second global local upsampling module and the sum of the second abstract feature map are input into the first global local upsampling module, and the output of the first global local upsampling module and the sum of the first abstract feature map are used as the initial defect segmentation map.

[0063] The first coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 64, and the channel size remains unchanged; the second coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 128, and the channel size is reduced to 112; the third coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 256, and the channel size is reduced to 56; the fourth coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 512, and the channel size is reduced to 28; the fifth coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 1024, and the channel size is reduced to 14.

[0064] This embodiment also uses a global local upsampling module. During the upsampling process, the feature map is upsampled using upsampling operations of multiple different receptive fields and a series of residual structures to obtain refined upsampled feature maps. For a batch of upsampled feature maps, the feature maps are first upsampled using 3×3 deconvolution with three different expansion rates {1,2,3} to obtain T1, T2, and T3 respectively. Then, T1, T2, and T3 are added in pairs, and then residual learning is performed on the added feature maps. The residual learning convolution size is 3×3, and the expansion rate is 3 to prevent problems such as image boundary discontinuity and target confusion in the feature maps after upsampling with different receptive fields. Finally, the feature maps obtained by the three residual learning are added, and a residual block is introduced again to integrate the three to obtain the final upsampled map. As shown in the figure Figure 3 As shown, it specifically includes: a first deconvolution, a second deconvolution, a third deconvolution, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer and a ninth convolution layer; the first deconvolution, the second deconvolution and the third deconvolution have different expansion rates; the sixth convolution layer, the seventh convolution layer and the eighth convolution layer are all residual convolutions;

[0065] Upsampling the feature map to be upsampled by the first deconvolution, the second deconvolution, and the third deconvolution, respectively, to obtain a first upsampling result, a second upsampling result, and a third upsampling result, respectively;

[0066] Respectively adding the first upsampling result, the second upsampling result and the third upsampling result in pairs to obtain a first addition result, a second addition result and a third addition result;

[0067] Input the first addition result into the sixth convolution layer, input the second addition result into the seventh convolution layer, and input the third up-sampling result into the eighth convolution layer;

[0068] The output results of the sixth convolutional layer, the seventh convolutional layer, and the eighth convolutional layer are added, the sum is input to the ninth convolutional layer, and the output of the ninth convolutional layer is used as the upsampling result.

[0069] Pre-trained GT decoder:

[0070] In this embodiment, four depth-wise separable convolutions are first used to encode the GT map (Ground Truth map, specifically, the picture is numbered with values ​​0-255, defects are 255, and background is 0). Each depth-wise separable convolution reduces the size of the GT map by two times and increases the number of channels by two times. Then, four global local upsampling modules (GLUMs) are used to decode it. Each GLUM enlarges the size of the feature map by two times and reduces the number of channels by two times. The GT decoder after freezing the decoding parameters is used as the GT decoder with frozen parameters, and the GT decoder with unfrozen decoding parameters is used as the GT decoder with unfrozen parameters.

[0071] (3) Classification stage:

[0072] In the category-based refinement stage, the separation module is first used to separate the initial defect feature map provided in the first stage by category, and then the defect feature maps of different categories are encoded and decoded again using the specific defect refinement module. In the last layer, they are fused with the GT self-decoder using the prior information fusion module to complete the fine segmentation.

[0073] Among them, the category separation module separates the initial defect segmentation map according to the defect category, so that the next stage can be refined for specific category defects. First, four convolutional layers are used to extract features from the initial defect segmentation map. Each convolutional layer consists of a 3×3 depth-separable convolution, batch normalization (BN) and Relu activation function. The four convolutional layers increase the number of feature map channels to {4, 16, 64, 256}, and reduce the size to {112, 56, 28, 14}, and calculate the mean vector of the feature map. Finally, two fully connected layers are used to obtain the BC*N category vector, where BC is the batch size and N is the number of categories. The category vector obtains the category index through the softmax activation function, and the feature maps of the same category are separated according to the category index for specific segmentation. Figure 4 As shown, it includes: a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a mean vector, a first fully connected layer, a second fully connected layer, and a category index; the tenth convolutional layer, the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the mean vector, the first fully connected layer, the second fully connected layer, and the category index are connected in sequence;

[0074] The initial defect segmentation map is input into the tenth convolutional layer, and passes through the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the mean vector, the first fully connected layer, the second fully connected layer and the category index in sequence, and is separated according to specific defect categories to obtain a specific category feature map.

[0075] Among them, the specific defect refinement module first uses four depth-separable convolutions to re-encode the initial defect segmentation map, then uses four GLUMs to upsample the feature map to obtain Fd, and uses the pre-trained GT decoder to decode the abstract feature map to obtain Fp. Finally, the prior information fusion module is used to fuse Fd and Fp to obtain the final segmentation map. Figure 5 As shown, it specifically includes: a fifteenth convolutional layer, a sixteenth convolutional layer, a seventeenth convolutional layer, an eighteenth convolutional layer, a fifth global local upsampling module, a sixth global local upsampling module, a seventh global local upsampling module, an eighth global local upsampling module and a priori information fusion module; the fifteenth convolutional layer, the sixteenth convolutional layer, the seventeenth convolutional layer, and the eighteenth convolutional layer are depth-separable convolutions;

[0076] Input the specific category feature map into the fifteenth convolution layer for encoding, and input the output of the fifteenth convolution layer into the sixteenth convolution layer;

[0077] The output of the sixteenth convolutional layer is input to the seventeenth convolutional layer, the output of the seventeenth convolutional layer is input to the eighteenth convolutional layer, the output of the eighteenth convolutional layer is input to the eighth global local upsampling module, the output of the eighth global local upsampling module is added to the output of the seventeenth convolutional layer, and the addition result is input to the seventh global local upsampling module, the output of the seventh global local upsampling module is added to the output of the sixteenth convolutional layer, and the addition result is input to the sixth global local upsampling module, the output of the sixth global local upsampling module is added to the output of the fifteenth convolutional layer, and the addition result is input to the fifth global local upsampling module, and the output of the fifth global local upsampling module is used as the first decoding feature map Fd and input to the prior information fusion module;

[0078] The GT decoder with unfrozen parameters is used to decode the first encoded feature map to obtain the second decoded feature map F p , the second decoding feature map F p The information is input into the prior information fusion module, and the output of the prior information fusion module is used as the strip surface defect segmentation result.

[0079] It can be understood that the GT decoder without freezing the decoding parameters adopts the fifth global local upsampling module to the eighth global local upsampling module, four global local upsampling modules to realize decoding.

[0080] Among them, the prior information fusion module, for the two feature maps Fd and Fp to be fused, first uses channel attention and spatial attention to generate channel attention weights CAd, CAp and spatial attention weights SAp, SAd respectively. Then the corresponding attention weights are multiplied with the corresponding feature maps to obtain Fd1 and Fp1, and then different attention weights are multiplied and added with different feature maps to obtain Ff1 and Ff2. Finally, the four feature maps are spliced, and a 1×1 convolution compression channel and Sigmoid activation function are used to compress the result to 0-1 to obtain the final segmentation map, as shown in Figure 6 As shown, it includes: a first channel attention unit, a second channel attention unit, a first spatial attention unit, a second spatial attention unit, a channel splicing unit and a fifth convolutional layer;

[0081] The first channel attention unit is used to generate a first channel attention unit weight, multiply the first decoded feature map by the first channel attention weight channel by channel to obtain a first multiplication result, and multiply the second decoded feature map by the first channel attention weight channel by channel to obtain a second multiplication result;

[0082] The second channel attention unit is used to generate a second channel attention weight, multiply the first decoded feature map by the second channel attention weight channel by channel to obtain a third multiplication result, and multiply the second decoded feature map by the second channel attention weight channel by channel to obtain a fourth multiplication result;

[0083] The first spatial attention unit is used to generate a first spatial attention weight, perform a corresponding element multiplication operation on the first multiplication result and the first spatial attention weight to obtain a fifth multiplication result, and perform a corresponding element multiplication operation on the third multiplication result and the first spatial attention weight to obtain a sixth multiplication result;

[0084] The second spatial attention unit is used to generate a second spatial attention weight, perform a corresponding element multiplication operation on the second multiplication result and the second spatial attention weight to obtain a seventh multiplication result, and perform a corresponding element multiplication operation on the fourth multiplication result and the second spatial attention weight to obtain an eighth multiplication result;

[0085] The channel splicing unit is used to splice the fifth multiplication result, the sixth multiplication result, the seventh multiplication result and the eighth multiplication result to obtain a splicing result;

[0086] The fifth convolutional layer is used to compress the splicing result to obtain the strip surface defect segmentation result.

[0087] In order to more clearly describe the specific implementation process of this embodiment, an example is first given to describe the specific implementation process:

[0088] Step S100: Obtain a picture of the hot-rolled steel strip to be inspected, and increase the picture size to 224*224.

[0089] Step S101: normalize the enlarged image, with a mean of {0.485, 0.456, 0.406} and a standard deviation of {0.229, 0.224, 0.225}.

[0090] Step S102: Construct a pre-trained two-stage encoding and multi-decoding model, which includes: an initial defect detection stage and a class-based refinement stage. Specifically: First, train the GT decoder. The input of the GT self-encoder is the GT graph, and the output is the prediction of the GT. The training loss function is a binary cross entropy loss function. First, train GT graphs of different categories, with a training batch size of 32 and a training round number of 50. After training, this example saves the decoder part of the GT self-encoder, and uses the GT decoder with frozen decoding parameters as the GT decoder with frozen parameters, and uses the GT decoder with unfrozen decoding parameters as the GT decoder with unfrozen parameters for subsequent fusion.

[0091] Specifically: The input of the initial defect prediction stage is the original defect image, and the output is the prediction of the GT map. The encoder of the initial defect prediction stage uses the last four layers of ResNet34 pre-trained on ImageNet as the first four layers of the encoder, denoted as: the first encoding layer-the fourth encoding layer, followed by a 3×3 depth-separable convolution and a batch normalization, denoted as the fifth encoding layer, where the first encoding layer increases the number of feature channels to 64, with the size unchanged, and the second encoding layer-the fifth encoding layer respectively increases the number of feature map channels to {128, 256, 512, 1024} and reduces the size to {112, 56, 28, 14}. When decoding the abstract feature map of the defect image, first use 1×1 convolution to compress the channels of the five encoded feature maps to obtain the abstract feature maps E1-E5, and then use four GLUMs to upsample and decode the feature maps. After each upsampling, the abstract feature map of the corresponding size is added to the upsampled feature map to finally obtain the initial defect segmentation map.

[0092] Specifically: The input of the global local upsampling module is a batch of feature maps to be upsampled, and the output is a feature map with a size twice that of the input feature map and a channel number twice that of the input feature map. The specific details are as follows: First, three deconvolution operations with different expansion rates {1, 2, 3}, a size of 4×4, a step size of 2, and padding {1, 3, 4} are used to double the feature map and double the number of channels. Then, the upsampled feature maps with different expansion rates are added in pairs, and then convolution operations are performed on the added feature maps with a size of 3×3, a step size of 1, a padding of 3, and a dilation rate of 3, and skip links are used to prevent gradient disappearance. Finally, the three feature maps are spliced ​​together and learned again using a residual block to obtain high-quality upsampled feature maps.

[0093] Specifically: In the category-based refinement stage, CSM is first used to separate the feature maps by category, and then four convolution layers consisting of a depth-wise separable convolution with a size of 3×3, a step size of 2, and a padding of 1, batch normalization (BN), and a Relu activation function are used to extract features from the feature maps of each category again. Then, the mean of the second and third dimensions of a batch of feature maps is calculated to obtain a BC*64 mean vector, and then two fully-linked layers are used to obtain the category feature vector BC*N, where BC is the batch size and N is the number of categories. The category feature vector is softmaxed to obtain the category index, and finally the feature map is separated by category according to the category index so that it can be decoded using a specific decoder.

[0094] Specifically: After separating the feature maps by category in the category refinement stage, the feature maps of specific categories are refined using the specific defect refinement module. First, four convolutional layers are used, including: a depthwise separable convolution with a size of 3×3, a stride of 2, and a padding of 1, batch normalization (BN) and a Relu activation function, to extract features from the initial defect segmentation map, and then four GLUMs are used to upsample the feature maps and add them to the encoded features.

[0095] Specifically: After extracting features from the initial defect segmentation map, the decoder of the GT autoencoder with frozen parameters is used to decode the feature map, and the GT decoder is fused with the main decoder (GT decoder with unfrozen parameters) using the prior information feature fusion module. The specific steps are as follows: First, the two feature maps to be fused: F d and F p First, channel attention and spatial attention are used to generate their respective attention weights CA d , C.A. p and SA p , S.A. d Then, the corresponding attention weights are multiplied by the corresponding feature maps to obtain F d1 , F p1 , then multiply different attention weights with different feature maps and add them together to get F f1 , F f2 Finally, the four feature maps are concatenated, and a 1×1 convolution is used to compress the feature channel to 1, and the Sigmoid activation function is used to compress the result to 0-1 to obtain the final segmentation map.

[0096] Step S103: training the constructed model: input the preprocessed image with defects, the model performs segmentation prediction on the image, then compares the prediction result with the true label image, calculates the loss value, and updates the model parameters by gradient descent.

[0097] Step S104: Test the model: In order to fairly and accurately evaluate the segmentation performance of the network proposed in this paper, this paper uses the mean intersection-over-union ratio (mIou), recall rate (Recall), Dice coefficient and Roc curve as evaluation indicators. Miou represents the ratio of the intersection and union between the predicted results and the true labels, which can comprehensively consider the network's segmentation ability for each category and prevent the model from being biased towards predicting categories with a large number of categories. Recall represents the ratio of the number of predicted positive examples (defective pixels) to the number of positive examples in the true labels, which effectively evaluates the network's ability to successfully capture positive examples. The Dice coefficient comprehensively considers the prediction accuracy and recall rate of the network, and is more stable and effective for unbalanced samples. It can better evaluate the network's recognition performance for positive examples and avoid the network segmenting all the background when positive samples are scarce, but the evaluation indicators are still good. The ROC curve intuitively demonstrates the network's ability to segment defects by showing the true positive rate and false positive rate of the network. The calculation formula is as follows:

[0098]

[0099]

[0100] Among them, TP represents the true positive, that is, the number of pixels that the network correctly predicts as positive examples, TN represents the true negative, that is, the number of pixels that the network correctly predicts as negative examples, FP represents the false positive, that is, the number of pixels that the network incorrectly predicts as positive examples, and FN represents the false negative, that is, the number of pixels that the network incorrectly predicts as positive examples.

[0101] Table 1: Comparison results of different methods on the SD-saliency-900 dataset

[0102]

[0103] Table 2: Comparison results of different methods under X-SDD dataset

[0104]

[0105] Among them, EDRNet: Encoder-Decoder Residual Network for Salient Object Detection of Strip Steel Surface Defects; TSERNet: Two-Stage Edge Reuse Network for Salient Object Detection of Strip Steel Surface Defects; LFRNet: Localizing, Focusing, and Refinement Network for Salient Object Detection of Surface Defects; PGA-Net: Pyramid Feature Fusion and Global Context Attention Networkfor Automated Surface Defect Detection; DACNet: Dense Attention-Guided Cascaded Network forSalient Object Detection of Strip Steel Surface Defects; CGTD-Net: Channel-wise global Transformerbased Dual-branch network for industrial strip steel surface defect detection A dual-branch network based on channel global transformer is used for industrial strip steel surface defect detection; AFF-Net: AStrip Steel Surface Defect Detection Network via Adaptive Focusing Features A network for strip steel surface defect detection via adaptive focusing features. The comparison results of different methods under the SD-saliency-900 dataset in this example are shown in Table 1, the comparison results of different methods under the X-SDD dataset are shown in Table 2, and the visual comparison of segmentation results of different methods is shown in Figure 8 As shown. Figure 7 As shown in the figure, the X-SDD dataset contains six categories, and the number of defects in different categories is unbalanced, which will cause the data-driven deep learning method to be biased towards categories with more samples during training, thereby reducing the recognition performance of categories with fewer samples. This may cause the model to have high accuracy in prediction, but poor performance on small categories, ultimately affecting the overall generalization ability of the model. This example mainly aims at the poor performance of strip surface defect segmentation under low contrast conditions and the large difference between the tail category defect performance and the head category in the unbalanced distribution. A two-stage encoding and multi-decoding strip surface defect segmentation method is proposed, which effectively improves the accuracy of defect detection, especially tail defects. In this method, all decoders use the global local upsampling module proposed in this example to upsample the feature map to obtain high-quality abstract feature maps. At the same time, this example uses a two-stage method to detect defects, and uses the category separation module to separate the feature map by category. In the second stage, specific parameters are used to refine the detection of each category of defects. The strip defect detection method proposed in this example can not only detect strip surface defects under low contrast conditions, but also improve the accuracy of tail category defects when facing unbalanced category distribution. In summary, the method proposed in this example can accurately detect surface defects of strip steel and meet industrial needs. This method has high reliability in practical applications, providing strong support for its wide application in industrial production.

[0106] Embodiment 2:

[0107] This embodiment proposes an electronic device, comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the two-stage encoding and multi-decoding strip surface defect segmentation method.

[0108] The electronic device may be a mobile phone, a computer or a tablet computer, etc., including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the strip surface defect segmentation method with two-stage encoding and multiple decoding as described in the embodiment is implemented. It can be understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0109] The processor is used to execute all or part of the steps in the two-stage encoding and multi-decoding strip surface defect segmentation method as described in the above embodiment. The memory is used to store various types of data, which may include instructions of any application or method in the electronic device, as well as data related to the application.

[0110] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute the two-stage encoding and multi-decoding strip surface defect segmentation method described in the above embodiment.

[0111] Embodiment 3:

[0112] This embodiment provides a computer-readable storage medium storing executable instructions. When the instructions are executed and implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0113] The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the two-stage encoding and multi-decoding strip surface defect segmentation method described in each embodiment of the present application.

[0114] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, CD, server, APP (Application, abbreviation of application software) application store and other media that can store program verification codes, on which computer programs are stored. When the computer program is executed by the processor, the various steps of the above-mentioned two-stage encoding and multi-decoding strip surface defect segmentation method can be implemented.

[0115] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0116] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A two-stage encoding and multi-decoding strip surface defect segmentation method, characterized in that: include: Obtain a picture of the hot-rolled steel strip to be inspected; Inputting the hot-rolled steel strip image to be inspected into a pre-trained two-stage encoding and multi-decoding model to obtain a strip surface defect segmentation result; The pre-trained two-stage encoding and multi-decoding model includes: an initial defect detection stage and a class-based refinement stage; The initial defect detection stage is used to encode the hot-rolled strip image to be detected using a U-type encoding and decoding structure to obtain a first encoding feature map, and upsample the first encoding feature map using a global local upsampling module to obtain an initial defect segmentation map; the class-based refinement stage is used to separate the initial defect segmentation map by category using a category separation module to obtain a specific category feature map, and encode the specific category feature map again using a specific defect refinement module to obtain a second encoding feature map, and decode the second encoding feature map using a GT decoder with frozen parameters to obtain a first decoding feature map, and decode the second encoding feature map using a GT decoder with unfrozen parameters to obtain a second decoding feature map, and fuse the first decoding feature map with the second decoding feature map using a priori information fusion module to obtain a strip surface defect segmentation result.

2. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 1 is characterized in that: The training process of the GT decoder with frozen parameters includes: using four depth-separable convolutions to encode the truth map of the historical hot-rolled steel strip, each depth-separable convolution reduces the size of the truth map of the historical hot-rolled steel strip by two times, and increases the number of channels by two times; using four global local upsampling modules to decode the encoding result, each global local upsampling module enlarges the size of the truth map of the historical hot-rolled steel strip by two times, and reduces the number of channels by two times; and using the GT decoder after the decoding parameters are frozen as the GT decoder with frozen parameters; The GT decoder without freezing the decoding parameters uses four global-local upsampling modules to implement decoding.

3. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 1 is characterized in that: The initial defect detection stage includes: a first coding layer, a second coding layer, a third coding layer, a fourth coding layer, a fifth coding layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first global local upsampling module, a second global local upsampling module, a third global local upsampling module and a fourth global local upsampling module; The last four layers of the pre-trained ResNet34 are used as the first, second, third, and fourth coding layers respectively; a 3×3 depthwise separable convolution and a batch normalization layer are used as the fifth coding layer; The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer are all 1×1 convolutions; Input the hot-rolled steel strip image to be detected into the first coding layer, input the output of the first coding layer into the second coding layer and the first convolution layer, and obtain the first abstract feature map; input the output of the second coding layer into the third coding layer and the second convolution layer, and obtain the second abstract feature map; input the output of the third coding layer into the fourth coding layer and the third convolution layer, and obtain the third abstract feature map; input the output of the fourth coding layer into the fifth coding layer and the fourth convolution layer, and obtain the fourth abstract feature map; use the output of the fifth coding layer as the fifth abstract feature map; use the first abstract feature map, the second abstract feature map, the third abstract feature map, the fourth abstract feature map, and the fifth abstract feature map as the first coding feature map; The fifth abstract feature map is input into the fourth global local upsampling module, the output of the fourth global local upsampling module and the sum of the fourth abstract feature map are input into the third global local upsampling module, the output of the third global local upsampling module and the sum of the third abstract feature map are input into the second global local upsampling module, the output of the second global local upsampling module and the sum of the second abstract feature map are input into the first global local upsampling module, and the output of the first global local upsampling module and the sum of the first abstract feature map are used as the initial defect segmentation map.

4. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 3 is characterized in that: The first coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 64, and the channel size remains unchanged; the second coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 128, and the channel size is reduced to 112; the third coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 256, and the channel size is reduced to 56; the fourth coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 512, and the channel size is reduced to 28; the fifth coding layer increases the number of channels of the hot-rolled steel strip image to be detected to 1024, and the channel size is reduced to 14.

5. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 1 is characterized in that: The global local upsampling module includes: a first deconvolution, a second deconvolution, a third deconvolution, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer and a ninth convolution layer; the first deconvolution, the second deconvolution and the third deconvolution have different expansion rates; the sixth convolution layer, the seventh convolution layer and the eighth convolution layer are all residual convolutions; Upsampling the feature map to be upsampled by the first deconvolution, the second deconvolution, and the third deconvolution, respectively, to obtain a first upsampling result, a second upsampling result, and a third upsampling result, respectively; Respectively adding the first upsampling result, the second upsampling result and the third upsampling result in pairs to obtain a first addition result, a second addition result and a third addition result; Input the first addition result into the sixth convolution layer, input the second addition result into the seventh convolution layer, and input the third up-sampling result into the eighth convolution layer; The output results of the sixth convolutional layer, the seventh convolutional layer, and the eighth convolutional layer are added, the sum is input to the ninth convolutional layer, and the output of the ninth convolutional layer is used as the upsampling result.

6. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 1 is characterized in that: The category separation module includes: a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a mean vector, a first fully connected layer, a second fully connected layer, and a category index; the tenth convolutional layer, the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the mean vector, the first fully connected layer, the second fully connected layer, and the category index are connected in sequence; The initial defect segmentation map is input into the tenth convolutional layer, and passes through the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the mean vector, the first fully connected layer, the second fully connected layer and the category index in sequence, and is separated according to specific defect categories to obtain a specific category feature map.

7. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 1 is characterized in that: The specific defect refinement module includes: a fifteenth convolution layer, a sixteenth convolution layer, a seventeenth convolution layer, an eighteenth convolution layer, a fifth global local upsampling module, a sixth global local upsampling module, a seventh global local upsampling module, an eighth global local upsampling module and a priori information fusion module; the fifteenth convolution layer, the sixteenth convolution layer, the seventeenth convolution layer and the eighteenth convolution layer are depth-separable convolutions; Input the specific category feature map into the fifteenth convolution layer for encoding, and input the output of the fifteenth convolution layer into the sixteenth convolution layer; Input the output of the sixteenth convolutional layer to the seventeenth convolutional layer, input the output of the seventeenth convolutional layer to the eighteenth convolutional layer, input the output of the eighteenth convolutional layer to the eighth global local upsampling module, add the output of the eighth global local upsampling module to the output of the seventeenth convolutional layer, and input the addition result to the seventh global local upsampling module, add the output of the seventh global local upsampling module to the output of the sixteenth convolutional layer, and input the addition result to the sixth global local upsampling module, add the output of the sixth global local upsampling module to the output of the fifteenth convolutional layer, and input the addition result to the fifth global local upsampling module, and use the output of the fifth global local upsampling module as the first decoding feature map to input into the prior information fusion module; The second decoded feature map is input into the prior information fusion module, and the output of the prior information fusion module is used as the strip surface defect segmentation result.

8. The two-stage encoding and multi-decoding strip surface defect segmentation method according to claim 1 is characterized in that: The prior information fusion module includes: a first channel attention unit, a second channel attention unit, a first spatial attention unit, a second spatial attention unit, a channel splicing unit and a fifth convolutional layer; The first channel attention unit is used to generate a first channel attention unit weight, multiply the first decoded feature map by the first channel attention weight channel by channel to obtain a first multiplication result, and multiply the second decoded feature map by the first channel attention weight channel by channel to obtain a second multiplication result; The second channel attention unit is used to generate a second channel attention weight, multiply the first decoded feature map by the second channel attention weight channel by channel to obtain a third multiplication result, and multiply the second decoded feature map by the second channel attention weight channel by channel to obtain a fourth multiplication result; The first spatial attention unit is used to generate a first spatial attention weight, perform a corresponding element multiplication operation on the first multiplication result and the first spatial attention weight to obtain a fifth multiplication result, and perform a corresponding element multiplication operation on the third multiplication result and the first spatial attention weight to obtain a sixth multiplication result; The second spatial attention unit is used to generate a second spatial attention weight, perform a corresponding element multiplication operation on the second multiplication result and the second spatial attention weight to obtain a seventh multiplication result, and perform a corresponding element multiplication operation on the fourth multiplication result and the second spatial attention weight to obtain an eighth multiplication result; The channel splicing unit is used to splice the fifth multiplication result, the sixth multiplication result, the seventh multiplication result and the eighth multiplication result to obtain a splicing result; The fifth convolutional layer is used to compress the splicing result to obtain the strip surface defect segmentation result.

9. An electronic device, characterized in that: include: One or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the two-stage encoding and multi-decoding strip surface defect segmentation method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: It stores executable instructions, which, when executed, enable the processor to execute the two-stage encoding and multi-decoding strip surface defect segmentation method described in any one of claims 1 to 8.

Images