Strip steel surface defect few-sample segmentation method of multi-scale adaptive prototype network

Abstract

The invention relates to a strip steel surface defect few-sample segmentation method of a multi-scale self-adaptive prototype network, which is characterized in that a multi-scale self-adaptive prototype transformer network is used and embedded into a strip steel surface defect segmentation framework so as to improve the feature representation capability. The network comprises a feature extraction module, a self-adaptive prototype transformer module and a multi-scale feature fusion decoder, and through self-adaptive prototype generation and multi-scale feature fusion, the spatial position and feature expression of the defect can be captured more accurately. Particularly, a self-adaptive prototype generated by the self-adaptive prototype transformer module can better match and query the features of the image in a complex and changeable defect scene, so that the segmentation precision is enhanced.

Description

Technical Field

[0001] The invention relates to a few-sample segmentation method for strip steel surface defects based on a multi-scale adaptive prototype network, and belongs to the technical field of computer vision. Background Art

[0002] In the industrial production process, the detection of surface defects of strip steel is a key step to ensure product quality. Traditional defect detection methods mainly rely on manual inspection, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in inconsistent detection results. With the development of computer vision technology, automated detection methods based on deep learning have gradually become a research hotspot. However, existing automated detection technologies still face many challenges.

[0003] Existing deep learning methods have shown certain limitations in strip surface defect detection. First, these methods often rely on a large amount of labeled data for training, while the data acquisition and labeling process of strip surface defects is expensive. In addition, the defects on the strip surface are diverse and complex in shape. Different types of defects may have significant differences in visual features, which makes it difficult for existing models to show good generalization ability in diverse scenarios. Especially in the absence of sufficient labeled samples, it is difficult for existing models to effectively learn the features to distinguish defects, resulting in unsatisfactory segmentation results. As an effective method to deal with small sample data, few-shot learning has made significant progress in object detection and segmentation tasks in natural scenes. However, there are still many challenges in directly applying few-shot learning to strip surface defect detection. Strip surface defects usually show discrete distribution, irregular shape, high inter-class similarity, and large intra-class variance, which are quite different from object detection in natural scenes. In addition, when generating prototypes, existing few-shot learning methods usually only rely on the features of support samples and ignore the potential information of query samples, resulting in poor performance of the generated prototypes when dealing with complex defects. Summary of the invention

[0004] In order to overcome the defects of the prior art, the present invention provides a strip surface defect few-sample segmentation method of a multi-scale adaptive prototype network that combines multi-scale feature extraction and adaptive prototype generation technology to improve the accuracy and robustness of strip surface defect detection tasks and effectively cope with complex and changeable defect scenarios.

[0005] The method for few-sample segmentation of strip surface defects based on a multi-scale adaptive prototype network includes the following steps:

[0006] Step 1: Dataset preprocessing: Perform a series of data enhancement and standardization processes on the strip surface defect dataset FSSD-12;

[0007] Step 2: Construct a strip surface defect segmentation network: The strip surface defect segmentation network consists of three parts: feature extraction module, adaptive prototype module and multi-scale feature fusion decoder. The convolutional neural network ResNet-50 is used as the feature extraction module to extract multi-scale feature information from the strip surface image. These features are sent to the adaptive prototype module for prototype generation and feature optimization. The feature map is fused and restored through the multi-scale feature fusion decoder to generate the final defect segmentation result.

[0008] Step 3: Training the network: Input the preprocessed training set images into the strip surface defect segmentation network, and generate a segmentation prediction map of the same size as the input image through the feature extraction module, adaptive prototype module and multi-scale feature fusion decoder; then use the binary cross entropy loss function to calculate the loss, and supervise the output of each layer through the deep dual supervision mechanism, optimize the network weights through the selected optimizer and corresponding parameters, and obtain the final network model after multiple rounds of training;

[0009] Step 4: Model testing: Input the preprocessed test set images into the trained strip surface defect segmentation network model to obtain the segmentation prediction map, compare it with the real annotation, and calculate the relevant evaluation indicators.

[0010] The step 1 specifically includes:

[0011] The strip surface defect dataset is preprocessed, and the images of the training set undergo the following data augmentation operations: random rotation with the angle range set to -10 degrees to 10 degrees; random scaling with the ratio range set to 0.9 to 1.1; and mirror flipping operations in the horizontal and vertical directions.

[0012] The step 2 specifically includes:

[0013] A feature extraction module based on ResNet-50 is constructed to extract multi-scale feature information from the surface image of the strip. The input image is firstly subjected to preliminary feature extraction through the convolution layer and the pooling layer. The image features are then fed into the four residual blocks of ResNet-50 to extract features of different scales.

[0014] After the feature extraction module, the extracted multi-scale features are input into the adaptive prototype module, and the prototype is generated through the attention mechanism. The prototype is used to capture the relationship between the support sample and the query sample. This module uses the features of the support sample and the query sample to generate an adaptive multi-scale prototype. The prototype can accurately match the target position of the defect and generate a fine-grained feature representation. In this module, position embedding is first performed to enhance spatial information, and then the relationship between features is captured through the multi-head attention mechanism. Finally, the feature transformation is performed through the fully connected layer to generate an optimized prototype. The main attention expression used is:

[0015]

[0016] Where Attention(Q,K,V) is the obtained attention value, Q is used to query relevant information, K is used to represent the features of each position in the input sequence, and V represents the content related to these positions. By calculating the similarity between Q and K, the model weights V and extracts the most relevant information;

[0017] The multi-scale prototype features generated from the above modules are sent to the multi-scale feature fusion decoder. Through layer-by-layer upsampling and feature fusion, the spatial resolution of the image is gradually restored. The decoder uses feature splicing and convolution operations for feature fusion, and finally generates an accurate segmentation prediction map.

[0018] The step three specifically includes:

[0019] Initialize the built model, including loading the pre-trained ResNet-50 as the feature extraction module and initializing the parameters in the adaptive prototype module and the multi-scale feature fusion decoder. The feature extraction module uses the weights pre-trained on the ImageNet dataset, and the parameters of the adaptive prototype module and the decoder are initialized according to the random initialization strategy;

[0020] The binary cross entropy loss function is used to measure the difference between the predicted segmentation map and the true segmentation map. A deep dual supervision mechanism is introduced. The BCE loss function is applied to the output of each layer to supervise the final output and the feature map of the intermediate layer, which promotes the model to learn useful features at different levels. The relevant formula is as follows:

[0021]

[0022] Among them, y i represents the true label, represents the predicted probability of the output of the lth layer, N represents the total number of samples, L represents the total number of layers of the model, BCE l represents the binary cross entropy loss of the lth layer, and Total Loss represents the total loss function;

[0023] Two different optimizers were selected to handle the parameter updates of different modules respectively. For the other parts except the adaptive prototype module, the SGD optimizer was used, with the initial learning rate set to 0.003, the momentum to 0.9, and the weight decay to 0.0001. For the adaptive prototype module, the Adam optimizer was used, with the initial learning rate of 6e-5.

[0024] During the training process, the preprocessed image is input into the model. The model extracts multi-scale features through the feature extraction module, generates an adaptive prototype in the adaptive prototype module, fuses the multi-scale features in the decoder, and generates a segmentation prediction map. In each training batch, the model calculates the loss between the predicted map and the true map, and updates the network weights through back propagation. The learning rate will gradually decay to ensure the stability and convergence of the training. It is defined as follows:

[0025]

[0026] Where lr is the current learning rate, base_lr is the initial base learning rate, curr_iter represents the current number of iterations, max_iter is the total number of iterations, and power is the exponent that controls the decay rate;

[0027] As the training progresses, the model gradually focuses on learning detailed features. After each epoch, the model will be verified to evaluate the current training effect and adjust the training strategy. When the training reaches the predetermined number of rounds, the model with the best performance on the verification set is selected as the final output model and saved for subsequent testing and practical application.

[0028] The step 4 specifically includes:

[0029] The preprocessed test set images are input into the trained model to generate defect segmentation prediction images, which are compared with the true labels and various evaluation indicators are calculated, including the average intersection-over-union ratio and the boundary area intersection-over-union ratio.

[0030] The average intersection-over-union ratio is a standard metric for evaluating the overall segmentation performance of the model, and its calculation formula is:

[0031]

[0032] Among them, C represents the number of categories, TP is the number of true positive pixels, FP is the number of false positive pixels, and FN is the number of false negative pixels;

[0033] The boundary area intersection ratio focuses more on evaluating the segmentation effect of the model in the defect boundary area. The formula is as follows:

[0034]

[0035] Among them, TPB, FPB and FNB represent the number of true positive, false positive and false negative pixels in the boundary area, respectively.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The invention relates to a method for segmenting surface defects of steel strips using a multi-scale adaptive prototype network with a few samples. The method uses a multi-scale adaptive prototype transformer network and embeds it into a framework for segmenting surface defects of steel strips to improve feature representation capabilities. The network includes a feature extraction module, an adaptive prototype transformer module, and a multi-scale feature fusion decoder. Through adaptive prototype generation and multi-scale feature fusion, the spatial location and feature expression of defects can be captured more accurately. In particular, the adaptive prototype generated by the adaptive prototype transformer module can better match the features of the query image in complex and changeable defect scenes, thereby enhancing the accuracy of segmentation.

[0038] 1. It has stronger adaptability to complex defect shapes and diversified scenarios, can more accurately locate and segment defect areas in the detection of surface defects on strip steel, and has better processing effects on problems such as low contrast and irregular shapes of defects.

[0039] 2. The generated strip surface defect segmentation results are more accurate, which can effectively reduce the occurrence of missed detection and false detection, and improve the overall detection accuracy and robustness BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a network structure diagram of the method for segmenting surface defects of steel strips with a small number of samples based on the multi-scale adaptive prototype network of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The method for few-sample segmentation of strip surface defects based on a multi-scale adaptive prototype network includes the following steps:

[0044] Step 1: Dataset preprocessing: Perform a series of data enhancement and standardization processes on the strip surface defect dataset FSSD-12;

[0045] First, the surface defect dataset of strip steel was preprocessed. Specifically, the images of the training set were subjected to the following data augmentation operations: random rotation with the angle range set to -10 degrees to 10 degrees; random scaling with the ratio range set to 0.9 to 1.1; and mirror flipping in the horizontal and vertical directions. In addition, in order to avoid overfitting of the model, the image data was standardized during the preprocessing process to ensure the balanced distribution of the data and the stability of the model.

[0046] Step 2: Construct a strip surface defect segmentation network: The strip surface defect segmentation network consists of three parts: feature extraction module, adaptive prototype module and multi-scale feature fusion decoder. The convolutional neural network ResNet-50 is used as the feature extraction module to extract multi-scale feature information from the strip surface image. These features are sent to the adaptive prototype module for prototype generation and feature optimization. The feature map is fused and restored through the multi-scale feature fusion decoder to generate the final defect segmentation result.

[0047] like Figure 1 As shown in the figure, a feature extraction module based on ResNet-50 is first constructed to extract multi-scale feature information from the surface image of the strip. Specifically, the input image is first subjected to preliminary feature extraction through the convolution layer and the pooling layer. Then, the image features are fed into the four residual blocks (block1 to block4) of ResNet-50 to extract features of different scales. These features capture the low-level, mid-level, and high-level information in the image, which facilitates subsequent defect detection.

[0048] After the feature extraction module, the extracted multi-scale features are input into the adaptive prototype module. This module generates prototypes through the attention mechanism, which are used to capture the relationship between the support samples and the query samples. Specifically, this module uses the features of the support samples and the query samples to generate adaptive multi-scale prototypes, which can accurately match the target position of the defect and generate fine-grained feature representations. In this module, position embedding is first performed to enhance spatial information, and then the relationship between features is captured through the multi-head attention mechanism. Finally, the feature transformation is performed through the fully connected layer to generate an optimized prototype. Among them, the main attention expression used is:

[0049]

[0050] Among them, Attention(Q,K,V) is the value of attention obtained, Q is used to query related information, K is used to represent the features of each position in the input sequence, and V represents the content related to these positions. By calculating the similarity between Q and K, the model weights V and extracts the most relevant information.

[0051] Finally, the multi-scale prototype features generated from the above modules are fed into the multi-scale feature fusion decoder. The design of the decoder is inspired by residual learning and feature pyramid networks. It gradually restores the spatial resolution of the image through layer-by-layer upsampling and feature fusion. The decoder uses feature concatenation and convolution operations for feature fusion instead of simple feature addition. This approach can retain global context information and detail features from different scales, and ultimately generate an accurate segmentation prediction map.

[0052] Step 3: Training the network: Input the preprocessed training set images into the strip surface defect segmentation network, and generate a segmentation prediction map of the same size as the input image through the feature extraction module, adaptive prototype module and multi-scale feature fusion decoder; then use the binary cross entropy loss function to calculate the loss, and supervise the output of each layer through the deep dual supervision mechanism, optimize the network weights through the selected optimizer and corresponding parameters, and obtain the final network model after multiple rounds of training;

[0053] First, the constructed model is initialized, including loading the pre-trained ResNet-50 as the feature extraction module and initializing the parameters in the adaptive prototype module and the multi-scale feature fusion decoder. The feature extraction module uses the weights pre-trained on the ImageNet dataset to provide a more stable feature extraction basis. The parameters of the adaptive prototype module and the decoder are initialized according to the random initialization strategy.

[0054] In order to ensure that the model can accurately segment the surface defects of the strip, the binary cross entropy (BCE) loss function is used. This loss function is used to measure the difference between the predicted segmentation map and the actual segmentation map. In order to further improve the performance of the model, a deep dual supervision mechanism is introduced. The BCE loss function is applied to the output of each layer, which not only supervises the final output, but also supervises the feature map of the intermediate layer, thereby promoting the model to learn useful features at different levels. The relevant formula is as follows:

[0055]

[0056] Among them, y i represents the true label, represents the predicted probability of the output of the lth layer, N represents the total number of samples, L represents the total number of layers of the model, BCE l represents the binary cross entropy loss of the lth layer, and Total Loss represents the total loss function.

[0057] In order to optimize the parameters of the model, two different optimizers are selected to handle the parameter updates of different modules respectively. For the other parts except the adaptive prototype module, the SGD optimizer is used, and its initial learning rate is set to 0.003, momentum is 0.9, and weight decay is 0.0001. For the adaptive prototype module, the Adam optimizer is used, and its initial learning rate is 6e-5. The reason for this optimizer selection is that SGD has better global search capabilities, while Adam can better adapt to the parameter updates of the attention mechanism in the relevant modules.

[0058] During the training process, the preprocessed images are input into the model. The model extracts multi-scale features through the feature extraction module, generates adaptive prototypes in the adaptive prototype module, and finally fuses the multi-scale features in the decoder to generate the segmentation prediction map. In each training batch, the model calculates the loss between the predicted map and the true map, and updates the network weights through back propagation. Specifically, the learning rate will gradually decay to ensure the stability and convergence of the training, which is defined as follows:

[0059]

[0060] Where lr is the current learning rate, base_lr is the initial base learning rate, curr_iter represents the current number of iterations, max_iter is the total number of iterations, and power is the exponent that controls the decay rate.

[0061] In order to ensure the generalization ability and accuracy of the model, the entire training process lasted for 201 epochs. In the early stage of training, the model mainly learns global features; as the training deepens, the model gradually focuses on learning detailed features. After each epoch, the model will be validated to evaluate the current training effect and adjust the training strategy. When the training reaches the predetermined number of rounds, the model with the best performance on the validation set is selected as the final output model and saved for subsequent testing and practical application.

[0062] Step 4: Model testing: Input the preprocessed test set images into the trained strip surface defect segmentation network model to obtain the segmentation prediction map, compare it with the real annotation, and calculate the relevant evaluation indicators.

[0063] The preprocessed test set images are input into the trained model to generate defect segmentation prediction maps, which are compared with the true labels to calculate various evaluation indicators, including the average intersection-over-union ratio and the boundary area intersection-over-union ratio.

[0064] The mean intersection over union (MIoU) is a standard metric for evaluating the overall segmentation performance of a model and is calculated as:

[0065]

[0066] Among them, C represents the number of categories, TP is the number of true positive pixels, FP is the number of false positive pixels, and FN is the number of false negative pixels.

[0067] The boundary area intersection over union (FBIoU) focuses more on evaluating the segmentation effect of the model in the defect boundary area. The formula is as follows:

[0068]

[0069] Among them, TPB, FPB and FNB represent the number of true positive, false positive and false negative pixels in the boundary area, respectively.

[0070] By calculating the above evaluation indicators, the performance of the model in different test scenarios can be comprehensively measured, thereby evaluating the effectiveness of the method in the task of strip surface defect segmentation.

[0071] In addition, in order to fully verify the model of the present invention, we conducted extensive comparisons on a public strip surface defect dataset FSSD-12 dataset. FSSD-12 is a collection specifically for the semantic segmentation of few samples of strip surface defects, which comes from various stages of production and manufacturing. The dataset contains a total of 12 defect categories, including cracks, scratches, oxidation, etc. All defect images in the present invention are standardized to a resolution of 200×200 to maintain consistency and reduce computational overhead. It is worth noting that the number of samples for each defect category is limited to 50 to mitigate the impact of long-tail distribution. The dataset is divided into three folds for cross-validation, each fold contains 4 categories, and these categories do not overlap in other folds. Specifically, the experimental data results are as follows:

[0072]

[0073]

[0074] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A few-sample segmentation method for strip surface defects based on a multi-scale adaptive prototype network, characterized by: The following steps are involved: Step 1: Dataset preprocessing: Data enhancement and standardization of the strip surface defect dataset FSSD-12; Step 2: Construct a strip surface defect segmentation network: The strip surface defect segmentation network consists of three parts: feature extraction module, adaptive prototype module and multi-scale feature fusion decoder. The convolutional neural network ResNet-50 is used as the feature extraction module to extract multi-scale feature information from the strip surface image. The features are sent to the adaptive prototype module for prototype generation and feature optimization. The feature map is fused and restored through the multi-scale feature fusion decoder to generate the final defect segmentation result. Step 3: Training the network: Input the preprocessed training set images into the strip surface defect segmentation network, and generate a segmentation prediction map of the same size as the input image through the feature extraction module, adaptive prototype module and multi-scale feature fusion decoder; use the binary cross entropy loss function to calculate the loss, supervise the output of each layer through the deep dual supervision mechanism, optimize the network weights through the selected optimizer and corresponding parameters, and obtain the final network model after multiple rounds of training; Step 4: Model testing: Input the preprocessed test set images into the trained strip surface defect segmentation network model to obtain the segmentation prediction map, compare it with the real annotation, and calculate the relevant evaluation indicators.

2. The method for strip surface defect few-sample segmentation based on multi-scale adaptive prototype network according to claim 1 is characterized by: The step 1 specifically includes: The strip surface defect dataset is preprocessed, and the images of the training set undergo data augmentation operations: random rotation with the angle range set to -10 degrees to 10 degrees; random scaling with the ratio range set to 0.9 to 1.1; and mirror flipping operations in the horizontal and vertical directions.

3. The method for strip surface defect few-sample segmentation based on multi-scale adaptive prototype network according to claim 1 is characterized by: The step 2 specifically includes: A feature extraction module based on ResNet-50 is constructed to extract multi-scale feature information from the surface image of the strip. The input image is first subjected to preliminary feature extraction through the convolution layer and the pooling layer. The image features are then fed into the four residual blocks of ResNet-50 to extract features of different scales. After the feature extraction module, the extracted multi-scale features are input into the adaptive prototype module, and the prototype is generated through the attention mechanism. The prototype captures the relationship between the support sample and the query sample. The module uses the features of the support sample and the query sample to generate an adaptive multi-scale prototype. The prototype can accurately match the target position of the defect and generate a fine-grained feature representation. In the module, position embedding is performed to enhance spatial information, and then the relationship between features is captured through the multi-head attention mechanism. The feature transformation is performed through the fully connected layer to generate an optimized prototype. The attention expression used is: Where Attention(Q,K,V) is the obtained attention value, Q is used to query relevant information, K is used to represent the features of each position in the input sequence, and V represents the content related to these positions. By calculating the similarity between Q and K, the model weights V and extracts the most relevant information; The multi-scale prototype features generated by the module are fed into the multi-scale feature fusion decoder. Through layer-by-layer upsampling and feature fusion, the spatial resolution of the image is gradually restored. The decoder uses feature splicing and convolution operations for feature fusion to generate an accurate segmentation prediction map.

4. The method for strip surface defect few-sample segmentation based on multi-scale adaptive prototype network according to claim 1 is characterized by: The step three specifically includes: Initialize the built model, including loading the pre-trained ResNet-50 as the feature extraction module, initializing the parameters in the adaptive prototype module and the multi-scale feature fusion decoder. The feature extraction module uses the weights pre-trained on the ImageNet dataset, and the parameters of the adaptive prototype module and the decoder are initialized according to the random initialization strategy. The binary cross entropy loss function is used. The loss function measures the difference between the predicted segmentation map and the true segmentation map. A deep dual supervision mechanism is introduced. The BCE loss function is applied to the output of each layer to supervise the final output and the feature map of the intermediate layer, which promotes the model to learn useful features at different levels. The relevant formula is as follows: Among them, y i represents the true label, represents the predicted probability of the output of the lth layer, N represents the total number of samples, L represents the total number of layers of the model, BCE l represents the binary cross entropy loss of the lth layer, and TotalLoss represents the total loss function; Two different optimizers are selected to handle the parameter updates of different modules respectively. For the other parts except the adaptive prototype module, the SGD optimizer is used, and its initial learning rate is set to 0.003, momentum is 0.9, and weight decay is 0.0001. For the adaptive prototype module, the Adam optimizer is used, and its initial learning rate is 6e-5; During the training process, the preprocessed image is input into the model. The model extracts multi-scale features through the feature extraction module, generates an adaptive prototype in the adaptive prototype module, fuses multi-scale features in the decoder, and generates a segmentation prediction map. In each training batch, the model calculates the loss between the predicted map and the real map, and updates the network weights through back propagation. The learning rate will gradually decay to ensure the stability and convergence of the training. It is defined as follows: Where lr is the current learning rate, base_lr is the initial base learning rate, curr_iter represents the current number of iterations, max_iter is the total number of iterations, and power is the exponent that controls the decay rate; As the training progresses, the model gradually focuses on learning detailed features. After each epoch, the model will be verified to evaluate the current training effect and adjust the training strategy. When the training reaches the predetermined number of rounds, the model with the best performance on the verification set is selected as the final output model and saved for subsequent testing and practical application.

5. The method for surface defect segmentation of steel strip using a multi-scale adaptive prototype network according to claim 1, characterized in that: The step 4 specifically includes: The preprocessed test set images are input into the trained model to generate defect segmentation prediction images, which are compared with the true labels and various evaluation indicators are calculated, including the average intersection-over-union ratio and the boundary area intersection-over-union ratio. The average intersection-over-union ratio is a standard metric for evaluating the overall segmentation performance of the model, and its calculation formula is: Among them, C represents the number of categories, TP is the number of true positive pixels, FP is the number of false positive pixels, and FN is the number of false negative pixels; The boundary area intersection ratio focuses more on evaluating the segmentation effect of the model in the defect boundary area. The formula is as follows: Among them, TPB, FPB and FNB represent the number of true positive, false positive and false negative pixels in the boundary area, respectively.

Images