A foreground and background decoupled prototype interactive strip steel surface defect small sample segmentation method

Abstract

The application discloses a foreground-background decoupled prototype interactive strip steel surface defect small sample segmentation method, first, the multi-scale features of a support image and a query image are obtained through a shared feature extraction network; second, the multi-receptive field structure of the support foreground features is enhanced according to a support mask, a support foreground prototype is generated, and a support background prototype is generated from a support background area; then, the background difference modeling of the query high-level features is performed by using the support background prototype to enhance the potential defect area; finally, the support foreground prototype and the query image features are interacted to obtain an enhanced prototype adapted to the query image, and the enhanced prototype is gradually fused with the query multi-scale features to output a defect segmentation prediction map of the query image. The application improves the segmentation precision under the condition of a small amount of support samples, and provides an effective scheme for solving the problem of the scarcity of labeled data in strip steel defect segmentation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically relates to a prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling, which can be used to automatically segment the strip steel surface defect region at the pixel level under the condition of only having a small number of labeled support samples. Background Technology

[0002] Strip steel is a crucial basic material in the steel industry, widely used in automobile manufacturing, shipbuilding, aerospace, bridge construction, and machinery equipment. During strip steel production and processing, factors such as raw material quality, rolling processes, equipment condition, and the external environment can easily lead to various defects on the strip steel surface, including cracks, scratches, inclusions, patches, pitting, and indentations. These defects not only affect the product's appearance quality but can also reduce the material's mechanical properties, corrosion resistance, and service reliability. Therefore, accurate detection and pixel-level segmentation of strip steel surface defects are of great significance for product quality control, production process optimization, and safety risk reduction.

[0003] Traditional methods for detecting surface defects in strip steel mainly include manual visual inspection and rule-based image processing methods. Manual visual inspection relies on the experience of inspectors and suffers from high subjectivity, low efficiency, and poor consistency. Traditional image processing methods, such as threshold segmentation, edge detection, and texture analysis, typically rely on manually designed features and are sensitive to changes in lighting, surface texture disturbances, and defect morphology, making them difficult to adapt to complex industrial production environments. In recent years, deep learning-based image segmentation methods have achieved good results in industrial defect detection, enabling more precise pixel-level localization of defect regions. However, existing deep learning segmentation models typically rely on large amounts of high-quality pixel-level labeled data for training. In actual strip steel production scenarios, defect samples occur infrequently, have uneven category distributions, and rare and novel defects are difficult to collect sufficiently. Furthermore, pixel-level labeling requires high labor costs and professional experience, resulting in very limited labeled data available for training.

[0004] To alleviate the problem of insufficient labeled data, few-shot semantic segmentation methods have been gradually introduced into the task of strip steel surface defect segmentation. These methods typically utilize a small number of labeled support images and their support masks to extract prior information about categories, guiding the model to segment similar defect regions in the query image. Most existing few-shot segmentation methods rely on the global aggregation of support set features to generate single or a small number of category prototypes. While some progress has been made, two major technical bottlenecks remain. First, strip steel surface defects are characterized by "large background, small target," large scale variations, blurred boundaries, and low contrast. Weak defect features are easily submerged by background texture during deep feature extraction and global feature aggregation. Second, similar defects differ significantly in morphology, texture, and spatial distribution. Furthermore, different defect categories may exhibit local texture similarities, making it difficult for the model to learn stable and discriminative defect representations with limited samples. This leads to insufficient utilization of supporting background information and a lack of explicit modeling of the differences between supporting background regions and high-level query features, resulting in false positives, false negatives, and inaccurate defect localization.

[0005] In summary, existing small-sample segmentation methods for strip surface defects still suffer from insufficient support for foreground prototype representation and difficulty in suppressing background texture interference. Therefore, there is an urgent need for a small-sample segmentation method for strip surface defects that can effectively enhance support for foreground prototypes, explicitly model background differences, and improve the interaction between prototypes and query features, even with a limited number of labeled samples. Summary of the Invention

[0006] To address the problems of existing technologies, this invention provides a prototype-interactive small-sample segmentation method for surface defects in strip steel with foreground-background decoupling. This invention utilizes support masks to enhance the multi-scale structural representation of the defect foreground region and models the differences in background interference in the query image using support background information. Based on this, it establishes the interaction relationship between the support foreground prototype and the query features, thereby significantly improving the accuracy and robustness of defect segmentation.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps: A prototype interactive strip surface defect small-sample segmentation method with foreground-background decoupling is proposed for segmenting strip surface defect images with a limited number of labeled support samples. The method includes the following steps: S1: Obtain the supporting image, supporting mask, and query image, and perform preprocessing.

[0008] In a small-sample segmentation task for surface defects on strip steel, support images, support masks, and query images are obtained. The support images are strip steel surface images containing defect regions of the target category. The support masks are pixel-level annotations that correspond one-to-one with the spatial locations of the support images. The query images are strip steel surface images of the target defect category for which pixel-level prediction of defect regions is to be performed. Subsequently, the support images and query images undergo preprocessing, primarily including size normalization, pixel value standardization, and data augmentation during the training phase. S2: Extract multi-scale features from the supporting and query images.

[0009] The preprocessed support image and query image are input into a shared feature extraction network to obtain multi-scale features of the support image and multi-scale features of the query image, respectively. S3: Enhance supporting foreground features and generate supporting foreground prototypes.

[0010] The support foreground region in the multi-scale features of the support image is constrained according to the support mask, and the support foreground features are enhanced by multi-receptive field structure to strengthen the structural expression of defect regions of different scales and shapes. Then, a support foreground prototype is generated based on the enhanced support foreground features. S4: Generate supporting background prototypes and perform query background difference modeling.

[0011] The supporting background region is determined based on the supporting mask, a supporting background prototype is generated based on the background region, and the supporting background prototype is used to model the background difference of the high-level features of the query image to obtain the query features after background decoupling, so as to reduce the interference of complex background texture on defect segmentation. S5: Enable prototype interaction between the foreground prototype and the query features.

[0012] The query features after decoupling the supporting foreground prototype from the background are subjected to prototype interaction, which enables the supporting foreground prototype to dynamically combine the defect-related context information in the query image to obtain an enhanced prototype that is adapted to the query image, thereby improving the supporting prototype's ability to match defects of different shapes in the query image. S6: Perform progressive multi-scale fusion and output a defect segmentation prediction map; The enhanced prototype is progressively fused with the multi-scale features of the query image, that is, upsampling and feature fusion are performed step by step from the high-level scale to the low-level scale to generate multi-scale query prediction results, and finally output the defect segmentation prediction map of the query image.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention addresses the challenges of scarce labeled samples, complex defect morphologies, and strong background texture interference in small-sample segmentation of steel strip surface defects. It proposes a prototype-interactive segmentation method with foreground-background decoupling. This method utilizes support masks to constrain the defect foreground region in the support image and enhances the support foreground features with a multi-receptive field structure, enabling the model to extract more comprehensive defect structural information from a limited number of support samples. Compared to directly generating support foreground prototypes using masked average pooling, this invention better preserves the multi-scale morphology, local texture, and fine-grained structural features of the defect region, reducing the inadequacy of a single prototype in representing complex defects. This improves the model's ability to recognize small-scale, low-contrast, blurred-boundary, and irregularly shaped defects.

[0014] Meanwhile, this invention fully utilizes background region information in the supporting image to generate a supporting background prototype, and models the background differences between it and the high-level features of the query image. This allows the model to explicitly perceive the difference between normal background textures and potential defect regions, thereby reducing the interference of complex background factors such as rolling textures, oxidation marks, water spots, oil stains, and uneven lighting on defect judgment, and reducing false detections and missed defects. Furthermore, this invention establishes an interaction relationship between the supporting foreground prototype and the query features after decoupling from the background. This allows the supporting foreground prototype to adaptively enhance itself by incorporating defect-related contextual information from the query image, mitigating the prototype mismatch problem caused by differences in defect scale, shape, and texture between the supporting samples and the query image. This invention can simultaneously utilize high-level semantic information and low-level detail information to improve the accuracy of defect region localization and the integrity of segmentation boundaries, thereby reducing the model's dependence on large-scale pixel-level labeled data and enhancing its applicability in novel defects, rare defects, and industrial scenarios with limited samples. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0016] Figure 2 This is a network structure diagram of the prototype interactive strip steel surface defect small sample segmentation method of the present invention, which decouples the foreground and background.

[0017] Figure 3 This is a schematic diagram of the structure for enhancing the support features with a multi-receptive field structure in this invention.

[0018] Figure 4 This is a schematic diagram illustrating the query background difference modeling in this invention.

[0019] Figure 5 This is a schematic diagram illustrating hierarchical prototype interaction in this invention.

[0020] Figure 6 This is a visualization of the segmentation results from the FSSD-12 task. Detailed Implementation

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0022] The method and process are as follows Figure 1 As shown, the implementation steps of this invention are as follows: Step 1: Obtain the supporting image, supporting mask, and query image, and perform preprocessing.

[0023] In the small-sample segmentation task of strip steel surface defects, the support image and its corresponding support mask are used to provide a small amount of prior annotation of the target defect category, while the query image is the strip steel surface image to be segmented. Due to the scale variations, imaging condition differences, and background texture perturbations present in real-world industrial scenarios, directly inputting the original image into the network can easily lead to insufficient adaptability of the model to defects of different sizes, brightness levels, and spatial distributions. Therefore, before feature extraction, both the support image and the query image need to undergo the same preprocessing operations to meet the subsequent network input requirements. Specific steps include: (1a) Obtain the support image, support mask, and query image from the small sample segmentation task of strip surface defects. The support image is denoted as... Supports masking as The image to be queried is denoted as In the K-shot task, the support set can be represented as: in, Indicates the number of supporting samples. Indicates the first Zhang supports images, Indicates the first Zhang supports the support mask corresponding to the image.

[0024] (1b) The size of the supporting image and the query image are normalized and the pixel value is standardized so that the image input size and pixel distribution meet the input requirements of the shared feature extraction network.

[0025] (1c) Perform data augmentation processing on the support image and the query image, including random scaling, random rotation, random Gaussian blur, random horizontal flip and random cropping; the angle range of the random rotation is set to -10 degrees to 10 degrees, and the ratio range of the random scaling is set to 0.9 to 1.1; the random cropping is used to crop the support image and the query image to a preset training size.

[0026] (1d) The support mask is transformed synchronously with the corresponding support image during scaling, rotation and cropping, and the nearest neighbor interpolation method is used during size adjustment to keep the pixel label values ​​from being mixed.

[0027] Step 2: Extract multi-scale features from the supporting image and the query image.

[0028] Surface defects in strip steel exhibit different characteristics at different feature levels. Higher-level features contain stronger semantic information, which is beneficial for determining the overall location of the defect region; lower-level features contain more edge, texture, and spatial detail information, which is beneficial for recovering defect boundaries and local structures. Therefore, this invention employs a shared feature extraction network to extract features at multiple scales from both the support image and the query image, enabling subsequent prototype generation, background difference modeling, and segmentation prediction to be performed on both images within a shared feature space.

[0029] refer to Figure 2 As shown, the shared feature extraction network in this step specifically includes a backbone feature extraction module and a feature pyramid fusion module, wherein: (2a) The preprocessed support image and query image are respectively input into the shared backbone feature extraction module for feature extraction to obtain multi-level features of the support image and query image. The backbone feature extraction module can use VGG network, ResNet network, etc., to extract hierarchical features from the input image layer by layer from low-level texture information to high-level semantic information.

[0030] (2b) Input the multi-level features output by the backbone network into the feature pyramid fusion module. The feature pyramid fusion module maps the multi-level features to the same channel dimension and fuses high-level semantic information and low-level detail information to obtain multi-scale features supporting the image and multi-scale features of the query image, respectively represented as: in, Indicates the first Support image features at each scale Indicates the first Query image features at various scales, Indicates the feature scale level. .

[0031] Step 3: Perform multi-receptive field structure enhancement on the supporting foreground features and generate a supporting foreground prototype.

[0032] Because surface defects in strip steel often exhibit characteristics such as large scale variations, irregular shapes, blurred boundaries, and complex local textures, directly using support masks to perform average pooling on support features to obtain the foreground prototype easily weakens the fine-grained structural information within the defect region, resulting in insufficient expressive power of the generated support foreground prototype. Therefore, such as... Figure 3 As shown, this invention enhances the support foreground features with a multi-receptive field structure under support mask constraints, enabling the network to simultaneously perceive local detail features of defect regions and structural context information over a larger range, and further generates more representative support foreground prototypes through conditional prototype pooling.

[0033] The multi-receptive-field foreground enhancement process in step 3 includes: (31a) Adjust the support mask to match the first The supporting image features of each scale have the same spatial size, resulting in the first... Support foreground mask at each scale The supporting foreground mask is then used to perform foreground constraints on the supporting image features to obtain supporting foreground constrained features. : in, Indicates the first Support image features at each scale This is for element-wise multiplication.

[0034] (31b) The supporting foreground constraint features are input into multiple convolutional branches with different receptive fields to obtain multi-receptive field features. The different receptive field branches are used to capture local details and larger-scale structural context information of the defect region. The multiple convolutional branches include 3×3 dilated convolutional branches with dilation rates of 1, 2, and 3, respectively. Subsequently, the features output by the convolutional branches with different dilation rates are concatenated and fused to obtain fused multi-receptive field features. A spatial attention map is then generated based on the fused multi-receptive field features, as follows: in, Indicates the expansion rate of Dilated convolution operation, Indicates the first Features of the receptive field branch output This indicates a channel splicing operation. Represents the convolution fusion function. This indicates the multi-receptor field characteristics after fusion. Represents the attention mapping function, This represents the Sigmoid activation function. Represents a spatial attention map, and .

[0035] (31c) Perform residual enhancement on the supporting foreground features based on the spatial attention map to obtain the enhanced supporting foreground features: in, This indicates the enhanced support prospect features. Indicates the residual enhancement coefficient. Indicates the learnable weight parameters. This indicates the multi-receptor field characteristics after fusion. Representing a spatial attention map, Represents the feature scale level, and .

[0036] Furthermore, after obtaining the enhanced supporting foreground features, step 3 generates supporting foreground prototypes through conditional prototype pooling, specifically including: (32a) Conditional pooling weights are generated based on the enhanced support foreground features to reflect the importance of different foreground locations to prototype generation, enabling the model to adaptively focus on more representative defect areas instead of performing a simple equal-weighted average of all foreground pixels.

[0037] (32b) Perform weighted pooling on the enhanced support foreground features according to the conditional pooling weights to obtain the first... Support prospect prototype at various scales : in, Indicates the first Spatial location at each scale is Conditional pooling weights, Indicates the first Location space at each scale is Enhanced support for future features, This represents a preset positive number used to prevent the denominator from being zero, and .

[0038] Step 4: Generate a supporting background prototype and model the background differences for high-level features of the query image.

[0039] The normal areas of strip steel surfaces often contain complex background factors such as rolling textures, oxidation marks, water spots, oil stains, uneven lighting, and surface undulations. These background textures may visually resemble real defects, easily leading to false detections. Existing few-sample segmentation methods typically focus on matching the supporting foreground prototype with query features, while underutilizing supporting background information. Therefore, this invention further determines the supporting background region based on the supporting mask, generates a supporting background prototype, and uses this prototype to model background differences in high-level features of the query image, highlighting potential defect areas that differ from the supporting background, such as... Figure 4 As shown, this enhances the model's ability to perceive the differences between normal background textures and potential defect areas, thereby reducing the interference of complex background textures on defect judgment.

[0040] The implementation of background prototype generation and background difference modeling in this step includes: (4a) Determine the supporting background region based on the supporting mask, and based on the supporting image features Generate a background-supporting prototype from the feature responses within the supporting background region: in, Indicates the first Supporting background prototypes at various scales Indicates the first Spatial location at each scale is Supporting image features, Indicates the first Spatial location at each scale is Supports background mask values. This represents a preset positive number used to prevent the denominator from being zero, and .

[0041] (4b) Calculate high-level features of the query image , The difference features between the original and the supporting background prototype are used to reduce the interference of shallow texture noise on the identification of defect areas.

[0042] in, Indicates the first Background difference features at each scale, and , Indicates the first Supporting background prototypes at various scales.

[0043] (4c) Based on the high-level features of the query image , and the background difference features Dynamic gating weights are generated to adaptively control the strength of background difference information injected into query features, so as to avoid excessive background difference response from destroying the original semantic expression of query features.

[0044] (4d) The background difference features are weighted by dynamic gating weights and then injected into the high-level features of the query image as residuals to obtain the query features after background decoupling: in, Indicates the first Query features after background decoupling at each scale Indicates the first Query image features at various scales, Indicates the background differential injection intensity. Indicates the background difference gating weight. For element-wise multiplication, and .

[0045] Step 5: By supporting prototype interaction between the query features after the foreground prototype and the background are decoupled, an enhanced prototype adapted to the query image is obtained.

[0046] Because there may be differences in defect scale, defect morphology, texture representation, and imaging conditions between the supporting image and the query image, directly using a fixed supporting foreground prototype for query image segmentation can easily lead to a mismatch between the supporting prototype and the query defect region. To improve the adaptability of the supporting prototype to the query image content, this invention constructs an interaction relationship between the supporting foreground prototype and the query features after decoupling from the background, such as... Figure 5 The diagram shows how the supporting foreground prototype can obtain defect-related contextual information from the query image and further obtain an enhanced prototype adapted to the query image by modeling the relationships between prototypes.

[0047] This step, which supports prototype interaction of query features after decoupling the foreground prototype from the background, specifically includes: (51a) Construct the first based on the supporting prospect prototype Support prototype representation at each scale And the query features after decoupling the background , Spatial flattening is performed, while for shallow scales where background difference modeling is not performed, i.e. It can retrieve query image features at the corresponding scale. As a query feature involved in the interaction, the first... Query feature representation at each scale : in, This indicates a flattening operation.

[0048] (51b) The supporting prototype representation and the query feature representation are subjected to attention interaction, so that the supporting foreground prototype can obtain contextual information related to the defect region from the query image features, and dynamically adjust according to the context response to improve its adaptability to the query image, thus obtaining the interactive prototype representation. : in, Indicates by the first Support prototype representation composed of support foreground prototypes at various scales. This indicates prototype embedding. Indicates the query location code. Represents the cross-attention interaction function. .

[0049] Furthermore, after obtaining the prototype representation after interaction, the prototype representation is modeled for inter-prototype relationships and subjected to nonlinear feature transformation to obtain an enhanced prototype adapted to the query image. Specific implementation includes: (52a) Representation of the prototype after the interaction Self-attention modeling is performed to enhance the expression of relationships between different supporting prototypes, thereby learning the correlation between different morphological and texture patterns within defect categories.

[0050] in, This represents the prototype representation after self-attention modeling. This indicates a self-attention operation. .

[0051] (52b) Perform a feedforward transformation on the prototype representation after self-attention modeling, and obtain an enhanced prototype adapted to the query image through pooling operation. : in, This indicates a pooling operation. Indicates a feedforward network. .

[0052] Step 6: Output the defect segmentation prediction map of the query image through progressive multi-scale fusion.

[0053] Surface defects in strip steel exhibit different feature representations at different scales. High-level features are beneficial for semantic localization of defect regions but have lower spatial resolution; low-level features contain more edge and texture details but are easily affected by background noise. Therefore, this invention fuses the query enhancement prototype with multi-scale features of the query image and performs upsampling and feature fusion stepwise from high-level to low-level scales to simultaneously utilize high-level semantic information and low-level detail information, thereby improving the completeness and boundary accuracy of defect region segmentation results.

[0054] The progressive multi-scale fusion process in this step includes: (6a) will be the first The query image features at each scale are concatenated and fused with the corresponding scale's query enhancement prototype. For shallow scales, i.e. Using query image features without background difference modeling For high-level characteristics, i.e. Using query image features modeled with background differences Thus, prototype-guided fusion features are obtained. ; in, This indicates that the query adaptive enhancement prototype is extended to the same spatial size operation as the query image features. Indicates the first A multi-scale adaptive enhancement prototype for queries. This indicates a channel splicing operation. Represents the convolution fusion function. ; (6b) Perform upsampling fusion step by step from high-level scale to low-level scale to obtain step-by-step fused features: in, Indicates an upsampling operation. This represents a step-by-step fusion function. This represents the result prediction graph from the previous layer; (6c) Generate multi-scale query prediction results based on the hierarchical fusion features, and output the final defect segmentation prediction map. Each pixel position corresponds to the prediction result of whether that pixel in the query image belongs to a defective foreground or a normal background.

[0055] Step 7: Optimize network parameters based on training loss.

[0056] To improve the segmentation accuracy and training stability of the network under small sample conditions, this invention calculates the training loss during the training phase based on the defect segmentation prediction map of the query image and the ground truth mask of the query image, and optimizes the network parameters based on the training loss to obtain a trained small sample segmentation network for steel strip surface defects. The training loss includes the final segmentation loss, multi-scale deep supervision loss, prototype-supported auxiliary supervision loss, and prototype consistency constraint loss. The total loss function is expressed as: in, Represents the total loss function. Indicates the loss from partitioning. This indicates multi-scale deep monitoring loss. This indicates support for prototype-assisted supervision loss. This represents the loss due to prototype consistency constraints. , and These represent the weighting coefficients of the corresponding loss terms; This represents the final defect segmentation prediction map. This indicates the actual mask of the query image. Represents cross-entropy loss, This indicates Dice's loss. Indicates the Dice loss weights. Indicates the first Segmentation prediction maps at various scales, and .

[0057] To demonstrate the advantages of this invention, the specific implementation effects can be further illustrated by the following simulation results: I. Simulation Conditions The segmentation method designed in the simulation was based on Python 3.10 and PyTorch 2.8 frameworks; the machine used to train the segmentation network in the simulation was equipped with an Intel(R) Core (TM), 2.5GHz CPU, 24GB RAM and one GeForce RTX 3090 GPU; The simulation uses strip defect images and masks obtained from the publicly available FSSD-12 dataset. A cross-validation experimental setup is used, and the defects are divided into three Folds, each containing four types of defects.

[0058] II. Simulation Content Under the above simulation conditions, the method of this invention was used to segment the obtained query image. The segmentation evaluation metrics used were the mean Intersection over Union (mIoU) and the foreground / background Intersection over Union (FBIoU). The mean IoU is a metric that measures the degree of region overlap; it calculates the ratio of the intersection to the union of the predicted and real regions. The value of this metric ranges from [0, 1], with higher values ​​indicating better performance. The foreground / background intersection over Union measures the model's overall segmentation ability for defective foreground and normal background. It first treats all defective images as the foreground category and non-defective pixels as the background category, then calculates the foreground and background intersection over Unions separately, and finally averages them. The value of this metric ranges from [0, 1], with higher values ​​indicating better model differentiation between foreground defects and background regions.

[0059] Tables 1 and 2 respectively show the performance comparison of the method of the present invention and the current mainstream few-sample defect segmentation methods on the FSSD-12 task.

[0060] Table 1. Performance comparison of different small-sample segmentation methods on the FSSD-12 (1-shot) task. Table 2 Performance comparison of different small-sample segmentation methods on the FSSD-12 (5-shot) task As shown in Tables 1 and 2, the present invention achieves optimal results across all performance metrics on the 1-shot and 5-shot tasks of the FSSD-12 dataset. In the 1-shot task, using VGG-16 as the backbone network, the method of the present invention achieves an average mIoU of 67.66% and an FB-IoU of 79.78%, respectively higher than the second-best method by 2.92% and 1.21%. When using ResNet-50 as the backbone network, the method of the present invention achieves an average mIoU of 68.69% and an FB-IoU of 80.96%, respectively higher than the second-best method by 2.31% and 1.14%. In the 5-shot task, the method of the present invention maintains its superior performance. When using VGG-16 and ResNet-50 as backbone networks, the method of the present invention achieves optimal results in both average mIoU and FB-IoU metrics. The above results show that even with only 1 or 5 supporting samples and minimal annotations, the method of the present invention can still effectively utilize the foreground and background information of defects in the supporting images to improve the segmentation accuracy of defect regions in the query images.

[0061] To more intuitively demonstrate the performance advantages of this invention, Figure 6The visualization results on the FSSD-12 dataset are presented. From top to bottom, the figure shows the supporting image, the ground truth mask of the supporting image, the query image, the ground truth mask of the query image, the prediction results of existing methods, and the prediction results of the method of this invention. Figure 5 As can be seen, when faced with different types of strip surface defects, the method of this invention can generate more complete, continuous, and more accurately defined defect prediction regions. For defects with low contrast, blurred boundaries, or complex background textures, existing methods are easily affected by background factors such as rolling textures, water stains, patches, and uneven lighting, leading to misclassification of some background areas as defects or insufficient response to real defect areas. In contrast, this invention, by introducing supporting background information and modeling background differences in high-level query features, can more effectively distinguish between normal background textures and potential defect areas, thereby reducing false background detections and improving the accuracy of defect area localization.

[0062] Meanwhile, existing methods often exhibit issues such as breakage, missed detections, or incomplete boundaries in the prediction results for slender scratches, minute defects, and irregularly shaped defect regions. This invention utilizes a multi-receptive field structure enhancement supporting the foreground region and a support-query prototype interaction mechanism, enabling the supporting foreground prototype to more fully express the multi-scale structure and local texture features of the defect. Therefore, this invention maintains good structural coherence and regional integrity even in scenarios with small defects, weak texture defects, and defects with significant morphological variations.

Claims

1. A prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling, characterized in that, Includes the following steps: S1: Obtain the support image, support mask, and query image in the small sample segmentation task of strip steel surface defects, and preprocess the support image and query image; wherein, the support image is a strip steel surface image containing the defect region of the target category, the support mask is a pixel-level annotation map that corresponds one-to-one with the spatial position of the support image, and the query image is a strip steel surface image under the same target defect category for which pixel-level prediction of defect region is to be performed. S2: Input the preprocessed support image and query image into the shared feature extraction network to obtain multi-scale features of the support image and multi-scale features of the query image, respectively; S3: Based on the support mask, perform multi-receptive field structure enhancement on the support foreground features in the multi-scale features of the support image, and generate a support foreground prototype based on the enhanced support foreground features; S4: Determine the supporting background region based on the supporting mask, generate a supporting background prototype based on the background region, and use the supporting background prototype to model the background difference of the high-level features of the query image to obtain the query features after background decoupling. S5: Perform prototype interaction between the supported foreground prototype and the query features after decoupling from the background to obtain an enhanced prototype that matches the query image. S6: Perform progressive multi-scale fusion of the enhanced prototype with the multi-scale features of the query image to output a defect segmentation prediction map of the query image.

2. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 1, characterized in that, In S1, the preprocessing process includes: The support image and query image undergo size normalization, pixel value standardization, and data augmentation during the training phase. The data augmentation includes random scaling, random rotation, random Gaussian blur, random horizontal flipping, and random cropping. The random rotation angle range is set to -10 degrees to 10 degrees, and the random scaling ratio range is set to 0.9 to 1.

1. The random cropping is used to crop the support image and query image to a preset training size. The support mask is synchronized with the corresponding support image during scaling, rotation, and cropping, and nearest-neighbor interpolation is used during size adjustment to prevent pixel label values ​​from mixing.

3. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 1, characterized in that, The shared feature extraction network in S2 includes a backbone feature extraction module and a feature pyramid fusion module, wherein: The backbone feature extraction module is used to extract features from the support image and the query image respectively, and obtain multi-level features of the support image and the query image. The feature pyramid fusion module is used to map the multi-level features to the same channel dimension and fuse high-level semantic information and low-level detail information to obtain multi-scale features supporting the image and multi-scale features of the query image, respectively represented as: in, Indicates the first Support image features at each scale Indicates the first Query image features at various scales, Indicates the feature scale level. .

4. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 1, characterized in that, The multi-receptive-field structure enhancement in S3 includes: S31a, adjust the support mask to match the first The supporting image features of each scale have the same spatial size, resulting in the first... Support foreground mask at each scale The supporting foreground mask is then used to perform foreground constraints on the supporting image features to obtain supporting foreground constrained features. : in, Indicates the first Support image features at each scale For element-wise multiplication; S31b, the supporting foreground constraint features are input into convolutional branches of multiple different receptive fields to obtain multi-receptive field features, which are then spliced ​​and fused to obtain fused multi-receptive field features and generate a spatial attention map. S31c, Perform residual enhancement on the supporting foreground features based on the spatial attention map to obtain the enhanced supporting foreground features: in, This indicates the enhanced support prospect features. Indicates the residual enhancement coefficient. Indicates the learnable weight parameters. This indicates the multi-receptor field characteristics after fusion. Representing a spatial attention map, Represents the feature scale level, and .

5. The prototype interactive strip steel surface defect small sample segmentation method for foreground-background decoupling according to claim 4, characterized in that, In step S3, after obtaining the enhanced supporting foreground features, a supporting foreground prototype is generated through conditional prototype pooling, specifically including: S32a, generate conditional pooling weights based on the enhanced support foreground features; S32b, perform weighted pooling on the enhanced support foreground features according to the conditional pooling weights to obtain the first... Support prospect prototype at various scales : in, Indicates the first Spatial location at each scale is Conditional pooling weights, Indicates the first Location space at each scale is Enhanced support for future features, This represents a preset positive number used to prevent the denominator from being zero, and .

6. The prototype interactive strip steel surface defect small sample segmentation method for foreground-background decoupling as described in claim 1, characterized in that, The support for background prototype generation and background difference modeling in S4 includes: S4a, determine the supporting background region based on the supporting mask, and generate a supporting background prototype based on the feature response of the supporting image features within the supporting background region: in, Indicates the first Supporting background prototypes at various scales Indicates the first Spatial location at each scale is Supporting image features, Indicates the first Spatial location at each scale is Supports background mask values. This represents a preset positive number used to prevent the denominator from being zero, and ; S4b, Calculate high-level features of the query image , The difference features between the original and the supporting background prototype are used to reduce the interference of shallow texture noise on the judgment of defect areas; in, Indicates the first Background difference features at each scale, and ; S4c, based on the high-level features of the queried image , Dynamic gating weights are generated based on the background difference features; S4d, the background difference features are weighted by dynamic gating weights and then injected into the high-level features of the query image in a residual manner to obtain the query features after background decoupling: in, Indicates the first Query features after background decoupling at each scale Indicates the first Query image features at various scales, Indicates the background differential injection intensity. Indicates the background difference gating weight. For element-wise multiplication, and .

7. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 1, characterized in that, The S5 will support prototype interaction of query features after the foreground prototype and background are decoupled, specifically including: S51a, construct the first based on the supporting foreground prototype Supports prototype representations at various scales and decouples query features from the background. , Spatial flattening is performed, while for shallow scales where background difference modeling is not performed, i.e. It can retrieve query image features at the corresponding scale. As a query feature involved in the interaction, the first... Query feature representation at each scale : in, Indicates the flattening operation; S51b, the supporting prototype representation and the query feature representation are subjected to attention interaction, so that the supporting foreground prototype can obtain contextual information related to the defect region from the query image features, and the interactive prototype representation is obtained. : in, Indicates by the first Support prototype representation composed of support foreground prototypes at various scales. This indicates prototype embedding. Indicates the query location code. Represents the cross-attention interaction function, and .

8. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 7, characterized in that, After obtaining the prototype representation after interaction, the prototype representation is modeled for inter-prototype relationships and nonlinear feature transformation is performed to obtain an enhanced prototype adapted to the query image, specifically including: S52a performs self-attention modeling on the prototype representation after interaction to enhance the expression of the relationship between different supporting prototypes; S52b performs a feedforward transformation on the prototype representation after self-attention modeling and obtains an enhanced prototype adapted to the query image through pooling operations.

9. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 1, characterized in that, The progressive multi-scale fusion in S6 includes: S6a, will the first The query image features at each scale are concatenated and fused with the corresponding scale's query enhancement prototype. For shallow scales, i.e. Using query image features without background difference modeling For high-level characteristics, i.e. Using query image features modeled with background differences Thus, prototype-guided fusion features are obtained. ; in, This indicates that the query adaptive enhancement prototype is extended to the same spatial size operation as the query image features. Indicates the first A prototype for query enhancement at various scales. This indicates a channel splicing operation. Represents the convolution fusion function. ; S6b performs upsampling fusion step-by-step from high-level to low-level scales to obtain step-by-step fused features: in, Indicates an upsampling operation. This represents a step-by-step fusion function. This represents the result prediction graph from the previous layer; S6c, generate multi-scale query prediction results based on the progressively fused features, and output the final defect segmentation prediction map.

10. The prototype interactive strip steel surface defect small sample segmentation method with foreground-background decoupling as described in claim 1, characterized in that, During the training phase, the training loss is calculated based on the defect segmentation prediction map of the query image and the ground truth mask of the query image. The network parameters are then optimized based on this training loss to obtain a trained small-sample segmentation network for steel strip surface defects. The training loss includes the final segmentation loss, multi-scale deep supervision loss, prototype-supported auxiliary supervision loss, and prototype consistency constraint loss. The total loss function is expressed as: in, Represents the total loss function. Indicates the loss from partitioning. This indicates multi-scale deep monitoring loss. This indicates support for prototype-assisted supervision loss. This represents the loss due to prototype consistency constraints. , and These represent the weighting coefficients of the corresponding loss terms; This represents the final defect segmentation prediction map. This indicates the actual mask of the query image. Represents cross-entropy loss, This indicates Dice's loss. Indicates the Dice loss weights. Indicates the first Segmentation prediction maps at various scales, and .

Images