Automatic labeling method for rolled metal surface defects based on multi-task adaptive model

Abstract

The present invention discloses a method for automatic labeling of rolled metal surface defects based on a multi-task adaptive model, including: S1, collecting rolled metal surface images, performing data labeling after preprocessing and enhancement, and constructing a rolled metal surface defect data set; S2, constructing a multi-task adaptive model, and training it using the data set to obtain a defect automatic labeling model; S3, inputting the metal surface image to be labeled into the defect automatic labeling model, and outputting the surface defect labeling result by executing the target detection task. The method combines a multi-task learning framework to realize the joint training of target detection and instance segmentation tasks for metal surface defects in the rolling process, and focuses on the target detection task in the reasoning stage to improve the labeling efficiency. The labeling method of the present invention significantly reduces labor costs, can be widely used in industrial defect detection and data automatic labeling scenarios of rolled metal, and is suitable for defect image automatic labeling tasks in complex industrial environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial defect detection and image processing, and in particular relates to an automatic labeling method for rolled metal surface defects based on a multi-task adaptive model (MAM). Background Art

[0002] The deep learning-based surface defect detection method for rolled metal relies on a large number of accurately annotated defect image datasets to support the in-depth learning of defect features in model training, so as to achieve accurate positioning and classification of different types of industrial defects. The scale and quality of the dataset are crucial to the performance and robustness of the target detection algorithm. Large-scale and high-quality datasets can provide a rich sample distribution for model training, thereby significantly improving the accuracy and reliability of the model in actual detection tasks. However, obtaining such large-scale, high-quality datasets is a huge challenge. Low-quality data with incorrect annotations or lack of detailed annotation information may cause the model to learn incorrect feature patterns or fail to fully extract target features, thereby affecting the overall performance of the target detection task. Famous datasets such as COCO and ImageNet have millions of high-quality samples, which have undergone a rigorous screening and annotation process to ensure the high quality of the dataset. It can be seen that large-scale and accurately annotated datasets are crucial for training high-performance target detection models.

[0003] Currently, the method of manually annotating defect data sets has significant disadvantages and limitations. The manual annotation process is labor-intensive, and annotators need to check images one by one, accurately draw bounding boxes or perform pixel-level annotations. However, the amount of data generated daily in industrial production processes may reach TB levels. Manual annotation is difficult to meet such efficient data needs and is extremely costly. In addition, the quality of manual annotation is difficult to guarantee, and the annotation results may lack stability due to the subjective judgment and fatigue of the annotators. Differences in annotation styles between annotators can also lead to inconsistent annotation results. Long-term and high-intensity annotation work further aggravates fatigue accumulation and affects the quality of annotation.

[0004] Defect detection in the rolling metal production process has its own unique technical challenges. First, due to the high-intensity deformation and stretching involved in the rolling process, it is easy to cause many different types of defects on the metal surface, such as scratches, dents, pores, impurities and oxide spots. These defects are complex in shape, unevenly distributed and have significant size differences. Some of them are difficult to distinguish visually, which poses a great challenge to traditional single detection methods. In addition, the continuity and production speed of the rolling process require real-time defect detection to prevent unqualified products from entering subsequent processing links. Based on the above characteristics, traditional single-task labeling methods are difficult to fully learn the characteristics of different defect types, which can easily lead to insufficient model generalization ability and difficulty in adapting to diverse detection needs.

[0005] In response to the above problems, the Chinese invention patent with the publication (announcement) number CN118279660A proposes a method for automatic labeling of intelligent driving data based on deep learning. The method includes the following steps: Step S1, obtaining and preprocessing intelligent driving data, expanding the preprocessed data through a data enhancement method, and then sending the data to the first-stage convolutional neural network. Multi-scale features are extracted through the backbone network, and these features are sent to the naive detector, and the preliminary first-level box positioning coordinates are output after post-processing; Step S2, expanding the positioning range of the first-level box and obtaining the external expansion coordinates. This invention improves the accuracy of the labeling results and improves the efficiency and quality of data labeling.

[0006] Another Chinese invention patent with the publication (announcement) number CN118334429B proposes a method for fast labeling of industrial X-ray inspection defect images combined with the Deeplabv3plus network. This method includes the following steps: Step 1, use Photoshop to mark, crop and filter part of the data; Step 2, divide the processed data into a training set and a validation set, and use the Deeplabv3plus network for training; Step 3, test the average intersection-over-union ratio and accuracy on the validation set, and if the standard is met, automatically detect and label the remaining data; Step 4, fine-tune the labeled data to improve the labeling quality.

[0007] Although the above two inventions have realized some functions of automatic labeling, there are still some technical bottlenecks. First, they rely on a single network and a small amount of labeled data to assist in the labeling of unlabeled data. The labeling accuracy is limited, and the labeling quality is easily affected by the model prediction error. In a single network framework, if errors occur in the initial labeling, these errors may accumulate and spread in subsequent tasks, affecting the accuracy of labeling. In addition, these methods are usually designed and trained for specific labeling tasks, lack flexibility, and are difficult to adapt to a variety of different labeling needs. The differences between fields limit the effect of transfer learning. Transfer learning requires a certain similarity between the source field and the target field. When the difference between the two is too large, the transfer learning effect of a single task is significantly reduced, and the practical significance of labeling may even be lost. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the method for automatic labeling of rolled metal surface defects based on a multi-task adaptive model provided by the present invention solves the problems that the existing related methods rely on a single network or a small amount of labeled data for labeling, and the labeling accuracy is limited. At the same time, the model transfer learning effect obtained by training for a specific labeling task is insufficient, and it is difficult to realize the detection of different defect types.

[0009] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for automatically marking surface defects of rolled metal based on a multi-task adaptive model, comprising the following steps:

[0010] S1. Collect rolled metal surface images, perform data annotation after preprocessing and enhancement, and construct a rolled metal surface defect dataset;

[0011] S2. Build a multi-task adaptive model and train it using a rolled metal surface defect dataset to obtain an automatic defect annotation model.

[0012] The training includes sequential single training and adaptive re-labeling cycle mechanism training;

[0013] The defect automatic annotation model includes a backbone network, a feature extraction and fusion module and a multi-task prediction head; the backbone network is used to extract feature information of the input image from low to high levels; the feature extraction and fusion module is used to extract multi-layer feature maps with global context information and long-range dependencies based on the feature information extracted by the backbone network, and perform cross-level adaptive fusion to extract ROI features; the multi-task prediction head is used to use the extracted ROI features to simultaneously perform target detection and instance segmentation prediction to obtain annotation results;

[0014] S3. Input the metal surface image to be annotated into the automatic defect annotation model, and output the surface defect annotation result by executing the target detection task.

[0015] Furthermore, in step S2, the backbone network is a pre-trained ResNeXt-50 network, including an initial convolutional layer, a maximum pooling layer, and four stages of residual blocks connected in sequence;

[0016] The method for extracting feature information from the backbone network is specifically as follows:

[0017] S21, processing the input image through an initial convolutional layer to obtain a first feature map;

[0018] S22, processing the first feature map through a maximum pooling layer to obtain a first feature map;

[0019] S23, processing the first feature map through four stages of residual blocks in sequence to obtain corresponding second feature maps, third feature maps, fourth feature maps, and fifth feature maps;

[0020] S24, combining the first to fifth feature maps into feature maps of different semantic levels and spatial resolutions as feature information extracted from low-level to high-level.

[0021] Furthermore, in step S2, the feature extraction and fusion module includes a residual attention feature pyramid network, a cross-level adaptive fusion network and a region proposal network;

[0022] The method for extracting ROI features by the feature extraction and fusion module is specifically as follows:

[0023] S25. In the residual attention feature pyramid network, based on the introduced residual structure attention mechanism, the high-level semantic information in the feature information is fused with the low-level spatial information to obtain a multi-layer feature map that captures global context information and long-range dependencies.

[0024] S26, using a cross-level adaptive fusion network to fuse features of different scales on multiple layers of feature maps to obtain fused feature maps of each layer;

[0025] S27. Use the region proposal network to generate candidate regions from the fused feature map, and suppress the generated candidate regions by non-maximum value to obtain ROI features of fixed size in each fused feature map.

[0026] Furthermore, the step S25 includes the following sub-steps:

[0027] S25-1, unifying the number of channels of each layer of feature maps in the feature information to obtain feature maps with unified dimensions;

[0028] S25-2, perform context attention calculation on each layer of feature maps with unified dimensions, and multiply them by the value matrix to obtain a feature map after attention mapping;

[0029] S25-3, adjust the feature map after each attention mapping to the shape of the input feature map through the reshape function, and perform residual connection with the original feature to obtain a multi-layer feature map that captures global context information and long-range dependencies;

[0030] Among them, the feature map of the i-th layer It is expressed as:

[0031]

[0032] In the formula, represents a feature map of uniform dimension, represents the query matrix for contextual attention computation, represents the key matrix for contextual attention computation, A matrix of values ​​representing the contextual attention computation, represents the scaling factor, Represents the reshape function, and the superscript T represents the transpose operator.

[0033] Furthermore, in step S26, the fusion feature map of the i-th layer It is expressed as:

[0034]

[0035] In the formula, , , Represents the learnable fusion weight parameters, corresponding to the current scale feature, the upsampling of the previous scale feature, and the downsampling of the next scale feature. represents the feature map of the i-th layer, represents the feature map of the i-1th layer, represents the feature map of the i+1th layer, represents the convolution operation, represents upsampling, Indicates downsampling.

[0036] Furthermore, in step S27, the region proposal network fuses feature maps at each layer. Generate anchor boxes in a sliding window manner and predict the target existence score and boundary regression parameters of each anchor box;

[0037] Among them, the target existence score It is expressed as:

[0038]

[0039] In the formula, Indicates at location The eigenvector at and Represent the weight and bias of the classification branch respectively;

[0040] The boundary regression parameter is the offset of the anchor box relative to the base anchor box. , which is expressed as:

[0041]

[0042] In the formula, and are the weight and bias of the regression branch respectively;

[0043] The loss function of the region proposal network for:

[0044]

[0045] In the formula, represents the true label of the anchor box, represents the true bounding box regression target, represents the balance coefficient, and Represent the effective number of samples for classification and regression respectively, represents the classification loss, represents the regression loss.

[0046] Furthermore, the multi-task prediction head includes a category target detection branch and an instance segmentation branch; wherein the target detection branch is used to classify and regress a bounding box for each ROI feature, and the instance segmentation branch is used to perform pixel-level mask prediction for the target within each ROI feature;

[0047] The method for the target detection branch to classify each ROI feature and regress the bounding box is specifically as follows:

[0048] The ROI features are passed through two shared fully connected layers to extract shared global features, and the shared global features are passed through the fully connected layer of the classification branch to output the category probability distribution. At the same time, the shared global features are passed through the fully connected layer of the regression branch to output the regression parameters of the bounding box.

[0049] The instance segmentation branch adopts a structure based on a fully convolutional network and introduces an attention mechanism and a feature fusion strategy. The method for performing pixel-level mask prediction on the target within the ROI feature is as follows:

[0050] S28, processing the ROI features through four consecutive convolutional layers to obtain convolutional features;

[0051] S29, enhancing the convolutional features through a spatial attention mechanism based on global average pooling and a channel attention mechanism based on two fully connected layers, respectively, to obtain a feature map applying channel attention weights and a feature map applying spatial attention;

[0052] S30, upsampling the feature map to which the spatial attention weight and the channel attention weight are applied, and fusing the low-level features to obtain an enhanced fused feature map;

[0053] S31. Pass the enhanced fusion feature map through a convolution layer and activate it using the ReLU activation function, and then adjust its channel number through another convolution layer to obtain the corresponding mask prediction result.

[0054] Furthermore, the loss function of the target detection branch is for:

[0055]

[0056] In the formula, represents the balance coefficient of the target detection task, Represents the number of effective samples for target detection tasks, represents the classification loss of the target detection task, represents the regression loss of the target detection task, represents the true category label, represents the predicted target category probability distribution, represents the regression parameters of the predicted bounding box, represents the true bounding box regression target;

[0057] The loss function of the instance segmentation branch is for:

[0058]

[0059] In the formula, Indicates the number of valid ROIs involved in the calculation in the instance segmentation task, represents the binary cross loss for instance segmentation task, represents the Dice loss of the instance segmentation task, represents the edge-aware loss for instance segmentation task, represents the prediction mask of the kth layer, represents the true mask of the kth layer, represents the edge information of the predicted mask, Represents the edge information of the real mask, and Represents the weight coefficient used to balance the contribution of Dice loss and edge-aware loss.

[0060] Furthermore, the loss function of the automatic defect annotation model is for:

[0061]

[0062] In the formula, represents the loss function of the region proposal network in the feature extraction and fusion module, represents the loss function of the target detection branch task, represents the loss function of the instance segmentation branch, and Represents a learnable homogeneous uncertainty parameter used to dynamically adjust the loss weights of object detection and instance segmentation tasks.

[0063] Furthermore, in step S2, the method for training the multi-task adaptive model is specifically as follows:

[0064] A1. Pre-train the backbone network in the multi-task adaptive model, and use normal distribution to randomly initialize the weight parameters of each layer in the feature extraction and fusion module and the multi-task prediction head;

[0065] A2. Use the current training set to train the multi-task adaptive model and use the validation set to evaluate its performance indicators after each training step.

[0066] A3. When the performance index decreases less than the preset value within the continuously set step threshold, the early stopping strategy is triggered, the model converges, a single training is completed, and the adaptive re-labeling cycle mechanism training is started;

[0067] A4. During the training process of the adaptive re-labeling cycle mechanism, the unlabeled metal surface image is input into the current multi-task adaptive model, and the corresponding labeling prediction result is output;

[0068] A5. Based on the confidence and uncertainty of the labeled prediction results, a preset threshold is used to screen samples, and new pseudo labels and / or manually labeled data are formed based on the screening results, and added to the training set to form an expanded training set, and then return to step A2;

[0069] A6. Repeat steps A2 to A5 until the model performance improvement is less than the preset threshold, completing the model training and obtaining the automatic defect labeling model.

[0070] The beneficial effects of the present invention are:

[0071] (1) The present invention adopts a multi-task learning method to realize automatic defect labeling, which can not only process multiple different defect detection tasks in parallel, but also improve the labeling quality and efficiency by sharing the underlying feature layer, and reduce the overfitting of the model to a single type of feature. This method realizes collaborative learning between different tasks, sharing the underlying features while retaining the specific differences of each task. It not only effectively improves the detection accuracy of the detection model for different defect types, but also significantly speeds up the labeling speed and reduces the dependence on manual labeling.

[0072] (2) The present invention is a high-precision automatic labeling method based on multi-tasks. The method will help to build a high-quality data set, can adapt to the complexity of rolled metal surface defects, achieve efficient feature learning and accurate labeling, help to build a high-quality data set, and promote the automation process of industrial defect detection, ultimately ensuring product quality and industrial safety.

[0073] (3) The labeling method provided by the present invention significantly reduces labor costs and improves labeling efficiency. It can be widely used in industrial defect detection and data automatic labeling scenarios of rolled metals, and is suitable for automatic labeling tasks of defect images in complex industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A flow chart of the method for automatic labeling of rolled metal surface defects based on a multi-task adaptive model provided by the present invention.

[0075] Figure 2 This is a framework diagram of the automatic defect annotation model provided by the present invention. DETAILED DESCRIPTION

[0076] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0077] The embodiment of the present invention provides a method for automatically marking defects on the surface of rolled metal based on a multi-task adaptive model, such as Figure 1 As shown, the following steps are included:

[0078] S1. Collect rolled metal surface images, perform data annotation after preprocessing and enhancement, and construct a rolled metal surface defect dataset;

[0079] S2. Build a multi-task adaptive model and train it using a rolled metal surface defect dataset to obtain an automatic defect annotation model.

[0080] Among them, the training of the multi-task adaptive model includes sequential single training and adaptive re-labeling cycle training;

[0081] like Figure 2 As shown, the defect automatic annotation model includes a backbone network, a feature extraction and fusion module and a multi-task prediction head; the backbone network is used to extract feature information of the input image from low-level to high-level; the feature extraction and fusion module is used to extract multi-layer feature maps with global context information and long-range dependencies based on the feature information extracted by the backbone network, and perform cross-level adaptive fusion to extract ROI features; the multi-task prediction head is used to use the extracted ROI features to simultaneously perform target detection and instance segmentation prediction to obtain annotation results;

[0082] S3. Input the metal surface image to be annotated into the automatic defect annotation model, and output the surface defect annotation result by executing the target detection task.

[0083] In step S1 of the embodiment of the present invention, in the data preparation and preprocessing stage, in order to ensure the representativeness and diversity of the data, a total of 20,000 metal surface images with a resolution of 2300×2200 were collected by surround shooting with a linear array camera in this embodiment, and all images are unlabeled data. The collected images cover the metal surface conditions of different production batches and different environments under the same process conditions, fully reflecting various defect types in practical applications. Metal surface defects are subdivided into five categories, namely U-shaped cracks, cracks, crushing, pulling and bulging. These defect categories provide the basis for the accurate identification and classification of subsequent models.

[0084] Furthermore, in the preparation of the labeled data, this embodiment adopts two forms: target detection labeling and instance segmentation labeling. In order to ensure the quality and balance of the labeled data, the labeled data is balanced between the categories to avoid the model training deviation caused by uneven categories. In order to improve the generalization ability of the model, this embodiment performs a variety of data enhancement processing on the training data in the image preprocessing stage, aiming to simulate various changes in the actual production environment and enhance the model's adaptability to different defect forms and environmental conditions.

[0085] Specifically, in this embodiment, the target detection annotation data contains 400 annotated images for each type of defect, totaling 2,000 images. The annotation format adopts the YOLO format (txt file). Each annotation file contains the bounding box coordinates of the defect (center point coordinates, width and height) and the category label; the instance segmentation annotation data contains 50 images for each type of defect, totaling 250 images. The annotation format adopts the JSON format, which includes pixel-level mask information of each defect area to accurately mark the shape and boundary of the defect.

[0086] In an optional embodiment of the present invention, the labeling work uses the professional labeling tool LabelImg. This tool can be used for both the labeling of target detection boxes and the fine labeling of instance segmentation masks, and can generate txt and JSON format files that meet the requirements. During the manual labeling process, the following requirements are strictly followed: (1) Ensure that each defect category is accurately labeled to avoid missing labels and mislabeling; (2) The bounding box and instance mask are as close to the actual defect shape as possible to ensure labeling accuracy; (3) Different defect categories are clearly distinguished to avoid category confusion.

[0087] During the annotation process, object detection annotation and instance segmentation annotation are based on the same image set, but the specific annotated images do not need to overlap completely, in order to reduce the annotation workload and increase data diversity. This strategy effectively reduces the cost of manual annotation and enhances the model's adaptability when processing different types of annotated data.

[0088] After image enhancement, the corresponding annotation content will also undergo corresponding transformations. For object detection annotation, the coordinate information of the bounding box is linearly transformed according to the image rotation, flipping, scaling, and translation operations to maintain the consistency between the annotation box and the defect location. For instance segmentation annotation, the pixel-level mask also needs to undergo corresponding geometric transformation and mask interpolation according to the image transformation operation to ensure the accurate correspondence between the mask and the defect area.

[0089] In step S2 of the embodiment of the present invention, the design of the multi-task adaptive model structure aims to realize the automatic labeling of the defect positions on the metal surface through multi-task learning. The model adopts a multi-task learning framework in the training stage, and jointly trains the target detection and instance segmentation tasks to make full use of the complementary information between different tasks and improve the overall performance of the model. However, in the inference stage, the model only performs the target detection task to realize the rapid automatic labeling of the defect positions.

[0090] In step S2 of the embodiment of the present invention, Figure 2 In the multi-task adaptive model shown, the backbone network is a pre-trained ResNeXt-50 network, which includes an initial convolutional layer, a maximum pooling layer, and four stages of residual blocks connected in sequence;

[0091] In this embodiment, the method for extracting feature information by the backbone network is specifically as follows:

[0092] S21, processing the input image through an initial convolutional layer to obtain a first feature map;

[0093] S22, processing the first feature map through a maximum pooling layer to obtain a first feature map;

[0094] S23, processing the first feature map through four stages of residual blocks in sequence to obtain corresponding second feature maps, third feature maps, fourth feature maps, and fifth feature maps;

[0095] S24, combining the first to fifth feature maps into feature maps of different semantic levels and spatial resolutions as feature information extracted from low-level to high-level.

[0096] Specifically, let the input image be , which is first processed by an initial convolutional layer with a kernel of 7×7, a stride of 2, and 64 channels to obtain the feature map , then use a maximum pooling layer with a kernel of 3×3 and a stride of 2 to output the first feature map , then the residual block of each stage will calculate a set of feature maps, which are , , and Together with C1, they form feature maps of different semantic levels and spatial resolutions, realizing the capture of feature information from low-level to high-level.

[0097] In step S2 of the embodiment of the present invention, Figure 2 The feature extraction and fusion module shown includes a residual attention feature pyramid network, a cross-level adaptive fusion network, and a region proposal network;

[0098] In this embodiment, the method for extracting ROI features by the feature extraction and fusion module is specifically as follows:

[0099] S25. In the residual attention feature pyramid network, based on the introduced residual structure attention mechanism, the high-level semantic information in the feature information is fused with the low-level spatial information to obtain a multi-layer feature map that captures global context information and long-range dependencies.

[0100] S26, using a cross-level adaptive fusion network to fuse features of different scales on multiple layers of feature maps to obtain fused feature maps of each layer;

[0101] S27. Use the region proposal network to generate candidate regions from the fused feature map, and suppress the generated candidate regions by non-maximum value to obtain ROI features of fixed size in each fused feature map.

[0102] In step S25 of this embodiment, a residual attention feature pyramid network (RA-FPN) is introduced in the feature extraction and fusion module to enhance the multi-scale feature representation. The traditional FPN fuses high-level semantic information with low-level spatial information through top-down paths and lateral connections. In this embodiment, in order to further improve the expressiveness of features, a residual structured attention mechanism is introduced on each layer of feature mapping of RA-FPN to capture global context information and long-range dependencies.

[0103] Specifically, step S25 of this embodiment includes the following sub-steps:

[0104] S25-1, unifying the number of channels of each layer of feature maps in the feature information to obtain feature maps with unified dimensions;

[0105] Among them, for each layer of feature mapping First, the number of channels of the feature map is adjusted to a uniform dimension through a 1×1 convolution. ,get ;

[0106] S25-2, perform context attention calculation on each layer of feature maps with unified dimensions, and multiply them by the value matrix to obtain a feature map after attention mapping;

[0107] Among them, Perform contextual attention calculation and calculate query ( ),key( ) and value ( )matrix;

[0108] , ,

[0109] In the formula, , , is a learnable weight matrix;

[0110] The residual structure attention will be with the value ( ) Matrix dot multiplication to obtain the feature map after attention mapping;

[0111] S25-3, adjust the feature map after each attention mapping to the shape of the input feature map through the reshape function, and perform residual connection with the original feature to obtain a multi-layer feature map that captures global context information and long-range dependencies;

[0112] Among them, the feature map of the i-th layer It is expressed as:

[0113]

[0114] In the formula, represents a feature map of uniform dimension, represents the query matrix for contextual attention computation, represents the key matrix for contextual attention computation, A matrix of values ​​representing the contextual attention computation, represents the scaling factor, Represents the reshape function, and the superscript T represents the transpose operator.

[0115] Through the above attention mechanism, each layer of feature maps of RA-FPN can better capture global context information and long-range dependencies, significantly improving the expressiveness and discriminative ability of features.

[0116] In step S26 of this embodiment, after completing the self-attention enhancement, the cross-level adaptive fusion (CAF) technology is used to fuse features of different scales. Specifically, for each layer of feature mapping , dimensionality reduction and dimensionality increase are performed through 1×1 convolution, and weighted fusion is performed with the upsampling result of the previous level scale feature and the downsampling result of the next level scale feature to form a fused feature map , which is expressed as:

[0117]

[0118] In the formula, , , Represents the learnable fusion weight parameters, corresponding to the current scale feature, the upsampling of the previous scale feature, and the downsampling of the next scale feature. represents the feature map of the i-th layer, represents the feature map of the i-1th layer, represents the feature map of the i+1th layer, represents the convolution operation, represents upsampling, Indicates downsampling. At the same time, in order to ensure the effectiveness of the fusion weight, the weight parameters need to meet the normalization conditions.

[0119] Through the above cross-level adaptive fusion technology, the model can dynamically adjust the contribution of features at different scales, further enhancing the diversity and discrimination ability of feature mapping obtained by the feature pyramid network. This adaptive fusion method not only retains the rich information of features at each scale, but also enhances the expressiveness of features through residual connections and attention mechanisms, improving the performance of the model in multi-scale object detection and segmentation tasks.

[0120] In step S27 of this embodiment, after the feature fusion is completed, the model uses the Region Proposal Network (RPN) to extract the region proposal from the fused feature map. Specifically, the region proposal network fuses feature maps at each layer. Generate anchor boxes in a sliding window manner and predict the target existence score of each anchor box and the boundary regression parameters ; Furthermore, RPN is designed with a shared convolutional layer and two parallel output layers: one 1×1 convolutional layer for classifying foreground and background, and the other 1×1 convolutional layer for bounding box regression.

[0121] In this embodiment, the classification branch of RPN calculates the target existence score in the following way It is expressed as:

[0122]

[0123] In the formula, Indicates at location The eigenvector at and Represent the weight and bias of the classification branch respectively;

[0124] In this embodiment, the boundary regression parameter predicted by the bounding box regression branch is the offset of the anchor box relative to the base anchor box. , which is expressed as:

[0125]

[0126] In the formula, and are the weight and bias of the regression branch respectively;

[0127] The loss function of the region proposal network for:

[0128]

[0129] In the formula, represents the true label of the anchor box, represents the true bounding box regression target, represents the balance coefficient, and Represent the effective sample numbers for classification and regression respectively, represents the classification loss, Represents the regression loss. Among them, the classification loss of RPN It is defined as a binary cross entropy function to measure the existence score of the prediction With the true label The difference between; RPN regression loss It consists of a smooth L1 loss function (Smooth function) and is used to measure the predicted bounding box offset. The actual offset The difference between.

[0130] In step S27 of this embodiment, the generated candidate regions are screened by non-maximum suppression (NMS) to obtain high-quality candidate regions (RoIs). Subsequently, the ROI Align layer is used to extract a fixed-size feature map from the fused feature map. ,in is the number of channels, and With predefined height and width.

[0131] In an embodiment of the present invention, Figure 2 The multi-task prediction head shown includes a category target detection branch and an instance segmentation branch; wherein the target detection branch is used to classify and regress bounding boxes for each ROI feature, and the instance segmentation branch is used to perform pixel-level mask prediction for the target within each ROI feature;

[0132] In this embodiment, the method for the target detection branch to classify each ROI feature and regress the bounding box is specifically as follows:

[0133] The ROI features are passed through two shared fully connected layers to extract shared global features, and the shared global features are passed through the fully connected layer of the classification branch to output the category probability distribution. At the same time, the shared global features are passed through the fully connected layer of the regression branch to output the regression parameters of the bounding box.

[0134] Specifically, ROI features The intermediate features are obtained by sharing the fully connected layer through the first parameter , and then the shared global features are obtained by sharing the fully connected layer through the second parameter :

[0135]

[0136]

[0137] In the formula, , are the weight matrices of the two fully connected layers, , is the corresponding bias vector, Represents flattening the feature map into a one-dimensional vector.

[0138] The classification branch receives the shared global features , output the category probability distribution through a fully connected layer:

[0139]

[0140] In the formula, and are the weights and biases of the classification branch.

[0141] The regression branch also receives shared features , through a fully connected layer to output the regression parameters of the bounding box:

[0142]

[0143] in, and are the weights and biases of the regression branch.

[0144] The loss function of the target detection branch includes the classification loss and regression loss .

[0145] Among them, the classification loss of the target detection branch Defined by:

[0146]

[0147] in, is the number of categories, is a true category indicator variable, The class probabilities predicted by the model.

[0148] Regression loss for object detection branch It is defined as:

[0149]

[0150] In the formula, Based on The decoded predicted bounding box, is the true bounding box, and GIoU is the generalized IoU (Intersection over Union), which is defined as:

[0151]

[0152] in, , is the minimum closure surrounding the predicted box and the true box, Indicates area.

[0153] Therefore, the loss function of the detection head used to achieve the object detection task is The classification loss is defined as and regression loss The weighted sum of is expressed as:

[0154]

[0155] In the formula, represents the balance coefficient of the target detection task, Represents the number of effective samples for target detection tasks, represents the classification loss of the target detection task, represents the regression loss of the target detection task, represents the true category label, represents the predicted target category probability distribution, represents the regression parameters of the predicted bounding box, represents the true bounding box regression target;

[0156] The instance segmentation branch in this embodiment adopts a structure based on a fully convolutional network, and introduces an attention mechanism and a feature fusion strategy. The method for performing pixel-level mask prediction on the target within the ROI feature is specifically as follows:

[0157] S28, processing the ROI features through four consecutive convolutional layers to obtain convolutional features;

[0158] Among them, ROI features Through four consecutive convolutional layers, the convolution kernel size of each layer is 3×3, the number of channels is 256, and the activation function is ReLU:

[0159] ,

[0160] In the formula, , ;

[0161] S29, enhancing the convolutional features through a spatial attention mechanism based on global average pooling and a channel attention mechanism based on two fully connected layers, respectively, to obtain a feature map applying channel attention weights and a feature map applying spatial attention;

[0162] Among them, for the convolution feature , firstly obtain the channel description vector through global average pooling:

[0163]

[0164] The channel attention weights used by the segmentation head are generated through two fully connected layers:

[0165]

[0166] In the formula, is the Sigmoid function, , and , are weights and biases; apply channel attention weights to feature maps:

[0167]

[0168] In the formula, Indicates that the matrix is ​​multiplied element by element by channel;

[0169] Features for channel attention enhancement , calculate its spatial attention map:

[0170]

[0171] The spatial attention weights are then applied to the feature map:

[0172]

[0173] In the formula, It means that the matrix is ​​multiplied element by element according to the spatial position;

[0174] S30, upsampling the feature map to which the spatial attention weight and the channel attention weight are applied, and fusing the low-level features to obtain an enhanced fused feature map;

[0175] Among them, the enhanced features Upsampling is performed while fusing low-level features to retain detail information, resulting in:

[0176]

[0177] S31, passing the enhanced fusion feature map through a convolution layer and activating it using the ReLU activation function, and then adjusting the number of channels through another convolution layer to obtain the corresponding mask prediction result;

[0178] Among them, a 3×3 convolution layer is used to activate the ReLU function, and finally a 1×1 convolution layer is passed to adjust the number of channels to the number of categories. , The mask prediction result is:

[0179]

[0180]

[0181] In this embodiment, in order to improve the details and edge quality of the segmentation result, the loss function of the instance segmentation branch introduces a Dice loss function and an edge-aware loss function based on the traditional binary cross entropy loss.

[0182] In this embodiment, the introduced edge-aware loss strengthens the model's attention to the edge by calculating the difference between the predicted mask edge and the actual mask edge, and is defined as:

[0183]

[0184] In the formula, represents the total number of edge pixels, Traverse all pixel positions, and Respectively represent the prediction mask extracted using the Sobel operator and the true mask Edge information:

[0185]

[0186]

[0187] Therefore, the loss function used to implement the instance segmentation branch is Defined as binary cross loss , Dice loss and edge-aware loss The weighted sum of is expressed as:

[0188]

[0189] In the formula, Indicates the number of valid ROIs involved in the calculation in the instance segmentation task, represents the binary cross loss for instance segmentation task (i.e. Figure 2 Binary BCE loss in represents the Dice loss of the instance segmentation task, represents the edge-aware loss for instance segmentation task, represents the prediction mask of the kth layer, represents the true mask of the kth layer, represents the edge information of the predicted mask, Represents the edge information of the real mask, and Represents the weight coefficient used to balance the contribution of Dice loss and edge-aware loss.

[0190] The multi-task adaptive model in the embodiment of the present invention needs to simultaneously realize the learning of the target detection task and the instance segmentation task, and by sharing the feature extraction channel, fully utilize the correlation between different tasks, improve the learning efficiency and prediction performance of the model, and thus optimize the performance of the automatic labeling task. Therefore, the model needs to rely on the loss functions of multiple tasks at the same time during the training stage.

[0191] In this embodiment of the present invention, in order to effectively balance the impact of the loss of each task on the overall model optimization, the loss function of the defect automatic annotation model is It is expressed as:

[0192]

[0193] In the formula, represents the loss function of the region proposal network in the feature extraction and fusion module, represents the loss function of the target detection branch, represents the loss function of the instance segmentation branch, and Represents a learnable homogeneous uncertainty parameter, which is used to dynamically adjust the loss weights of target detection and instance segmentation tasks. This design is based on the uncertainty weighted multi-task learning method. Parameters enable the model to automatically adjust the respective loss weights of the two tasks according to the inherent uncertainty of the two tasks, thereby achieving a dynamic balance between tasks.

[0194] Specifically, when the loss of a task is large and the model has high uncertainty about this task, the corresponding The parameter will increase, thereby reducing the weight of the task in the total loss and preventing the high loss value of the task from having an excessive impact on the overall model training during the gradient distribution process. Conversely, when the loss of the task is small and the uncertainty is low, The parameter will decrease, increasing the weight of the task in the total loss, prompting the model to further optimize the performance of the task. In addition, the logarithmic term in the loss function and As a regularization term, it effectively avoids The parameters are infinitely increased, thus preventing the model from minimizing the loss by simply increasing the uncertainty.

[0195] This mechanism ensures that during the training process, the object detection and instance segmentation tasks can be adaptively adjusted in weight according to their respective learning progress and difficulty, avoiding the degradation of the performance of a task due to the imbalance of weights between tasks. Through this dynamic weight adjustment, the model can more effectively utilize the information in the shared feature extraction channel, improve the overall learning efficiency, and show better prediction performance in the automatic labeling stage.

[0196] In step S2 of the embodiment of the present invention, the method for training the multi-task adaptive model is specifically as follows:

[0197] A1. Pre-train the backbone network in the multi-task adaptive model, and use normal distribution to randomly initialize the weight parameters of each layer in the feature extraction and fusion module and the multi-task prediction head;

[0198] A2. Use the current training set to train the multi-task adaptive model and use the validation set to evaluate its performance indicators after each training step.

[0199] Specifically, during training, the loss function is set to ,pass Parameters dynamically adjust the allocation of training weights; after each epoch, the validation set is used to evaluate the performance indicators of the model, including mean average precision (mAP) and total loss value;

[0200] A3. When the performance index decreases less than the preset value within the continuously set step threshold, the early stopping strategy is triggered, the model converges, a single training is completed, and the adaptive re-labeling cycle mechanism training is started;

[0201] Specifically, when the loss on the validation set decreases by less than 0.01 within 5 consecutive epochs, the early stopping strategy is triggered, and it is considered that the model has reached the initial convergence state and enters the adaptive relabeling cycle mechanism training phase;

[0202] A4. During the training process of the adaptive re-labeling cycle mechanism, the unlabeled metal surface image is input into the current multi-task adaptive model, and the corresponding labeling prediction result is output;

[0203] A5. Based on the confidence and uncertainty of the labeled prediction results, a preset threshold is used to screen samples, and new pseudo labels and / or manually labeled data are formed based on the screening results, and added to the training set to form an expanded training set, and then return to step A2;

[0204] A6. Repeat steps A2 to A5 until the model performance improvement is less than the preset threshold, completing the model training and obtaining the automatic defect labeling model.

[0205] In this embodiment, during the training process of the adaptive re-labeling cycle mechanism, according to the confidence of the prediction result and uncertainty , use the preset threshold to screen samples. ≥ 0.9 (lower limit of the maximum confidence interval) and uncertainty For prediction results with a value of ≤ 0.1 (the upper limit of the minimum uncertainty parameter interval), directly use it as a pseudo-label and add it to the training set according to the target detection annotation file (txt) in YOLO format. For prediction results with a value of ≤ 0.7 (the upper limit of the minimum confidence interval) < 0.9 or 0.1 < Samples with a value less than or equal to 0.3 (the lower limit of the maximum uncertainty parameter interval) will be manually reviewed and labeled using target detection to ensure accuracy and consistency. < 0.7 or Samples with a probability > 0.3 are considered difficult examples and are finely labeled using instance segmentation to enhance the model's ability to learn complex defects.

[0206] After adding the newly acquired pseudo-labels and manually annotated data to the training set, an expanded training data set is formed. To prevent data distribution deviation, undersampling and oversampling techniques are used to ensure that the number of samples in each defect category does not differ by more than 10%. Specifically, for categories with more than the target value, undersampling is performed by randomly deleting some samples; for categories with insufficient sample numbers, more samples are generated for oversampling through the data enhancement techniques mentioned above.

[0207] The model is retrained using the expanded training data set according to the parameters and strategies set in the initial training phase. The learning rate adopts an exponential decay strategy to avoid overfitting of the model. The Adam optimizer is continued to be used during the training process, and the momentum parameter and weight decay coefficient remain unchanged. After each iteration, the performance indicators of the model on the validation set and the newly added data are evaluated, including mean average precision (mAP), recall rate (Recall), and F1 score. When the performance improvement of the model is less than 0.5%, the iteration is stopped and the model is considered to have reached the optimal state.

[0208] In each iteration, to avoid the accumulation of pseudo-label errors, only the high-confidence pseudo-labels generated by the model in the latest iteration are retained, and the pseudo-label data in the previous iteration is removed. This strategy ensures the quality of training data and prevents erroneous information from negatively affecting model training.

[0209] In step S3 of this embodiment, during the model's processing of the image to be annotated, only the target detection task is performed to achieve rapid and automatic annotation of metal surface defects and rapid and automatic annotation of input metal surface image defects. The input metal surface image is first preprocessed and then input into the model. Multi-scale features are extracted through the ResNeXt-50 backbone network, and then the features are fused and enhanced through the residual attention feature pyramid network and the cross-level adaptive fusion module to obtain rich semantic and spatial information.

[0210] RPN generates candidate regions on the fused feature map. For each generated candidate region, the ROIAlign layer is used to extract a feature map of fixed size 7×7 as the input of the object detection branch. The object detection branch outputs the category probability distribution and the precise bounding box position of each candidate region through two fully connected layers. The output dimension of the classification branch is (N, C), where N is the number of candidate regions and C is the number of categories plus 1 (including background classes); the output dimension of the regression branch is (N, 4), corresponding to the regression parameters (x, y, w, h) of the bounding box.

[0211] In the target detection prediction process, the Softmax function is used to calculate the probability distribution of each category, and the category with the highest probability is selected as the final prediction result. The bounding box regression branch outputs accurate bounding box coordinates. In order to improve the accuracy and simplicity of the detection results, the non-maximum suppression (NMS) algorithm is used, and the NMS threshold is set to 0.5. When the intersection and union ratio of two detection boxes is greater than the threshold, only the detection box with higher confidence is retained. The final output detection results include the category label, bounding box coordinates and confidence score of each defect, and are saved in a YOLO format txt file for subsequent automatic labeling and quality inspection.

[0212] Finally, the trained model is deployed to the real-time inspection system of the production line. The metal surface images collected in real time by a high-resolution camera are input into the model for defect detection and automatic labeling. The inspection results include the category label, bounding box coordinates, and confidence score of each defect, which are superimposed on the original image in a visual form for easy viewing and analysis by staff. At the same time, the inspection results are stored in the database as data support for production quality control and optimization. The inspection results of the model are manually checked regularly. When false detections or missed detections are found, they are manually labeled and added to the training set. The model parameters are continuously optimized and updated to ensure that the model maintains high performance and accuracy under different production conditions.

[0213] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0214] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. The method for automatic marking of rolled metal surface defects based on a multi-task adaptive model is characterized by: The following steps are involved: S1. Collect rolled metal surface images, perform data annotation after preprocessing and enhancement, and construct a rolled metal surface defect dataset; S2. Build a multi-task adaptive model and train it using a rolled metal surface defect dataset to obtain an automatic defect annotation model. The training includes sequential single training and adaptive re-labeling cycle mechanism training; The automatic defect annotation model includes a backbone network, a feature extraction and fusion module and a multi-task prediction head; the backbone network is used to extract feature information of the input image from low-level to high-level; The feature extraction and fusion module is used to extract multi-layer feature maps with global context information and long-range dependencies based on the feature information extracted by the backbone network, and perform cross-level adaptive fusion to extract ROI features; The multi-task prediction head is used to simultaneously perform target detection and instance segmentation prediction using the extracted ROI features to obtain a labeling result; S3, inputting the metal surface image to be annotated into the defect automatic annotation model, and outputting the surface defect annotation result by executing the target detection task; In step S2, the backbone network is a pre-trained ResNeXt-50 network, including an initial convolutional layer, a maximum pooling layer, and four stages of residual blocks connected in sequence; The method for extracting feature information from the backbone network is specifically as follows: S21, processing the input image through an initial convolutional layer to obtain a first feature map; S22, processing the first feature map through a maximum pooling layer to obtain a first feature map; S23, processing the first feature map through four stages of residual blocks in sequence to obtain corresponding second feature maps, third feature maps, fourth feature maps, and fifth feature maps; S24, combining the first to fifth feature maps into feature maps of different semantic levels and spatial resolutions as feature information extracted from low-level to high-level; In step S2, the feature extraction and fusion module includes a residual attention feature pyramid network, a cross-level adaptive fusion network and a region proposal network; The method for extracting ROI features by the feature extraction and fusion module is specifically as follows: S25. In the residual attention feature pyramid network, based on the introduced residual structure attention mechanism, the high-level semantic information in the feature information is fused with the low-level spatial information to obtain a multi-layer feature map that captures global context information and long-range dependencies. S26, using a cross-level adaptive fusion network to fuse features of different scales on multiple layers of feature maps to obtain fused feature maps of each layer; S27, using the region proposal network to generate candidate regions from the fused feature map, and performing non-maximum suppression on the generated candidate regions to obtain ROI features of fixed size in each fused feature map; The multi-task prediction head includes a category target detection branch and an instance segmentation branch; wherein the target detection branch is used to classify and regress a bounding box for each ROI feature, and the instance segmentation branch is used to perform pixel-level mask prediction for the target within each ROI feature; The method for the target detection branch to classify each ROI feature and regress the bounding box is specifically as follows: The ROI features are passed through two shared fully connected layers to extract shared global features, and the shared global features are passed through the fully connected layer of the classification branch to output the category probability distribution. At the same time, the shared global features are passed through the fully connected layer of the regression branch to output the regression parameters of the bounding box. The instance segmentation branch adopts a structure based on a fully convolutional network and introduces an attention mechanism and a feature fusion strategy. The method for performing pixel-level mask prediction on the target within the ROI feature is as follows: S28, processing the ROI features through four consecutive convolutional layers to obtain convolutional features; S29, enhancing the convolutional features through a spatial attention mechanism based on global average pooling and a channel attention mechanism based on two fully connected layers, respectively, to obtain a feature map applying channel attention weights and a feature map applying spatial attention; S30, upsampling the feature map to which the spatial attention weight and the channel attention weight are applied, and fusing the low-level features to obtain an enhanced fused feature map; S31, passing the enhanced fusion feature map through a convolution layer and activating it using the ReLU activation function, and then adjusting the number of channels through another convolution layer to obtain the corresponding mask prediction result; In step S2, the method for training the multi-task adaptive model is specifically as follows: A1. Pre-train the backbone network in the multi-task adaptive model, and use normal distribution to randomly initialize the weight parameters of each layer in the feature extraction and fusion module and the multi-task prediction head; A2. Use the current training set to train the multi-task adaptive model and use the validation set to evaluate its performance indicators after each training step. A3. When the performance index decreases less than the preset value within the continuously set step threshold, the early stopping strategy is triggered, the model converges, a single training is completed, and the adaptive re-labeling cycle mechanism training is started; A4. During the training process of the adaptive re-labeling cycle mechanism, the unlabeled metal surface image is input into the current multi-task adaptive model, and the corresponding labeling prediction result is output; A5. Based on the confidence and uncertainty of the labeled prediction results, a preset threshold is used to screen samples, and new pseudo labels and / or manually labeled data are formed based on the screening results, and added to the training set to form an expanded training set, and then return to step A2; A6. Repeat steps A2 to A5 until the model performance improvement is less than the preset threshold, completing the model training and obtaining the automatic defect labeling model.

2. The method for automatic marking of rolled metal surface defects based on a multi-task adaptive model according to claim 1 is characterized in that: The step S25 comprises the following sub-steps: S25-1, unifying the number of channels of each layer of feature maps in the feature information to obtain feature maps with unified dimensions; S25-2, perform context attention calculation on each layer of feature maps with unified dimensions, and dot-multiply them with the value matrix to obtain a feature map after attention mapping; S25-3, adjust the feature map after each attention mapping to the shape of the input feature map through the reshape function, and perform residual connection with the original feature to obtain a multi-layer feature map that captures global context information and long-range dependencies; Among them, the feature map of the i-th layer It is expressed as: ； In the formula, represents a feature map of uniform dimension, represents the query matrix for contextual attention computation, represents the key matrix for contextual attention computation, A matrix of values ​​representing the contextual attention computation, represents the scaling factor, Represents the reshape function, and the superscript T represents the transpose operator.

3. The method for automatic marking of rolled metal surface defects based on a multi-task adaptive model according to claim 1 is characterized in that: In step S26, the fusion feature map of the i-th layer It is expressed as: ； In the formula, , , Represents the learnable fusion weight parameters, corresponding to the current scale feature, the upsampling of the previous scale feature, and the downsampling of the next scale feature. represents the feature map of the i-th layer, represents the feature map of the i-1th layer, represents the feature map of the i+1th layer, represents the convolution operation, represents upsampling, Indicates downsampling.

4. The method for automatic marking of rolled metal surface defects based on a multi-task adaptive model according to claim 1 is characterized in that: In step S27, the region proposal network fuses feature maps at each layer. Generate anchor boxes in a sliding window manner and predict the target existence score and boundary regression parameters of each anchor box; Among them, the target existence score It is expressed as: ； In the formula, Indicates at location The eigenvector at and Represent the weight and bias of the classification branch respectively; The boundary regression parameter is the offset of the anchor box relative to the base anchor box. , which is expressed as: ； In the formula, and are the weight and bias of the regression branch respectively; The loss function of the region proposal network for: ； In the formula, represents the true label of the anchor box, represents the true bounding box regression target, represents the balance coefficient, and Represent the effective number of samples for classification and regression respectively, represents the classification loss, represents the regression loss.

5. The method for automatic marking of rolled metal surface defects based on a multi-task adaptive model according to claim 1 is characterized in that: The loss function of the target detection branch is for: ； In the formula, represents the balance coefficient of the target detection task, Represents the number of effective samples for target detection tasks, represents the classification loss of the target detection task, represents the regression loss of the target detection task, represents the true category label, represents the predicted target category probability distribution, represents the regression parameters of the predicted bounding box, represents the true bounding box regression target; The loss function of the instance segmentation branch is for: ； In the formula, Indicates the number of valid ROIs involved in the calculation in the instance segmentation task, represents the binary cross loss for instance segmentation task, represents the Dice loss of the instance segmentation task, represents the edge-aware loss for instance segmentation task, represents the prediction mask of the kth layer, represents the true mask of the kth layer, represents the edge information of the predicted mask, Represents the edge information of the real mask, and Represents the weight coefficient used to balance the contribution of Dice loss and edge-aware loss.

6. The method for automatic marking of rolled metal surface defects based on a multi-task adaptive model according to claim 1 is characterized in that: The loss function of the automatic defect annotation model for: ； In the formula, represents the loss function of the region proposal network in the feature extraction and fusion module, represents the loss function of the target detection branch, represents the loss function of the instance segmentation branch, and Represents a learnable homogeneous uncertainty parameter used to dynamically adjust the loss weights of object detection and instance segmentation tasks.

Images