A transfer learning method for steel surface defect detection

Abstract

The present invention relates to the field of image processing technology, and in particular to a transfer learning method for steel surface defect detection, comprising obtaining steel surface image samples through an industrial camera and preprocessing the image samples, dividing the preprocessed steel image samples into a source domain and a target domain, and marking the source domain steel defect samples; using a ResNet50 network to extract steel defect features in the source domain and target domain, and mapping the source domain and target domain data into a joint subspace; performing dimensionality reduction processing on high-dimensional features through a linear discriminant analysis method to achieve source domain and target domain subspace alignment; selecting a subset of target domain samples from both the source domain and the target domain for pseudo-labeling, and completing the pseudo-labeling of all target domain steel surface defect samples through iterative learning. The present invention can accurately and quickly detect steel surface defects, improve the generalization ability of steel surface defect detection, and reduce the data volume requirement for defect detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a transfer learning method for steel surface defect detection. Background Art

[0002] Defects on the steel surface not only affect the overall appearance of the product, but also reduce the corrosion resistance and fatigue strength of the steel product. Therefore, defect detection of steel surface defects is very important for metallurgical enterprises and even the entire society.

[0003] Traditional photoelectric detection methods and manual visual inspection are very slow. In recent years, with the continuous development of artificial intelligence, automated inspection technology has also continued to advance. He et al. achieved good detection accuracy for steel surface defect detection by improving the two-stage detector Faster RCNN. Ma et al. further improved the single-stage object detector YOLOv5 for steel surface defect detection, further improving detection accuracy and speed. However, fully supervised detection methods require a large amount of labeled data to train the model. In real-world production, collecting surface defect data samples takes a long time, and the generation of surface defects is somewhat random. Manual data labeling is very laborious, and obtaining a large number of labeled samples is difficult. Therefore, pseudo-labeling methods have emerged for steel surface inspection. Existing pseudo-labeling methods based on deep transfer learning have the following main shortcomings: pseudo-labels are generated in the target domain after the model is trained using source domain samples. The quality of pseudo-labels is limited by the source domain data. Due to the difference in data distribution between the source and target domains, pseudo-label quality is too low, which in turn affects detector performance. Summary of the Invention

[0004] Aiming at the shortcomings of existing methods, the present invention addresses the problems existing in the current steel surface defect detection. Taking into account the distribution differences between source domain data and target domain data, it is proposed to pseudo-label the steel surface defects in the target domain based on the label information of the source domain samples and the intrinsic structural information of the target domain samples, so as to make the pseudo-labeling more accurate and further improve the detector performance.

[0005] The technical solution adopted by the present invention is: a transfer learning method for steel surface defect detection includes the following steps:

[0006] Step 1: Use an industrial camera to obtain steel surface image samples and preprocess the image samples. The preprocessed steel image samples are divided into a source domain and a target domain, and the steel defect samples in the source domain are marked.

[0007] Furthermore, preprocessing includes contrast enhancement through linear transformation and mosaic data enhancement.

[0008] Furthermore, source domain steel defects include: mill scale, plaque, cracks, pitting surface, impurities, and scratches.

[0009] Step 2: Use the ResNet50 network to extract steel defect features in the source and target domains, and map the source and target domain data into the joint subspace;

[0010] Furthermore, the ResNet50 network consists of four groups of large blocks, each of which contains 3, 4, 6, and 3 small blocks, and each small block contains three convolutions.

[0011] Step 3: Use linear discriminant analysis to reduce the dimensionality of high-dimensional features and align the subspaces of the source and target domains;

[0012] Furthermore, step three specifically includes:

[0013] Step 31: Calculate the surface defect Ω of the i-th type of steel after dimensionality reduction i The center point is The center point of all samples after dimensionality reduction is

[0014] Step 32: Define the discrete degree S between each category in the new sample space after dimensionality reduction b ;

[0015]

[0016] Step 33: Define the compactness S within each category in the new sample space after dimensionality reduction W ;

[0017]

[0018] Step 34: According to the dispersion S between each category in the new sample space b , define the scatter matrix P between each category b ;

[0019]

[0020] Step 35: According to the compactness S of each category in the new sample space W , define the internal scatter matrix P of each category W ;

[0021]

[0022] Step 36: S b Simplified to S W Simplified to

[0023] Step 37: Establish the objective function based on the linear discriminant criterion:

[0024] Step 38: Taking the maximum value and combining it with the Lagrange multiplier method, the corresponding Lagrange function can be obtained as follows:

[0025]

[0026] Where w k is the kth column of the projection matrix W;

[0027] Step 39: The new sample matrix after dimensionality reduction by the projection matrix is: X′=W T X, X′∈R m′×n , m′ is the sample dimension after dimensionality reduction.

[0028] Step 4: Select a subset of target domain samples from both the source domain and the target domain for pseudo-labeling, and complete the pseudo-labeling of all target domain steel surface defect samples through iterative learning;

[0029] Furthermore, step four specifically includes:

[0030] The distribution structure of source domain samples is obtained based on the label information, and the projections of all source domain samples belonging to the same defect category are averaged:

[0031]

[0032] in, represents the set of samples with y-type defects in all source domain samples in the projection subspace, represents the mapping mean of the defects belonging to the same defect category, represents the i-th defect sample belonging to category y after projection. y=0,1,…|Y| represents the number of steel surface defect categories in the data sample.

[0033] Using the projection Z of the target domain sample in the subspace t Mapping mean to source domain The distance between Represents the difference between target domain samples and source domain samples:

[0034]

[0035] By calculating the projection Z of the target domain sample in the subspace t The difference between the prototype sample and the target sample x is obtained t Conditional probability of belonging to class y:

[0036]

[0037] Furthermore, step four also includes:

[0038] A one-to-one matching is established between a cluster in the target domain and a class in the source domain, so that the sum of the distances between all matched cluster centers and class prototypes is minimized. The formula is:

[0039]

[0040] Where A ij represents the correlation between the i-th cluster and category j, represents the cluster center belonging to the i-th cluster in the target domain mapping, Represents the class prototype in the source domain. For any A ij Existence relationship:

[0041]

[0042] The difference between the sample features projected by the target domain and the prototype sample features of the target domain is expressed as: Among them, z t is the sample projected into the target domain, is the prototype sample of the target domain;

[0043] The target sample x is obtained by calculating the structural differences between the target domain sample classes. t The conditional probability of belonging to class y is:

[0044]

[0045] Furthermore, step four also includes:

[0046] Calculate p1(y|x t )、p2(y|x t ), we get:

[0047] p(y|x t )=max{p1(y|x t ),p2(y|x t )}

[0048] The pseudo label of the target domain is obtained through the label information of the source domain and the intra-class structure information of the target domain sample:

[0049]

[0050] The pseudo labels of steel surface defect samples in all target domains and the probability of pseudo labels are expressed as:

[0051]

[0052] Where n t is the number of steel surface defect samples in the target domain.

[0053] Beneficial effects of the present invention:

[0054] 1. Steel surface defect image samples are divided into source and target domains, and the source and target domain data are mapped into a joint subspace. High-dimensional features are then reduced in dimensionality using linear discriminant analysis. Linear discriminant analysis is a supervised dimensionality reduction classification method that can use prior knowledge of categories during the dimensionality reduction process. Compared with other algorithms, it has greater advantages in processing steel surface feature information.

[0055] 2. During actual production, steel surfaces can exhibit varying degrees of defects, including inclusions, plaques, cracks, pitting, mill scale, and scratches. Collecting surface defect data samples takes a long time, and surface defects are generated with a certain degree of randomness. Manual data labeling is very arduous, making it difficult to obtain a large number of labeled samples. This approach pseudo-labels target steel surface defects based on both the label information of source domain surface defect samples and the intrinsic structural information of target domain surface defect samples. This reduces reliance on labeled samples, improves detection efficiency, and enhances the accuracy of steel surface defect detection through iterative learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of the transfer learning method for steel surface defect detection of the present invention;

[0057] Figure 2 It is the ResNet50 feature extraction module of the present invention;

[0058] Figure 3 Graph 1 is a pseudo-label iterative training model of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.

[0060] like Figure 1 As shown in Figure 1, a transfer learning method for steel surface defect detection includes the following steps:

[0061] Step 1: Use an industrial camera to obtain steel surface image samples and preprocess them. The preprocessed steel image samples are divided into source and target domains, and the steel defect samples in the source domain are marked.

[0062] By fixing the camera angle and keeping the light at the shooting location consistent with the actual working environment, high-definition images of the steel surface can be captured.

[0063] The acquired image is preprocessed using linear transformation contrast enhancement and Mosaic data enhancement methods. Since the grayscale value range of the steel surface defect image is small, in order to avoid the overall dark or bright appearance of some input images, the image is first processed using linear transformation contrast enhancement. The specific steps of linear contrast enhancement include:

[0064] Assume that the grayscale value of the original image is [f(i,j)], and the grayscale value of the result image after linear contrast enhancement is [g(i,j)]. The value ranges of [f(i,j)] and [g(i,j)] are both in [0,255]. The performance of g is better than that of f, so it is necessary to suppress the unimportant parts to enhance the contrast of the concerned parts. Assume that the grayscale image of the important target area of ​​the original image is distributed in [f a , f a ], the grayscale distribution of the target area of ​​the image after linear contrast enhancement is [g a , g b Calculation formula:

[0065]

[0066] Where i = 1, 2, ... m; j = 1, 2, ... n, By linear transformation contrast enhancement, pixels that are too bright or too dark are compressed, pixels with moderate brightness are expanded, and the visual effect of the image is changed.

[0067] Mosaic data enhancement uses four different steel surface defect image samples obtained by linear transformation contrast enhancement as the upper left corner, lower left corner, upper right corner and lower right corner of the synthetic image, and then splices the four images into one image through random scaling, flipping and cropping methods to expand the number of steel surface defect sample sets. Different steel surface defect targets are trained through a synthetic image, and random scaling adds many small targets, thereby improving the robustness of the system and the detection accuracy of steel surface defects.

[0068] The preprocessed steel surface image samples are divided into two categories. The categories and locations of steel surface defects in the first category are marked as source domain data. The second category of unmarked defect samples are used as target domain samples. The steel surface defects are marked using the LabelImg tool.

[0069] Steel surface defects are divided into 6 categories: rolling scale, plaque, cracks, pitting surface, impurities, and scratches.

[0070] The rolled oxide scale is marked as rolled-in scale, the patches are marked as patches, the cracks are marked as crazing, the pitted surface is marked as pitted-surface, the impurities are marked as inclusion, and the scratches are marked as Scratches. After marking, the data is saved in the format of a json file.

[0071] Step 2: Use the ResNet50 network to extract steel defect features in the source and target domains, and map the source and target domain data into the joint subspace;

[0072] like Figure 2 For the ResNet50 feature extraction module, ResNet50 trained on ImageNet is used as the feature extractor to extract surface defect features of steel in the source and target domains, removing the last fully connected layer; the convolution step size of the convolution layer in the fourth stage is set to 1 to extract more complete defect features; ResNet50 has a relatively deep network structure and can extract more features for complex images. The unique Residual structure converts the function H(x) required to learn by the original convolutional neural network into F(x)+x. This structure can effectively solve the problem that the more layers of the convolutional neural network, the worse the extraction effect.

[0073] ResNet50 has four groups of large blocks, each with 3, 4, 6, and 3 small blocks, and each small block has three convolutions.

[0074] Convolution layer calculation: The input image size is W×H, where W is the image width and H is the image height; the size of the convolution kernel is F×F, the stride is S, the number of image channels is C, and P represents padding. The output image size is: The number of output image channels is C.

[0075] Calculation of the pooling layer: The output image size after pooling is:

[0076] Map the source domain feature samples and target domain feature samples after feature extraction into the joint subspace; the source domain has labeled steel surface defect samples Unlabeled steel surface defect samples in the target domain in, represents the i-th labeled image in the source domain, represents the label corresponding to the source domain sample, represents the i-th unlabeled image in the target domain, n s represents the number of labeled images in the source domain, n t Represents the number of unlabeled images in the target domain; the source domain data sample set after mapping is Indicates that the target domain data sample set is express.

[0077] Step 3: Use linear discriminant analysis to reduce the dimensionality of high-dimensional features to achieve subspace alignment between the source and target domains. The high-dimensional deep features extracted by the ResNet50 feature extractor contain redundant noise information. To reduce interference and unnecessary calculations, these high-dimensional features are reduced in dimensionality using linear discriminant analysis.

[0078] The linear discriminant method projects the steel surface defect sample data in the high-dimensional space into the low-dimensional space. After the projection, the same steel surface defect categories in the low-dimensional space are clustered together, while the distances between different steel surface defect categories after projection are far apart.

[0079] Specifically include:

[0080] Steel surface image sample matrix of source domain and target domain m is the sample dimension, n=n s +n t is the total number of samples; there are C types of steel surface defects in X, and the sample center point of each defect category is The center point of all missing samples on the steel surface is Among them, ω i It is the surface defect category of steel of type i.

[0081] According to the dimension reduction target requirement, the projection matrix W∈R m×m′ , where m′≤m; the sample matrix after dimensionality reduction is Z=W T X; where W T is the transposed matrix of W, and the surface defect Ω of the i-th type of steel after dimensionality reduction i The center point is The center point of all samples after dimensionality reduction is If they are homogeneous samples, the projection points of the sample features will be as close as possible; if they are heterogeneous samples, the projection points of the sample features will be as far away as possible.

[0082] Define the discreteness S between categories in the new sample space after dimensionality reduction b for:

[0083]

[0084] Define the compactness S within each category in the new sample space after dimensionality reduction W for:

[0085]

[0086] In the formula, the dispersion S between each category b Represents the distance from the center point of each category of steel surface defects to the center of all samples, is the parameter weight of each defect, and the compactness S within each category W Indicates the distance between the samples in each defect category and the center point of their category.

[0087] According to the compactness S of each category in the new sample space W , define the internal scatter matrix P of each category W for:

[0088]

[0089] According to the scatter matrix, S b Simplified to S W Simplified to

[0090] Combined with the intra-class scatter matrix P W and the inter-class scatter matrix P b , establish the objective function according to the linear discriminant criterion: To obtain the best projection direction, use Taking the maximum value and combining it with the Lagrange multiplier method, the corresponding Lagrange function can be obtained as follows:

[0091]

[0092] Where w k is the kth column of the projection matrix W, λ is the distribution weight parameter; for w k Taking the derivative and setting it to 0, we get Pick The eigenvectors corresponding to the first k largest eigenvalues ​​obtain the optimal projection matrix W.

[0093] The new sample matrix after dimensionality reduction by the projection matrix is: X′=W T X, X′∈R m′×n , m′ is the sample dimension after dimensionality reduction.

[0094] like Figure 3 The pseudo-label iterative training process shown in the figure, step 4: select a subset of target domain samples from both the source domain and the target domain for pseudo-labeling, and complete the pseudo-labeling of all target domain steel surface defect samples through iterative learning;

[0095] Source domain steel surface defect sample x s and target domain steel surface defect sample x t After the dimensionality reduction and alignment of the subspace, the projection Z of the corresponding source domain sample in the subspace is obtained s and the projection Z of the target domain sample in the subspace t; Considering both the source domain and target domain, a subset of target domain samples is selected for pseudo-labeling. The projection matrix is ​​improved by combining the pseudo-labeled target domain samples and labeled source domain samples to improve the accuracy of pseudo-labeling.

[0096] From the source domain perspective: Since source domain samples contain labels, the distribution structure of source domain samples is obtained based on the label information, and the projections of all source domain samples belonging to the same defect category are averaged:

[0097]

[0098] in, represents the set of samples with y-type defects in all source domain samples in the projection subspace, represents the mapping mean of the defects belonging to the same defect category, represents the i-th defect sample belonging to category y after projection. y=0,1,…|Y| represents the number of steel surface defect categories in the data sample.

[0099] Using the projection Z of the target domain sample in the subspace t Mapping mean to source domain The distance between Represents the difference between target domain samples and source domain samples:

[0100]

[0101] By calculating the projection Z of the target domain sample in the subspace t The difference between the prototype sample and the target sample x is obtained t Conditional probability of belonging to class y:

[0102]

[0103] From the perspective of the target domain: Since the distribution within the target domain class is somewhat different from that within the source domain class, considering the category structure of the target domain itself, the projection vector Z of all target domain samples in the subspace is obtained by cluster analysis. t Generate |Y| clusters and initialize the cluster centers by calculating class prototypes.

[0104] A one-to-one matching is established between a cluster in the target domain and a class in the source domain, minimizing the sum of the distances between all matching cluster centers and class prototypes. Let A ij is a one-to-one mapping matrix, and the optimization formula is:

[0105]

[0106] Where A ij represents the correlation between the i-th cluster and category j, represents the cluster center belonging to the i-th cluster in the target domain mapping, Represents the class prototype in the source domain. For any A ij Existence relationship:

[0107]

[0108] The differences between the target domain samples within each defect category are represented by the differences between the sample features of the target domain projection and the target domain prototype sample features: Among them, z t is the sample projected into the target domain, is the prototype sample of the target domain.

[0109] The target sample x is obtained by calculating the structural differences between the target domain sample classes. t The conditional probability of belonging to class y is:

[0110]

[0111] Target domain samples can be pseudo-labeled by using the label information of source domain samples or by using the intra-class structure information of the target domain. However, these two methods are essentially different. Pseudo-labeling using the label information of source domain samples often outputs high probabilities for samples close to the source data, while pseudo-labeling based on the intra-class structure information of the target domain outputs high probabilities for samples close to the target domain cluster center, regardless of the distance to the source domain. By combining these two pseudo-labeling methods and taking advantage of their complementarity, we can obtain the following:

[0112] p(y|x t )=max{p1(y|x t ),p2(y|x t )}

[0113] The most appropriate target domain pseudo label is obtained through the source domain label information and the target domain sample class structure information:

[0114]

[0115] The pseudo labels of steel surface defect samples in all target domains and the probability of pseudo labels are expressed as:

[0116]

[0117] Through iterative learning, projection learning and pseudo-labeling processes reinforce each other, and in the kth iteration a A subset of target domain samples Where, represents the i-th unlabeled image in the target domain, n t is the number of steel surface defect samples in the target domain, and T' is the number of iterations in the learning process.

[0118] Table 1 Pseudocode of transfer learning algorithm for steel surface defect detection

[0119]

[0120] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A transfer learning method for steel surface defect detection, characterized in that: The following steps are involved: Step 1: Use an industrial camera to obtain steel surface image samples and preprocess the image samples. The preprocessed steel image samples are divided into a source domain and a target domain, and the steel defect samples in the source domain are marked. Step 2: Use the ResNet50 network to extract steel defect features in the source and target domains, and map the source and target domain data into the joint subspace; Step 3: Use linear discriminant analysis to reduce the dimensionality of high-dimensional features and align the subspaces of the source and target domains; Step 3: Specific include: Step 31, calculate the first i Steel-like surface defects The center point is , is the sample center point of each defect category, The transpose of the projection matrix is ​​required for dimensionality reduction; the center point of all samples after dimensionality reduction is ; Step 32: Define the discreteness between categories in the new sample space after dimensionality reduction ; in, It is the center point of all missing samples on the steel surface; Step 33: Define the compactness within each category in the new sample space after dimensionality reduction S W ; ； in, is the sample matrix after dimensionality reduction; Step 34: Based on the discreteness between categories in the new sample space , define the scatter matrix between categories ; Step 35: According to the compactness of each category in the new sample space , define the internal scatter matrix of each category ; Step 36: According to the scatter matrix Simplified to , Simplified to ; Step 37: Establish the objective function based on the linear discriminant criterion: ; Step 38: Taking the maximum value and combining it with the Lagrange multiplier method, the corresponding Lagrange function is: Where, is the projection matrix The kth column of is the allocation weight parameter; Step 39: The new sample matrix after dimensionality reduction by the projection matrix is: , is the sample dimension after dimensionality reduction; Step 4: Select a subset of target domain samples from both the source domain and the target domain for pseudo-labeling, and complete the pseudo-labeling of all target domain steel surface defect samples through iterative learning; Step 4 specifically includes: The distribution structure of source domain samples is obtained based on the label information, and the projections of all source domain samples belonging to the same defect category are averaged: in, represents the mapping mean of the defects belonging to the same defect category, After projection i defect samples belonging to class y, y =0,1,…| Y | represents the number of steel surface defect categories in the data sample; Utilize the projection of target domain samples in the subspace Mapping mean to source domain The distance between Represents the difference between target domain samples and source domain samples: By calculating the projection of the target domain sample in the subspace The difference between the target sample and the prototype sample is obtained Conditional probability of belonging to class y: 。 2. The transfer learning method for steel surface defect detection according to claim 1, characterized in that: Preprocessing includes: Contrast enhancement and mosaic data enhancement are performed through linear transformation.

3. The transfer learning method for steel surface defect detection according to claim 1, characterized in that: Source domain steel defects include: mill scale, plaque, cracks, pitted surface, inclusions and scratches.

4. The transfer learning method for steel surface defect detection according to claim 1, characterized in that: The ResNet50 network consists of four groups of large blocks, each of which consists of 3, 4, 6, and 3 small blocks, and each small block has three convolutions.

5. The transfer learning method for steel surface defect detection according to claim 1, characterized in that: Step 4 also includes: A one-to-one matching is established between a cluster in the target domain and a class in the source domain, so that the sum of the distances between all matched cluster centers and class prototypes is minimized, as follows: Where, Indicates the i Clusters and categories j The correlation, Indicates that the target domain mapping belongs to i The cluster center, Represents the class prototype in the source domain. For any Existence relationship: The difference between the sample features projected by the target domain and the prototype sample features of the target domain is expressed as: ;in, is the sample projected into the target domain, is the prototype sample of the target domain; The target samples are obtained by calculating the structural differences between the target domain sample classes. belong The conditional probability of the class is: 。 6. The transfer learning method for steel surface defect detection according to claim 5, characterized in that: Step 4 also includes: calculate 、 The complementarity between them gives: The pseudo label of the target domain is obtained through the label information of the source domain and the intra-class structure information of the target domain sample: The pseudo labels of steel surface defect samples in all target domains and the probability of pseudo labels are expressed as: Where, is the number of surface defect samples of steel in the target domain, Indicates the target domain i unlabeled images.

Images