A cloth defect detection method based on Sparse-rcnn with enhanced feature fusion

Abstract

The present invention provides a cloth defect detection method based on Sparse-rcnn with enhanced feature fusion. The method comprises: step 1: collecting defect images and template images under different light sources, and manually annotating defects in the defect images; wherein the defect images refer to images of cloth with defects, and the template images refer to images of cloth without defects; step 2: preprocessing all collected image data; step 3: performing difference combination three-channel processing on the preprocessed image data to obtain new three-channels, and inputting the new three-channels into a pre-built cloth defect detection network model based on Sparse-rcnn with enhanced feature fusion for training; step 4: inputting the cloth image to be detected into the trained cloth defect detection network model to obtain a defect detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of cloth defect detection, and in particular to a cloth defect detection method based on Sparse-rcnn with enhanced feature fusion. Background Art

[0002] The textile industry is a traditional pillar of the national economy, but it also faces the problem of fabric defects seriously affecting fabric quality. Compared with manual methods that are inefficient and have high false positive rates, deep learning-based fabric inspection systems that can meet the actual needs of industrial sites hold great promise.

[0003] The industry already has algorithms for fabric defect detection. For example, Nantong Santian Textile Co., Ltd. proposed a textile surface defect detection device based on the YOLO neural network (publication number CN113433137A). This device, based on the YOLO neural network, is a relatively lightweight neural network with fast detection speed but low accuracy. For example, Suzhou Harbin Institute of Technology Jileyou Intelligent Equipment Technology Co., Ltd. proposed a fabric defect detection method, system, and device based on Gabor filtering and CNN (publication number CN111398292A). However, this method suffers from the drawbacks of unclear feature extraction and unsatisfactory detection results. Summary of the Invention

[0004] In order to solve the problems in cloth defect detection such as multiple defect styles, complex detection, low detection accuracy, and slow detection speed caused by too many parameters introduced by mainstream detection algorithms, the present invention provides a cloth defect detection method based on Sparse-rcnn with enhanced feature fusion.

[0005] The present invention provides a cloth defect detection method based on Sparse-rcnn with enhanced feature fusion, comprising:

[0006] Step 1: Collect defect images and template images under different light sources, and manually mark the defects in the defect images; the defect image refers to the image of the cloth with defects, and the template image refers to the image of the cloth without defects;

[0007] Step 2: Preprocess all collected image data;

[0008] Step 3: Perform three-channel difference processing on the pre-processed image data to obtain new three-channels, and input the new three-channels into the pre-built Sparse-rcnn cloth defect detection network model based on enhanced feature fusion for training;

[0009] Step 4: Input the cloth image to be detected into the trained cloth defect detection network model to obtain the defect detection result;

[0010] The structure of the cloth defect detection network model includes a backbone network, a dynamic instance interaction head, and a classification and regression layer.

[0011] The backbone network adopts FPN based on DHLC architecture, specifically including: dividing k identical ResNet101 backbones into two groups, where k-1 backbones constitute the first group and the remaining backbone is the second group alone; all backbones in the first group are connected through the DHLC architecture as auxiliary backbones; the second group serves as the dominant backbone.

[0012] Furthermore, the preprocessing includes grayscale processing and Gaussian filtering processing.

[0013] Furthermore, in step 3, the pre-processed image data is subjected to difference combination three-channel processing to obtain new three-channels, specifically including:

[0014] The labeled image to be detected is used as the first channel; the template image is used as the second channel; a vector subtraction operation is performed on the labeled image to be detected and the template image, and the operation result is used as the third channel.

[0015] Furthermore, the auxiliary backbone specifically includes: a P1 layer feature extraction module, a P2 layer feature extraction module, a P3 layer feature extraction module, a P4 layer feature extraction module, and a P5 layer feature extraction module; wherein the structures of the latter four feature extraction modules are the same; P1 represents l-fold downsampling of the input feature map, where l = 1, 2, 3, 4, 5;

[0016] The feature extraction process of the P1 layer feature extraction module includes: the input feature map is processed in sequence by 64 convolution kernels of size 7, a batch normalization layer, a linear activation layer Relu, and a maximum pooling layer to obtain a feature map C1; then the SE module is used to downsample the feature map C1 to obtain a feature map C1';

[0017] The feature extraction process of the P2 layer feature extraction module includes three sub-processes, namely: the input feature map passes through 64 convolution kernels of size 1 and linear activation layer Relu in sequence; the input feature map passes through 64 convolution kernels of size 3 and linear activation layer Relu in sequence; the input feature map passes through 256 convolution kernels of size 1 and linear activation layer Relu in sequence to obtain feature map C2; then the SE module is used to downsample the feature map C2 to obtain feature map C2'; similarly, the feature maps extracted by the P3 layer feature extraction module are respectively recorded as feature map C3 and feature map C3'; the feature maps extracted by the P4 layer feature extraction module are respectively recorded as feature map C4 and feature map C4'; the feature maps extracted by the P5 layer feature extraction module are respectively recorded as feature map C5 and feature map C5'.

[0018] Furthermore, the SE module in each feature extraction module processes the feature map in the following manner: the feature map to be processed with a size of W×H×C passes through the global pooling layer, the fully connected layer, and the activation function layer in sequence to obtain a 1×1×C vector;

[0019] The feature map to be processed with a size of w×H×C is weighted multiplied with the obtained 1×1×C vector to obtain a new feature map with a size of W×H×C.

[0020] Furthermore, the processing of the leading backbone specifically includes:

[0021] Use a 1×1×256 convolution kernel to reduce the dimension of feature map C5' to 256, and then pass it through a 3×3 convolution layer to obtain feature map P5;

[0022] After upsampling the feature map C5', add it to the feature map C4' that has undergone 1×1 dimensionality reduction, and then pass it through a 3×3 convolution layer to obtain the feature map P4;

[0023] After upsampling the feature map C4', add it to the feature map C3' that has undergone 1×1 dimensionality reduction, and then pass it through a 3×3 convolution layer to obtain the feature map P3;

[0024] After upsampling the feature map C3', it is added to the feature map C2' that has undergone 1×1 dimensionality reduction processing, and then passed through a 3×3 convolution layer to obtain the feature map P2.

[0025] Furthermore, the processing of the dynamic instance interaction header specifically includes:

[0026] Perform ROIAlign processing on the input feature map to obtain ROI features;

[0027] Randomly initialize pre-selected boxes and pre-selected features;

[0028] The pre-selected features are passed through a fully connected layer and then convolved with the ROI features, and then linearly processed and output to the classification and regression layer.

[0029] Furthermore, during the training process, the loss function used is:

[0030]

[0031] Among them, λ Rid ,λ L1 and λ giou They are loss, Loss and The weight corresponding to the loss; For Ridloss, Indicates the label is y i Sample x i Identified as y i The predicted value of is L1loss, Among them, y i is the true value, f(x i ) is the predicted value; in, A and B represent the area of the real box and the predicted box respectively; A c =A+B, U=A+B-|A∩B|.

[0032] Furthermore, the back propagation formula of the ROIAlign process is:

[0033]

[0034] in, Represents ROIAlign to x i 、y rj The partial derivative of x i Represents the point on the feature map before pooling; y rj represents the jth point in the rth candidate region after pooling; d(i,i*(r,j)) represents the point x i and point x i*(r,j) The distances between Δh and Δw are expressed with respect to x i with x i*(r,j) The difference between the horizontal and vertical coordinates.

[0035] Furthermore, in the convolution operation performed on the pre-selected features and the ROI features after passing through a fully connected layer, the convolution operation is specifically: first performing two consecutive 1×1 convolution processes, and then using the ReLU activation function for processing.

[0036] Beneficial effects of the present invention:

[0037] (1) Using a feature extraction network model based on Sparse-rcnn, Sparse-rcnn proposes a learnable candidate box to replace the RPN part of the traditional RCNN network, getting rid of the hard defects of the two-stage detector training process, such as the large number of parameters, high computing power requirements, and long calculation time, reducing unnecessary parameters while taking into account both accuracy and computing speed;

[0038] (2) Design a new backbone network: Using the fusion connection architecture DHLC of CBNetV2, the single backbone network is cascaded to form an enhanced backbone based on Resnet101, which can better extract features;

[0039] (3) The original FPN module is improved by adding SE and softmax modules after each gradient, and fusing each gradient feature map to form a new feature map, which is then merged with the highest gradient feature map to form a new feature vector;

[0040] (4) A new loss function was designed, which improved the convergence effect and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a process for detecting cloth defects using Sparse-rcnn based on enhanced feature fusion provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the structure of a cloth defect detection network model based on Sparse-rcnn with enhanced feature fusion provided by an embodiment of the present invention;

[0043] Figure 3 It is a schematic diagram of a backbone network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, an embodiment of the present invention provides a cloth defect detection method based on Sparse-rcnn with enhanced feature fusion, comprising the following steps:

[0046] S101: Collect defect images and template images under different light sources, and manually mark defects in the defect images; wherein the defect image refers to an image of a cloth with defects, and the template image refers to an image of a cloth without defects;

[0047] Specifically, a cloth inspection device can be used to collect images. During manual annotation, defects are marked with rectangles on the image to be processed using software.

[0048] S102: Preprocessing all collected image data;

[0049] Specifically, the preprocessing includes: grayscale processing and Gaussian filtering processing; the purpose of preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.

[0050] S103: performing difference combination three-channel processing on the pre-processed image data to obtain new three-channels, and inputting the new three-channels into a pre-built Sparse-rcnn based on enhanced feature fusion cloth defect detection network model for training;

[0051] Specifically, the pre-processed image data is subjected to difference combination three-channel processing to obtain new three channels, which specifically includes: taking the marked image to be detected as the first channel; taking the template image as the second channel; and performing a vector subtraction operation on the marked image to be detected and the template image, and the operation result is taken as the third channel.

[0052] Among them, Figure 2 As shown, the structure of the cloth defect detection network model includes a backbone network, a dynamic instance interaction head and a classification and regression layer;

[0053] like Figure 3 As shown, the backbone network adopts FPN based on DHLC architecture (also known as duplex FPN), which specifically includes: dividing k identical ResNet101 backbones into two groups, where k-1 backbones constitute the first group and the remaining 1 backbone is used as the second group alone; all backbones in the first group are connected through the DHLC architecture as auxiliary backbones; the second group serves as the dominant backbone.

[0054] As an implementation method, the auxiliary backbone specifically includes: a P1 layer feature extraction module, a P2 layer feature extraction module, a P3 layer feature extraction module, a P4 layer feature extraction module, and a P5 layer feature extraction module; wherein the structures of the last four feature extraction modules are the same; P1 represents a l-fold downsampling of the input feature map, where l = 1, 2, 3, 4, 5;

[0055] The feature extraction process of the P1 layer feature extraction module includes: the input feature map is processed in sequence by 64 convolution kernels of size 7, a batch normalization layer, a linear activation layer Relu, and a maximum pooling layer to obtain a feature map C1; then the SE module is used to downsample the feature map C1 to obtain a feature map C1';

[0056] The feature extraction process of the P2 layer feature extraction module includes three progressive sub-processes, namely: the input feature map first passes through 64 convolution kernels of size 1 and a linear activation layer Relu; then passes through 64 convolution kernels of size 3 and a linear activation layer Relu; finally passes through 256 convolution kernels of size 1 and a linear activation layer Relu to obtain a feature map C2; then the SE module is used to downsample the feature map C2 to obtain a feature map C2';

[0057] Similarly, the feature maps extracted by the P3 layer feature extraction module are respectively recorded as feature map C3 and feature map C3'; the feature maps extracted by the P4 layer feature extraction module are respectively recorded as feature map C4 and feature map C4'; the feature maps extracted by the P5 layer feature extraction module are respectively recorded as feature map C5 and feature map C5'.

[0058] Specifically, in this embodiment of the present invention, the number of channels in all pyramid layers is 256. Feature maps C1 and C1': [1, 64, 56, 56]; feature maps C2 and C2': [1, 256, 56, 56]; feature maps C3 and C3': [1, 512, 28, 28]; feature maps C4 and C4': [1, 1024, 14, 14]; and feature maps C5 and C5': [1, 2048, 7, 7].

[0059] As an implementable method, the SE module in each feature extraction module processes the feature map specifically including: first, the feature map to be processed with a size of W×H×C passes through the global pooling layer, the fully connected layer and the activation function layer in sequence to obtain a 1×1×C vector; secondly, the feature map to be processed with a size of W×H×C is weighted multiplied with the obtained 1×1×C vector to obtain a new feature map with a size of W×H×C.

[0060] Furthermore, each feature extraction module is connected to a softmax module after the SE module, and the softmax module is used to normalize the feature vector. The specific function used in the operation is:

[0061]

[0062] Among them, y i Represents the normalized prediction result of each layer feature, z i represents the eigenvector.

[0063] The output of the backbone network of the embodiment of the present invention should be four feature maps of different sizes, but the depth is 256. Therefore, as an implementable method, the processing process of the leading backbone specifically includes:

[0064] Use a 1×1×256 convolution kernel to reduce the dimension of feature map C5' to 256, and then pass it through a 3×3 convolution layer (in this embodiment, padding = 2, stride = 2) to obtain feature map P5 (in this embodiment, the size is [1, 256, 7, 7]);

[0065] After upsampling, feature map C5' is added to feature map C4' that has undergone 1×1 dimensionality reduction, and then passed through a 3×3 convolutional layer (in this embodiment, padding = 2, stride = 2) to obtain feature map P4 (in this embodiment, the size is [1, 256, 14, 14]);

[0066] After upsampling the feature map C4', add it to the feature map C3' that has undergone 1×1 dimensionality reduction, and then pass it through a 3×3 convolution layer to obtain the feature map P3;

[0067] After upsampling the feature map C3', it is added to the feature map C2' that has undergone 1×1 dimensionality reduction processing, and then passed through a 3×3 convolution layer to obtain the feature map P2.

[0068] The four feature maps of different sizes extracted by the backbone network serve as inputs to the dynamic instance interaction head. As an implementation method, the processing process of the dynamic instance interaction head specifically includes:

[0069] The input feature map is ROIAligned to obtain ROI features; candidate boxes and candidate features are randomly initialized; the candidate features are passed through a fully connected layer and convolved with the ROI features (in this embodiment, the convolution operation is specifically: two consecutive 1×1 convolutions are performed, and then the ReLU activation function is used for processing), and then linear processing is performed and output to the classification regression layer.

[0070] Specifically, the candidate box (also called the suggested box) is a set of parameters of size N*4; where N represents the number of candidate objects (N=200 in this embodiment), and 4 represents the four boundaries of the object box (ranging from 0 to 1, representing the standardized center coordinates, height, and width). The role of the candidate box is to extract the region of interest (ROI), predict regression, and classify. The candidate feature (also called the suggested feature) is a set of N*d embeddings; where N represents the number of candidate objects, as above, set N=200; and d represents the dimension of the feature, in this embodiment d=256.

[0071] The candidate feature learning process involves converting the deep connections between candidate boxes, represented by a sparse matrix, into a dense matrix using a fully connected layer. This dense matrix uses 256 features to represent all candidate boxes. Both the candidate boxes and candidate features are randomly initialized as learnable parameters and optimized along with other parameters in the entire network.

[0072] Furthermore, in the dynamic instance interaction head, the head portion adopts an iterative structure (six generations) and a self-attention module. The output features and output boxes of the previous head are processed by the self-attention module and used as candidate features and candidate boxes for the next head, enabling instance interaction in the current stage. The candidate boxes of the input head portion undergo ROI alignment. Based on the position coordinates of the candidate boxes, the corresponding regions in the feature map are pooled into a fixed-size feature map for subsequent classification and bounding box regression.

[0073] Based on the above content, the training process of the cloth defect detection network model specifically includes:

[0074] Step A1: Input a set of images into the backbone network for feature extraction to obtain feature maps;

[0075] Step A2: Convolution is performed on the ROI features obtained after the dynamic instance interaction head processing and the corresponding candidate features. After passing through a fully connected layer, classification and regression are finally performed to output an image with defect annotations.

[0076] In this step, the loss function used is:

[0077]

[0078] Among them, λ Rid ,λ L1 and λ giou They are loss, Loss and The weight corresponding to the loss.

[0079] Further, is Rid loss, Indicates the label is y i Sample x i Identified as y i The predicted value of

[0080] Further, is L1 loss, Among them, y i is the true value, f(x i ) is the predicted value.

[0081] Further, The function expression is in, A and B represent the area of the real box and the predicted box respectively; A c =A+B, U=A+B-|A∩B|;

[0082] The back propagation formula for the ROIAlign process is:

[0083]

[0084] in, Represents ROIAlign to x i 、y rj The partial derivative of x i Represents the point on the feature map before pooling; y rj represents the jth point in the rth candidate region after pooling; d(i, i*(r, j)) represents the point x i and point x i*(r，j) The distances between Δh and Δw are expressed with respect to x i with x i * (r，j) The difference between the horizontal and vertical coordinates.

[0085] S104: Inputting the cloth image to be detected into the trained cloth defect detection network model, a defect detection result can be obtained, that is, a rectangular frame is selected for the detected defects and the defect type is marked and output.

[0086] The method of the present invention is based on the Sparse-rcnn neural network as the detection network, and simultaneously incorporates multiple feature fusion methods to fuse images extracted under various light sources, thereby enhancing the feature extraction of the input image.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A cloth defect detection method based on Sparse-rcnn with enhanced feature fusion, characterized in that: include: Step 1: Collect defect images and template images under different light sources, and manually mark the defects in the defect images; the defect image refers to the image of the cloth with defects, and the template image refers to the image of the cloth without defects; Step 2: Preprocess all collected image data; Step 3: Perform three-channel difference processing on the pre-processed image data to obtain new three-channels, and input the new three-channels into the pre-built Sparse-rcnn cloth defect detection network model based on enhanced feature fusion for training; Step 4: Input the cloth image to be detected into the trained cloth defect detection network model to obtain the defect detection result; The structure of the cloth defect detection network model includes a backbone network, a dynamic instance interaction head, and a classification and regression layer. The backbone network adopts FPN based on DHLC architecture, specifically including: dividing k identical ResNet101 backbones into two groups, where k-1 backbones constitute the first group and the remaining backbone is the second group alone; all backbones in the first group are connected through the DHLC architecture as auxiliary backbones; the second group serves as the dominant backbone.

2. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 1, wherein the preprocessing comprises: Grayscale processing and Gaussian filtering processing.

3. According to the cloth defect detection method based on Sparse-rcnn with enhanced feature fusion in claim 1, in step 3, the pre-processed image data is subjected to three-channel difference combination processing to obtain a new three-channel, specifically comprising: Use the labeled image to be detected as the first channel; Use the template image as the second channel; Perform vector subtraction between the labeled image to be detected and the template image, and use the result as the third channel.

4. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 1, characterized in that: The auxiliary backbone specifically includes: a P1 layer feature extraction module, a P2 layer feature extraction module, a P3 layer feature extraction module, a P4 layer feature extraction module, and a P5 layer feature extraction module; wherein the structures of the latter four feature extraction modules are the same; P1 represents a l-fold downsampling of the input feature map, where l = 1, 2, 3, 4, 5; The feature extraction process of the P1 layer feature extraction module includes: the input feature map is processed in sequence by 64 convolution kernels of size 7, a batch normalization layer, a linear activation layer Relu, and a maximum pooling layer to obtain a feature map C1; then the SE module is used to downsample the feature map C1 to obtain a feature map C1'; The feature extraction process of the P2 layer feature extraction module includes three sub-processes, namely: the input feature map passes through 64 convolution kernels of size 1 and linear activation layer Relu in sequence; the input feature map passes through 64 convolution kernels of size 3 and linear activation layer Relu in sequence; the input feature map passes through 256 convolution kernels of size 1 and linear activation layer Relu in sequence to obtain feature map C2; then the SE module is used to downsample the feature map C2 to obtain feature map C2'; similarly, the feature maps extracted by the P3 layer feature extraction module are respectively recorded as feature map C3 and feature map C3'; the feature maps extracted by the P4 layer feature extraction module are respectively recorded as feature map C4 and feature map C4'; the feature maps extracted by the P5 layer feature extraction module are respectively recorded as feature map C5 and feature map C5'.

5. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 4 is characterized in that: The SE module in each feature extraction module processes the feature map in the following way: the feature map of size W×H×C to be processed passes through the global pooling layer, the fully connected layer, and the activation function layer in sequence to obtain a 1×1×C vector; where W is the width, H is the height, and C is the number of channels; The feature map to be processed with a size of W×H×C is weighted multiplied by the obtained 1×1×C vector to obtain a new feature map with a size of W×H×C.

6. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 4 is characterized in that: The processing process of the leading backbone specifically includes: Use a 1×1×256 convolution kernel to reduce the dimension of feature map C5' to 256, and then pass it through a 3×3 convolution layer to obtain feature map P5; After upsampling the feature map C5', add it to the feature map C4' that has undergone 1×1 dimensionality reduction, and then pass it through a 3×3 convolution layer to obtain the feature map P4; After upsampling the feature map C4', add it to the feature map C3' that has undergone 1×1 dimensionality reduction, and then pass it through a 3×3 convolution layer to obtain the feature map P3; After upsampling the feature map C3', it is added to the feature map C2' that has undergone 1×1 dimensionality reduction processing, and then passed through a 3×3 convolution layer to obtain the feature map P2.

7. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 1, characterized in that: The processing of the dynamic instance interaction header specifically includes: Perform ROIAlign processing on the input feature map to obtain ROI features; Randomly initialize pre-selected boxes and pre-selected features; The pre-selected features are passed through a fully connected layer and then convolved with the ROI features, and then linearly processed and output to the classification and regression layer.

8. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 1, characterized in that: During the training process, the loss function used is: Among them, λ Rid ,λ L1 and λ giou They are loss, Loss and The weight corresponding to the loss; For Ridloss, Indicates the label is y i Sample x i Identified as y i The predicted value of is L1loss, Among them, y i is the true value, f(x i ) is the predicted value; in, A and B represent the area of the real box and the predicted box respectively; A c =A+B, U=A+B-|A∩B|.

9. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 7, characterized in that: The back propagation formula for the ROIAlign process is: in, Represents ROIAlign to x i 、y rj The partial derivative of x i Represents the point on the feature map before pooling; y rj represents the jth point in the rth candidate region after pooling; d(i,i*(r,j)) represents the point x i and point x i*(r,j) The distances between Δh and Δw are expressed with respect to x i with x i*(r,j) The difference between the horizontal and vertical coordinates.

10. The cloth defect detection method based on Sparse-rcnn with enhanced feature fusion according to claim 7, characterized in that: In the convolution operation performed on the pre-selected features and the ROI features after passing through a fully connected layer, the convolution operation is specifically: first performing two consecutive 1×1 convolution processes, and then using the ReLU activation function for processing.

Images