A fabric defect detection method based on Swin-Transformer and NAS-FPN

Through the combination method of Swin-Transformer and NAS-FPN, the problem of detecting extremely small defects on fabrics is solved, and efficient and accurate online defect detection is achieved, which improves detection efficiency and reduces the risk to workers' health.

CN114066820BActive Publication Date: 2024-11-26WUHAN TEXTILE UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111247671.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-26
Publication Date
2024-11-26
Estimated Expiration
2041-10-26

AI Technical Summary

Technical Problem

The prior art is difficult to effectively and automatically detect small defects on fabrics, and the manual detection efficiency is low, the accuracy is poor, and it is harmful to the eyes of textile workers.

Method used

Using detection methods based on Swin-Transformer and NAS-FPN, high-precision positioning and classification of fabric defects is achieved through feature extraction, feature fusion and target detection network.

Benefits of technology

It realizes high accuracy and rapid detection of online fabric defect detection, improves detection efficiency and reduces eye damage to workers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114066820B_ABST
    Figure CN114066820B_ABST
Patent Text Reader

Abstract

The present invention relates to a fabric defect detection method based on Swin-Transformer and NAS-FPN, which mainly includes three stages: the first stage is to establish a data set, collect fabric images with defects, manually mark the location and type of the defects, and divide the data set into two parts: a training set and a test set; the second stage is to train the model, extract input image features through the Swin-Transformer module on the training set, then use the NAS-FPN module to perform feature fusion, and obtain a rough defect location frame through the RPN module; the third stage is to obtain the final location and type of the fabric defect by processing the features. The target detection model constructed by combining Swin-Transformer with NAS-FPN in the present invention is more accurate than the detection results of the general target detection network, especially more targeted for fabric defects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a fabric defect detection method based on Swin-Transformer and NAS-FPN. Background Art

[0002] During the production process, defects may occur on the surface of the cloth due to equipment, raw materials, processes or other external factors. Defects will affect the quality of the fabric and ultimately affect the economic benefits of the enterprise. There are many types of defects, with irregular shapes and sizes; the background texture of the fabric image is very complex, which makes it difficult to automatically detect fabric defects. In addition, the lack of large-scale defect image data sets also makes it difficult to train defect detection. For the defect detection process, most companies currently rely on manual visual inspection. This method is not only inefficient and inaccurate, but also causes irreversible damage to the eyes of textile workers due to long-term use of the eyes. Therefore, it is imperative to develop an online and automatic detection method using a computer.

[0003] Online fabric defect detection refers to the detection of defects during the fabric production process. While meeting the detection speed, accuracy must also be guaranteed to find the location and type of defects for statistics and alarms. With the development of deep learning, methods for detecting fabric defects using deep learning technology are constantly being proposed. Among them, defect detection as a target detection task is an important branch. However, it is not applicable to directly use general target detection methods to detect fabrics. This is mainly because general target detection models cannot recognize extremely small targets, and many defects on fabrics are exactly extremely small, such as stains, thread ends, holes, etc. Therefore, the present invention proposes Swin-Transformer and NAS-FPN to solve this problem. Summary of the invention

[0004] The purpose of the present invention is to provide a fabric defect detection method based on Swin-Transformer and NAS-FPN, and the obtained concrete has excellent working performance, mechanical properties, crack resistance and abrasion resistance.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A fabric defect detection method based on Swin-Transformer and NAS-FPN comprises the following steps:

[0007] Step 1: Based on the collected fabric images, defects are marked, the locations and types of defects in the fabric images are recorded, and the marked fabric images are divided into a training set and a test set;

[0008] Step 2: Input the training samples in the training set into the designed Swin-Transformer module for feature extraction to obtain feature maps at different levels;

[0009] Step 3: Use NAS-FPN to fuse the feature maps of different levels obtained in step 2 to obtain a fused feature map;

[0010] Step 4: Input the fused feature map obtained in step 3 into the RPN network to obtain a defect suggestion box;

[0011] Step 5: Combine the feature map obtained in step 3 with the defect suggestion box obtained in step 4 to perform final defect position regression and defect type classification to obtain the final defect detection result of each training sample, calculate the difference value between the detection result and the defect position and type marked on the sample, and calculate the loss function, adjust the parameters of each network module in steps 1-3 according to the loss function, and obtain the fabric defect detection model after training;

[0012] Step 6: Input the test set in step 1 into the fabric defect detection network model to obtain the target defects and position coordinates in the image, and mark them in the image.

[0013] Furthermore, the Swin-Transformer module designed in step 2 consists of a region division module and a feature extraction module. The region division module is used to divide the input image into multiple non-overlapping small blocks; the feature extraction module is used to perform multi-scale feature extraction on the divided image. The feature extraction module includes stage one, stage two, stage three and stage four:

[0014] Stage 1 is used to extract shallow features, including a linear embedding layer and 2 consecutive Swin-TransformerBlocks. The Swin-Transformer Block is a modified self-attention module. The standard multi-head self-attention module in the Transformer is replaced by a moving window-based module. The other layers remain unchanged. A LayerNorm layer is applied before each MSA module and each MLP.

[0015] The second stage is used to extract mid-level features, including a regional fusion module and two consecutive Swin-Transformer Blocks. The regional fusion module obtains hierarchical features by fusing adjacent small blocks.

[0016] Stage three consists of a regional fusion module and 6 Swin-Transformer Blocks; stage four consists of a regional fusion module and 2 Swin-Transformer Blocks; stages three and four are used to extract deep features.

[0017] Furthermore, the Swin-Transformer Block is used to divide an 8×8 feature map into 4×4 evenly using a moving window-based module. Then, the next module slides the window on the configuration of the previous module. pixels to replace the regular partition window.

[0018] Furthermore, in step 3, the feature fusion method is to use a neural architecture search method to obtain the optimal fusion method.

[0019] Furthermore, in step 4, after the fused feature map is input into the RPN network, it passes through a convolution layer and is divided into two branches. One branch represents the foreground and background probabilities of C suggestion boxes through a convolution with a Cx2 channel size of 1, and the other branch represents the coordinate values ​​of C suggestion boxes through a convolution with a Cx4 channel size of 1. The initial scale value of the suggestion box is determined according to the aspect ratio distribution of the defects in the training set images. The output data of the two branches are combined to obtain multiple defect suggestion boxes.

[0020] Furthermore, the classification method in step 5 is to input the input data into the fully connected layer, the convolution layer and the fully connected layer and then divide it into two parallel branches. One branch is connected to the fully connected layer and then regresses the position coordinates of the defect. The other branch passes through the fully connected layer and then passes through the Softmax function to obtain the type of defect. The output data of the two branches are combined to obtain the final defect position and corresponding type detected.

[0021] The beneficial effects of the present invention are as follows: 1. The present invention proposes a feature extraction step for fabric defect detection using Swin-Transformer, which has a better feature expression effect. 2. The present invention proposes a feature fusion method using NAS-FPN to search in an expandable space, and the obtained feature fusion effect is better. 3. The present invention uses the designed end-to-end fabric defect detection network model to perform online fabric defect detection with fast detection speed and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the main framework diagram of the fabric defect detection method based on Swin-Transformer and NAS-FPN of the present invention;

[0023] Figure 2 It is the main architecture of the Swin-Transformer of the present invention;

[0024] Figure 3 This is the structure diagram of this Swin-Transformer block;

[0025] Figure 4 This is the method of feature fusion performed by the common FPN of the present invention;

[0026] Figure 5 This is the workflow of merging cells, the main component of NAS-FPN of the present invention. DETAILED DESCRIPTION

[0027] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0028] like Figure 1 As shown, the embodiment of the present invention provides a fabric defect detection method based on Swin-Transformer and NAS-FPN, and the specific steps are as follows:

[0029] Step 1: Use the image acquisition system to collect fabric images in the cloth grinding workshop of a textile enterprise. 4365 images with a resolution of 4096×1696 are collected. The ratio of the number of images in the training set to the test set is 4:1.

[0030] Table 1 Number of various defects

[0031]

[0032] Step 2: Build the Swin-Transformer module, such as Figure 2 As shown in the figure, the Swin-Transformer module mainly consists of region partitioning and 4 stages. The region partitioning module divides the input image into non-overlapping small blocks. The size of each small block is designed to be 4×4 and their features are set to the connection of RGB channels. Therefore, the feature dimension of each small block is 4×4×3=48.

[0033] Stage 1 consists of a linear embedding layer and two consecutive Swin-Transformer Blocks. The Swin-Transformer Block is a modified self-attention module, such as Figure 3 As shown in Figure 5, by replacing the standard multi-head self-attention (MSA) module in Transformer with a moving window based module (SW-MSA), the other layers remain unchanged, and a LayerNorm (LN) layer is applied before each MSA module and each MLP.

[0034] Among them, the calculation formula of the attention module is:

[0035]

[0036] Among them, Q is the query matrix corresponding to the input vector, K is the key matrix, and V is the value matrix. Their initial values ​​are randomly generated. k is the dimension of the vector. The calculation formula of the Softmax activation function is as follows:

[0037]

[0038] Where S j is the softmax value of the jth component in the vector, e j is the value of the jth component.

[0039] The calculation formula of multi-head attention (MSA) is as follows:

[0040] MultiHead(Q,K,V)=Concat(head1,...,headh)WO

[0041] where head i =Attention(QW i Q ,KW i K ,VW i V )

[0042] Among them head i represents the i-th head, and the calculation formula for each head is the same as that for a single head. O Represents the corresponding weight matrix.

[0043] The second stage consists of a regional fusion module and two consecutive Swin-Transformer Blocks. The regional fusion module obtains hierarchical features through adjacent small blocks.

[0044] Stage three consists of a regional fusion module and 6 Swin-Transformer Blocks.

[0045] Stage 4 consists of a regional fusion module and two Swin-Transformer Blocks.

[0046] After stage one, we get tokens, each with C channels. Then, in order to produce a hierarchical representation, the number of tokens is reduced by merging adjacent patches. The patch merge layer concatenates the features of each group of adjacent 2×2 patches, thus obtaining token, the channel is 4C, and then after the Swin-TransformerBlock in stage 2, the output channel is set to 2C, and we get The output of the token becomes Phase 3 is similar to Phase 4 and Phase 2, and the outputs are and

[0047] The moving window based module (SW-MSA) in the Swin-Transformer Block divides an 8×8 feature map into 4×4 evenly (M=4). Then, the next module slides the window on the configuration of the previous module by using Pixels are used to replace the regular partitioned windows. The calculation of two consecutive Swin Transformer blocks is:

[0048]

[0049]

[0050]

[0051]

[0052] in and They represent the output features of the (S)W-MSA layer and the MLP layer of module l respectively. W-MSA represents window-based multi-head attention, and SW-MSA represents moving window multi-head attention.

[0053] Step 3: Construct the NAS-FPN layer, such as Figure 4 As shown, the conventional FPN layer manually specifies the feature fusion method, while NAS-FPN learns the best feature fusion method through neural architecture search. NAS-FPN consists of merging cells, such as Figure 5 As shown, building merging cells consists of four steps:

[0054] 1. Select a candidate feature layer h i

[0055] 2. Select another feature layer h j

[0056] 3. Choose the resolution of the output layer

[0057] 4. Choose a pair i Yuh j and output using the resolution selected in step 3.

[0058] Step 4: Construct the RPN layer. After obtaining the fused feature input, the RPN network undergoes a convolution of size 3 and is divided into two branches. One branch represents the foreground and background probabilities of C suggestion boxes through a convolution of size 1 in the Cx2 channel, and the other branch represents the coordinate values ​​of C suggestion boxes through a convolution of size 1 in the Cx4 channel. The initial ratio value of the preset suggestion box is manually designed based on the aspect ratio distribution of the defects in the training set images, and the value is [0.02, 0.1, 0.5, 1.0, 2.0, 10, 50]. In addition, the Cascade structure is adopted to select the suggestion box, and three IOU thresholds are designed for selecting the selection box, which are 0.5, 0.6, and 0.7 respectively.

[0059] Step 5: Construct the output layer. The output layer consists of a fully connected layer, a convolutional layer, and a fully connected layer, which is then divided into two parallel branches. One branch connects to the fully connected layer and then regresses the position coordinates of the defect. The other branch passes through the fully connected layer and then passes through the Softmax function to obtain the type of defect.

[0060] Step 6: Input the training data into the model in batches for training. The training parameters are 8 images per batch. During training, we scale the images to 1434,593, the initial learning rate is 0.001, and decay at 27 and 33. A total of 36 rounds of training.

[0061] The test results of the present invention are compared with the deep learning model Cascase RCNN. The test results are shown in Table 2. It can be seen that the stains, spots, seams, seam prints, and insect sticking, especially the seams, have been greatly improved due to the change in fusion method and the increase in candidate frames. In Table 3, the mAP of this method is improved by 0.112 compared with Cascase RCNN. It has a higher accuracy rate and a similar detection speed. The experiment shows that the model of the present invention is more suitable for fabric defect detection. The evaluation index used in the present invention is the target detection evaluation standard mAP, and the calculation formula is:

[0062]

[0063] Where n is the number of detected types, AP i is the AP of the i-th category, that is, the average precision. AP is the area of ​​the PR curve, which is generally estimated. The calculation formula is:

[0064]

[0065] Where N is the number of points of the PR curve, P(k) is the accuracy of the corresponding point, Δr(k) is the estimated recall rate of the corresponding point, and the calculation formulas are:

[0066]

[0067]

[0068] Among them, TP is the number of correct detections, FP is the number of wrong detections, and FN is the number of correct samples detected.

[0069] Table 2 Various defect detection results map

[0070]

[0071]

[0072] Table 3 Overall results comparison between the present invention and the deep learning model Cascase RCNN

[0073]

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fabric defect detection method based on Swin-Transformer and NAS-FPN, characterized in that: The following steps are involved: Step 1: Based on the collected fabric images, defects are marked, the locations and types of defects in the fabric images are recorded, and the marked fabric images are divided into a training set and a test set; Step 2: Input the training samples in the training set into the designed Swin-Transformer module for feature extraction to obtain feature maps at different levels; Step 3: Use NAS-FPN to fuse the feature maps of different levels obtained in step 2 to obtain a fused feature map. In step 3, the feature fusion method is to use the neural architecture search method to obtain the optimal fusion method. Step 4: Input the fused feature map obtained in step 3 into the RPN network to obtain a defect suggestion box. In step 4, after the fused feature map is input into the RPN network, it passes through a convolution layer and is divided into two branches. One branch represents the foreground and background probabilities of C suggestion boxes through a convolution with a Cx2 channel size of 1, and the other branch represents the coordinate values ​​of C suggestion boxes through a convolution with a Cx4 channel size of 1. The initial ratio value of the suggestion box is determined according to the aspect ratio distribution of the defects in the training set image. The output data of the two branches are combined to obtain multiple defect suggestion boxes. Step 5: Combine the feature map obtained in step 3 with the defect suggestion box obtained in step 4 to perform final defect position regression and defect type classification to obtain the final defect detection result of each training sample, calculate the difference value between the detection result and the defect position and type marked on the sample, and calculate the loss function, adjust the parameters of each network module in steps 1-3 according to the loss function, and obtain the fabric defect detection model after training; Step 6: Input the test set in step 1 into the fabric defect detection network model to obtain the target defects and position coordinates in the image, and mark them in the image.

2. The fabric defect detection method based on Swin-Transformer and NAS-FPN according to claim 1, characterized in that: The Swin-Transformer module designed in step 2 includes a region division module and a feature extraction module. The region division module is used to divide the input image into multiple non-overlapping small blocks; the feature extraction module is used to perform multi-scale feature extraction on the divided image. The feature extraction module includes stage one, stage two, stage three and stage four: Stage 1 is used to extract shallow features, including a linear embedding layer and 2 consecutive Swin-TransformerBlocks. The Swin-Transformer Block is a modified self-attention module. The modification method is to replace the standard multi-head self-attention module in the Transformer with a moving window-based module. The other layers remain unchanged. A LayerNorm layer is applied before each MSA module and each MLP. The second stage is used to extract mid-level features, including a regional fusion module and two consecutive Swin-TransformerBlocks. The regional fusion module obtains hierarchical features by fusing adjacent small blocks. Stage three consists of a regional fusion module and 6 Swin-Transformer Blocks; stage four consists of a regional fusion module and 2 Swin-Transformer Blocks; stages three and four are used to extract deep features.

3. The fabric defect detection method based on Swin-Transformer and NAS-FPN according to claim 2, characterized in that: The Swin-Transformer Block is used to evenly divide an 8×8 feature map into 4×4 sizes using a moving window-based module. Then, the next module slides the window on the configuration of the previous module by replacing the conventional partitioned window with pixels.

4. The fabric defect detection method based on Swin-Transformer and NAS-FPN according to claim 1, characterized in that: The classification method in step 5 is to input the input data into the fully connected layer, the convolution layer and the fully connected layer, and then divide it into two parallel branches. One branch is connected to the fully connected layer and then regresses the position coordinates of the defect. The other branch passes through the fully connected layer and then passes through the Softmax function to obtain the type of defect. The output data of the two branches are combined to obtain the final defect position and corresponding type detected.