Cloth defect detection method based on improved Faster-RCNN
By improving the Faster-RCNN algorithm, embedding an attention residual module and an FPN network for multi-scale feature fusion, and constructing an RPN network, the problem of poor small target detection performance in fabric defect detection is solved, and high-precision fabric defect detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2023-03-21
- Publication Date
- 2026-05-19
AI Technical Summary
Existing fabric defect detection methods mainly rely on manual visual inspection, which is inefficient and prone to missed or false detections. The Faster-RCNN algorithm is not effective for detecting small targets.
The Faster-RCNN algorithm is improved by embedding an attention residual module and an FPN network into the backbone network to perform multi-scale feature fusion, constructing an RPN network to generate candidate boxes, and using fully connected layers for classification. The dataset is optimized by combining data augmentation and annotation tools to improve the detection accuracy of small objects.
It significantly improves the accuracy of fabric defect detection, especially the detection capability of small target defects, with an accuracy increase of 3.1 percentage points, reaching a detection accuracy of 90.5%, and a 30% improvement in the detection accuracy of small targets.
Smart Images
Figure CN116563211B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and in particular to a fabric defect detection method based on an improved Faster-RCNN. Background Technology
[0002] Nowadays, the quality of fabrics is becoming increasingly important, especially since defects such as holes, stains, and scratches can occur during the fabric production process. Therefore, we need to correctly identify surface defects in fabrics to prevent defective fabrics from being delivered to customers.
[0003] Currently, the primary method for detecting surface defects in fabrics is still visual inspection. This is not only inefficient but also prone to missed or false detections, wasting human and material resources and increasing production costs. With the improvement of computer computing power and the optimization of neural network algorithms in recent years, deep learning has developed rapidly. More and more factories are adopting intelligent inspection methods such as deep learning to replace manual inspection, which can significantly reduce labor costs and improve inspection efficiency. The Faster-RCNN algorithm is a common object detection algorithm, but its ability to detect small objects is insufficient. Summary of the Invention
[0004] To overcome the aforementioned shortcomings and deficiencies of existing technologies, the present invention aims to provide a fabric defect detection method based on an improved Faster-RCNN. This invention improves the Faster-RCNN algorithm for fabric defect detection, solving the problems of poor detection performance and low accuracy for small target defects in fabric inspection processes in factories.
[0005] The objective of this invention is achieved through the following technical solution:
[0006] A fabric defect detection method based on an improved Faster-RCNN includes:
[0007] Collect images of fabric defects, process the images to obtain a dataset, and divide the dataset into a training set and a validation set.
[0008] The backbone network of Faster-RCNN is constructed, which is built with attention residual modules and multi-scale feature fusion is performed using FPN network to obtain prediction feature layers of different sizes.
[0009] Construct an RPN network to generate anchors and regress them on prediction feature layers of different sizes to obtain candidate boxes;
[0010] Candidate boxes are projected onto prediction feature layers of different sizes to obtain feature maps, and the feature maps are scaled to the same size.
[0011] The obtained feature maps of the same size are used to obtain classification results through a fully connected layer.
[0012] Furthermore, the process of collecting fabric defect images, processing the defect images to obtain a dataset, and dividing the dataset into a training set and a validation set is as follows:
[0013] Take pictures to obtain images of the fabric's defects;
[0014] The acquired defect images are annotated using the labelImg annotation tool;
[0015] The dataset was expanded by performing data augmentation on defective images—random horizontal flipping.
[0016] Standardize the photos in the dataset;
[0017] The standardized data is divided into training and validation sets.
[0018] Furthermore, the backbone network is constructed using an attention residual model, specifically as follows:
[0019] The backbone network is based on the ResNet50 network structure, including convolutional layers, max pooling layers, and residual sub-modules, with the attention mechanism module SeNet embedded in each residual module.
[0020] Furthermore, the residual submodule includes a main branch and a shortcut branch. The main branch sequentially includes a first 1×1 convolution, a 3×3 convolution, and a second 1×1 convolution. A batch normalization layer and a non-linear activation function are added to each convolutional layer. The shortcut branch directly passes through the 1×1 convolution and the batch normalization layer. Three residual submodules constitute residual module layer1, four residual submodules constitute residual module layer2, six residual submodules constitute residual module layer3, and three residual submodules constitute residual module layer4.
[0021] Furthermore, the attention mechanism module Senet is embedded before the first 1×1 convolution of each residual module.
[0022] Furthermore, the attention mechanism module Senet operates as follows:
[0023] First, global average pooling is used to reduce the width and height of the input feature map to 1. Then, two 1×1 convolutions are used. The output channel number of the first 1×1 convolution is relatively small, and the second 1×1 convolution restores the original number of channels of the input feature map. After two convolutions, the Sigmoid function is used to change the channel number parameter weights to between 0 and 1. The obtained weights are multiplied with the input feature map to obtain the importance of the channels in the input feature map.
[0024] Furthermore, the multi-scale feature fusion using the FPN network to obtain prediction feature layers of different sizes is specifically as follows:
[0025] The feature maps F1, F2, F3, and F4 obtained from the residual module are convolved with 1×1 to change the depth of the feature maps to 256. Then, they are fused from top to bottom to obtain four fused feature maps P4, P3, P2, and P1. P4 is convolved with a 1×1 convolution with a stride of 2 to obtain the fifth feature map P5.
[0026] Furthermore, the candidate box also includes a binary classification result to determine whether the candidate box contains defects. If the candidate box contains foreground, it is retained. Candidate boxes with width and height less than a threshold are filtered out to obtain the position information of the remaining candidate boxes.
[0027] Furthermore, the obtained feature maps of the same size are used to obtain classification results through a fully connected layer, specifically:
[0028] The feature maps of the same size are flattened and then processed through two fully connected layers to obtain the corresponding feature vectors.
[0029] The feature vector is passed through a fully connected layer to obtain the regression parameters of the candidate box, which are used to revise the position of the candidate box;
[0030] The feature vector obtained in the previous step is passed through a fully connected layer again to obtain the target classification score.
[0031] Furthermore, the process of projecting candidate boxes onto prediction feature layers of different sizes to obtain feature maps, and then scaling the feature maps to the same size, specifically involves scaling the feature maps to the same size using ROIAlign.
[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0033] (1) The present invention embeds the attention mechanism module into the backbone network to obtain the importance between channels and enhance the feature extraction capability.
[0034] (2) The present invention sets up a multi-level information extraction network to fuse the feature maps obtained by different residual convolution modules, thereby improving the localization and detection capability of small target defects.
[0035] (3) The present invention adopts an end-to-end network architecture, trains and learns the network model as a whole, has strong transferability, and is simple and easy to implement. Attached Figure Description
[0036] Figure 1 This is a flowchart of the present invention;
[0037] Figure 2 This is a network structure diagram of the present invention;
[0038] Figure 3 This is a specific network structure diagram of the backbone network of the present invention;
[0039] Figure 4 This is a diagram of the Senet network structure for the attention mechanism module of this invention.
[0040] Figure 5 This is a structural diagram of the residual submodule of the present invention;
[0041] Figure 6 This is a structural diagram of the residual attention submodule of the present invention;
[0042] Figures 7(a)-7(f) This is a schematic diagram of the test results of the present invention on a fabric dataset. Detailed Implementation
[0043] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.
[0044] like Figures 1-6 As shown, a fabric defect detection method based on an improved Faster-RCNN includes the following steps:
[0045] Step 1: Collect images of fabric defects, label the defect images, and perform image enhancement and preprocessing on the defect images to divide the training set and validation set.
[0046] Specifically:
[0047] S1.1 Take pictures of the fabric's defects;
[0048] S1.2 The acquired defect images are annotated using the labelImg (a software for marking defect images) annotation tool. Fabric defect types include warp, weft, open edges, scratches, irregular stains, and holes.
[0049] S1.3 expands the dataset by performing data augmentation on defective images – random horizontal flipping.
[0050] S1.4 Preprocess the defect image to be processed, including standardizing the defect image and scaling the defect image to a certain size (768px width and 576px height are recommended). The corresponding standardization is shown in Equation (1):
[0051]
[0052] Among them, img newis the standardized pixel value of the defective image, mean is the mean of the images pre-trained in ResNet50 using ImageNet (a huge image library), and its value is [0.485, 0.456, 0.406]. std is the variance of the images pre-trained in ResNet50 using ImageNet, and its value is [0.229, 0.224, 0.225].
[0053] S1.5 divides the preprocessed samples into training and validation sets.
[0054] Step 2: Construct the backbone network of Faster-RCNN. The backbone network is based on the ResNet50 network structure and is built with attention residual modules. Multi-scale feature fusion is performed using the FPN network to obtain prediction feature layers of different sizes.
[0055] The backbone network includes convolutional layers, max pooling layers, and residual submodules. Each residual submodule consists of a main branch and a shortcut branch. The main branch consists of 1×1 convolutions, 3×3 convolutions, and 1×1 convolutions. A batch normalization layer (BN layer) and a non-linear activation function ReLU are added to each convolutional layer. The shortcut branch directly passes through a 1×1 convolution and a batch normalization layer (BN layer). Three residual submodules constitute residual module layer1, four residual submodules constitute residual module layer2, six residual submodules constitute residual module layer3, and three residual submodules constitute residual module layer4, for a total of four residual modules. After the network is built, the ResNet50 pre-trained weight parameters are loaded.
[0056] An attention mechanism module, Senet, is constructed and embedded before the first 1×1 convolution of the main branch of the first residual submodule in each residual module layer1, layer2, layer3, and layer4 of ResNet50, resulting in four residual convolution modules.
[0057] Specifically:
[0058] The attention mechanism module Senet first uses global average pooling to reduce the width and height of the input feature map to 1. Then, it uses two 1×1 convolutions. The first 1×1 convolution has a relatively small number of output channels, and the second 1×1 convolution restores the original number of channels in the input feature map. After the two convolutions, the Sigmoid function is used to change the weights of the channel number parameters to between 0 and 1. The obtained weights are multiplied by the input feature map to obtain the importance of the channels in the input feature map. The corresponding expression is shown in equation (2):
[0059]
[0060] Where F represents the final generated feature map, and A is the input feature map. Global average pooling is used to adjust the width and height of the feature map. δ uses two 1×1 convolutions, and σ represents the Sigmoid function, which changes the parameter weights of the feature map channels to between 0 and 1.
[0061] The fabric defect image, after passing through the residual convolutional module of the backbone network, yields a feature map F1 with a size of B×C×H×W. Here, B represents the batch size, C represents the number of channels in the feature map, H represents the height of the feature map, and W represents the width of the feature map. The backbone network has four residual convolutional layers. With each layer, the depth of the feature map doubles, while its height and width are halved, resulting in four different feature map sizes.
[0062] To construct the RPN network, firstly, four feature maps of different sizes are transformed to a depth of 256 using 1×1 convolutions. Then, the four feature maps of different sizes are fused from top to bottom to obtain four fused feature maps. During fusion, the upsampling algorithm used is nearest neighbor interpolation. The top-level feature map after fusion is then fused using a 1×1 convolution with a stride of 2 to obtain a fifth feature map. There are a total of five feature maps of different sizes, from top to bottom: P5, P4, P3, P2, and P1. Among them, feature map P5 has the smallest width and height. Feature map P4 has a width and height that are twice that of feature map P5, and so on. Feature map P1 has the largest width and height, and its width and height are 16 times that of feature map P5.
[0063] Step 3: Construct an RPN network to generate anchors and regress them on prediction feature layers of different sizes to obtain candidate boxes;
[0064] Specifically:
[0065] S3.1 Based on the five feature maps generated in S2, anchor boxes are generated using the RPN network. The recommended anchor box scales for feature maps P5, P4, P3, P2, and P1 are 192, 96, 48, 24, and 12 respectively, while the recommended aspect ratio is {1:16, 1:2, 1:1, 2:1, 16:1}. This means five anchor boxes are generated for each feature point on each feature map, resulting in 25 anchor boxes from five feature maps. Feature map P5 has the smallest aspect ratio and largest receptive field, corresponding to an anchor box scale of 192, suitable for locating relatively large defects. Feature map P1 has the largest aspect ratio and smallest receptive field, corresponding to an anchor box scale of 12, suitable for locating relatively small defects.
[0066] S3.2 Based on the 5 feature maps generated in S2, the regression parameters of the anchor boxes and the binary classification results are obtained by using 3×3 convolution and 1×1 convolution.
[0067] S3.3 Apply the anchor box regression parameters obtained in S3.2 to the generated anchor boxes to obtain the regressed candidate boxes (Proposals), and use the binary classification results obtained in step 3.2 to determine whether the candidate boxes contain defects.
[0068] S3.4 Filter out the candidate boxes obtained in S3.3, including filtering out candidate boxes with width and height smaller than a certain size (0.0001 is recommended); perform NMS (non-maximum suppression) processing, with a threshold of 0.7 for the best effect, and obtain the position information of the remaining candidate boxes after filtering.
[0069] S3.5 selects positive and negative samples only in training mode (not in testing mode). Positive samples have an IOU (Intersection over Union) greater than 0.7 between the anchor box and the manually labeled ground truth bounding box (the area of overlap between the two regions divided by the total area of the two regions), while negative samples have an IOU less than 0.3. It is recommended to select 256 samples, with a positive sample ratio of 0.5 (128 samples) and a negative sample ratio of 0.5 (128 samples). If there are fewer than 128 positive samples, negative samples are used instead. Then, the sample loss is calculated, including regression loss (only positive samples have regression loss) and classification loss. The loss function used for regression loss is shown in equation (3).
[0070]
[0071] in, The value is 1 for positive samples and 0 for negative samples, meaning there is regression loss only for positive samples. N represents the number of anchor box positions (not the number of anchor boxes), n i Representing the true regression parameters, These are the regression parameters obtained in step 3.2. In formula (3) The definition is shown in equation (4).
[0072]
[0073] The classification loss is the binary cross-entropy loss, as shown in equation (5).
[0074]
[0075] Where N is the number of positive and negative samples selected when training the RPN network. It is the actual label value, p i It is the target prediction value obtained in step 3.2. The log defaults to base e, which is equivalent to In.
[0076] Step 4: Project the candidate boxes onto prediction feature layers of different sizes to obtain feature maps, and then scale the feature maps to the same size;
[0077] Specifically:
[0078] The obtained candidate box location information is projected onto the feature map, and then the feature map is scaled to a certain size (7×7 is recommended) by ROIAlign (a module that pools regions of interest). ROIAlign has a more accurate localization and matching capability than the previous ROIPooling.
[0079] Step five uses a fully connected layer to obtain classification results from the feature maps of the same size. This further yields probability scores for the six categories of defects, specifically:
[0080] S5.1 takes the 7×7 feature map obtained in S4, first flattens it, and then passes it through two fully connected layers to obtain the corresponding feature vector.
[0081] S5.2 The feature vector obtained in S5.1 is passed through a fully connected layer again to obtain the regression parameters of the candidate box, which are used to correct the position of the candidate box again; the feature vector obtained in step 5.1 is passed through a fully connected layer again to obtain the target classification score.
[0082] In S5.3, during training mode only, candidate boxes (128 recommended) obtained from the RPN network are selected to train the ROIHead. A candidate box with an IoU greater than 0.5 with the manually labeled ground truth box is considered a positive sample, and a candidate box with an IoU less than 0.5 with the manually labeled ground truth box is considered a negative sample. Then, the loss is calculated, including regression loss (only for positive samples) and classification loss. The loss function used for regression loss is shown in Equation (6).
[0083]
[0084] Where u is the true label class, [u≧1] is Iverson brackets, indicating that the value of u is 1 when u is greater than or equal to 1, while the background class is 0, so only the regression loss of positive samples is calculated. N represents the number of positive and negative samples selected for training the ROIHead network, n i Representing the true regression parameters, These are the predicted regression parameters obtained in step 5.2.
[0085] The classification loss is the multivalued cross-entropy loss, as shown in equation (7).
[0086]
[0087] Where N is the number of positive and negative samples selected when training the ROIHead network, and p i It is the result of the target score obtained in step 5.2 after Softmax processing. The log is based on base e by default, which is equivalent to In.
[0088] like Figures 7(a)-7(f) The image shown contains partial inspection results for fabric defects, including holes, scratches, weft defects, seam openings, warp defects, and irregular stains.
[0089] Compared to the original Faster-RCNN network, this network architecture improves the accuracy of fabric defect detection, especially for small fabric defects. The accuracy (map) reaches 90.5 with IoU = 0.5, an improvement of 3.1 points compared to the original 87.4; the detection accuracy for small targets (defect pixel area less than 32×32) is improved by 30%, an improvement of 30 points compared to the original 0%.
[0090] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A fabric defect detection method based on an improved Faster-RCNN, characterized in that, include: Collect images of fabric defects, process the images to obtain a dataset, and divide the dataset into a training set and a validation set. The backbone network of Faster-RCNN is constructed based on the attention residual module. The FPN network is used to perform multi-scale feature fusion to obtain prediction feature layers of different sizes. Construct an RPN network to generate anchors and regress them on prediction feature layers of different sizes to obtain candidate boxes; Candidate boxes are projected onto prediction feature layers of different sizes to obtain feature maps, and the feature maps are scaled to the same size. The obtained feature maps of the same size are used to obtain classification results through a fully connected layer; The backbone network is built based on the attention residual module, specifically as follows: The backbone network is based on the ResNet50 network structure, including convolutional layers, max pooling layers, and residual sub-modules, with the attention mechanism module SeNet embedded into each residual module; The residual submodule includes a main branch and a shortcut branch. The main branch includes the first 1×1 convolution, the 3×3 convolution and the second 1×1 convolution in sequence. A batch normalization layer and a non-linear activation function are added to each convolutional layer. The shortcut branch directly passes through the 1×1 convolution and the batch normalization layer. Three residual submodules constitute residual module layer1, four residual submodules constitute residual module layer2, six residual submodules constitute residual module layer3, and three residual submodules constitute residual module layer4. The attention mechanism module Senet is embedded before the first 1×1 convolution of each residual module; The attention mechanism module Senet operates as follows: First, global average pooling is used to reduce the width and height of the input feature map to 1. Then, two 1×1 convolutions are used. The output channel number of the first 1×1 convolution is relatively small. The second 1×1 convolution restores the original number of channels of the input feature map. After two convolutions, the Sigmoid function is used to change the weight of the channel number parameter to between 0 and 1. The obtained weights are multiplied with the input feature map to obtain the importance of the channels in the input feature map. The method of using an FPN network for multi-scale feature fusion to obtain prediction feature layers of different sizes is as follows: The feature maps F1, F2, F3 and F4 obtained by the residual module are convolved with 1×1 to change the depth of the feature maps to 256. Then, they are fused from top to bottom to obtain four fused feature maps P4, P3, P2 and P1. P4 is convolved with a 1×1 convolution with a stride of 2 to obtain the fifth feature map P5.
2. The fabric defect detection method according to claim 1, characterized in that, The process involves collecting and processing images of fabric defects to obtain a dataset, which is then divided into a training set and a validation set. Take pictures to obtain images of the fabric's defects; The acquired defect images are annotated using the labelImg annotation tool; The dataset was expanded by performing data augmentation on defective images—random horizontal flipping. Standardize the defective images in the dataset; The standardized data is divided into training and validation sets.
3. The fabric defect detection method according to claim 1, characterized in that, The candidate box also includes using binary classification results to determine whether the candidate box contains defects. If the candidate box contains defects, it is retained. Candidate boxes with width and height less than the threshold are filtered out to obtain the position information of the remaining candidate boxes.
4. The fabric defect detection method according to claim 1, characterized in that, The obtained feature maps of the same size are then used to obtain classification results through a fully connected layer, specifically: The feature maps of the same size are flattened and then processed through two fully connected layers to obtain the corresponding feature vectors. The feature vector is passed through a fully connected layer to obtain the regression parameters of the candidate box, which are used to correct the position of the candidate box; The feature vector obtained in the previous step is passed through a fully connected layer again to obtain the target classification score.
5. The fabric defect detection method according to claim 1, characterized in that, The process involves projecting candidate boxes onto prediction feature layers of different sizes to obtain feature maps, and then scaling the feature maps to the same size, specifically by scaling the feature maps to the same size using ROIAlign.