A wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network

Abstract

The present invention relates to the field of image recognition technology. The purpose is to provide a wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network, which should have the characteristics of fast recognition speed and high recognition accuracy. The technical solution is: a wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network, comprising the following steps: 1) collecting a number of wool and cashmere fiber images to produce a data set, and annotating each fiber in each fiber image in the data set; 2) inputting the data set into the improved Mask R-CNN neural network for training to obtain a trained improved Mask R-CNN neural network; 3) using the improved Mask R-CNN neural network to classify and identify wool and cashmere fibers.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network. Background Art

[0002] Wool and cashmere fibers are two important materials in the textile industry. However, cashmere offers significantly better performance than wool and is correspondingly more expensive. Both wool and cashmere are composed of protein, and they are highly similar in appearance, feel, physical properties, and chemical composition. Consequently, unscrupulous merchants exploit these characteristics by adulterating cashmere clothing with wool fibers during production to maximize profits, infringing on the legitimate rights and interests of consumers. my country's wool and cashmere fiber production is enormous, and the total annual output continues to increase. Therefore, effective and accurate identification of wool and cashmere fibers is essential.

[0003] Currently, traditional methods for identifying wool and cashmere fibers include optical microscopy (a physical method) and dyeing (a chemical method). The former uses a microscope to observe minute details of the fibers, which relies heavily on the operator's experience and requires a variety of criteria. The latter uses dyes to dye wool and cashmere, distinguishing the two by observing the dyeing. However, these methods require specialized technicians to complete the identification of wool and cashmere, and are only suitable for small quantities. They cannot meet the current requirements for efficient and rapid identification of large quantities of wool and cashmere.

[0004] To address the shortcomings of traditional wool and cashmere fiber recognition methods (physical and chemical), image segmentation can be used to first accurately separate each wool and cashmere fiber in the image. Features for each fiber, such as grayscale histogram features, LBP (Local Binary Pattern) features, Hu invariant rectangular features, and gray-level co-occurrence matrix features, are then calculated. These features are then fed into a Support Vector Machine (SVM) for classification and recognition. However, this approach presents two challenges. First, wool and cashmere fibers in an image are often intertwined, making image segmentation insufficiently robust. Second, the highly similar morphological characteristics of wool and cashmere fibers make it difficult to extract their underlying features, resulting in low SVM classification and recognition accuracy. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and provide a wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network, which should have the characteristics of fast recognition speed and high recognition accuracy.

[0006] The technical solution of the present invention is:

[0007] A wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network comprises the following steps:

[0008] 1) Collect several wool and cashmere fiber images to create a dataset, and annotate each fiber in each fiber image in the dataset;

[0009] 2) Input the data set into the improved Mask R-CNN neural network for training to obtain a trained improved Mask R-CNN neural network;

[0010] 3) Using the improved Mask R-CNN neural network to classify and identify wool and cashmere fibers;

[0011] The improved Mask R-CNN neural network includes a Transformer backbone network, a PANet intermediate layer, an RPN region proposal network, a ROI Align network, a head network, and a ReCls reclassification network; the head network includes a fully connected layer, a category recognition branch, a prediction bounding box branch, and a prediction mask branch, and the category loss function of the category recognition branch is a softmax function and a cross entropy loss function.

[0012] The annotation in step 1) includes: manually annotating each wool and cashmere fiber in each image in the dataset using Labelme image annotation software and saving the result as a JSON file. The JSON file stores the width, height, upper left corner coordinates, and number information of each wool and cashmere fiber bounding box, and converting all JSON files into the COCO dataset format.

[0013] The Transformer backbone network includes six encoder layers and six decoder layers. Each encoder layer includes a self-attention layer and a feed forward layer, and each decoder layer includes a self-attention layer, an encoder-decoder attention layer, and a feed forward layer.

[0014] The PANet intermediate layer includes feature pyramid network, bottom-up path enhancement, dynamic feature pooling, and fully connected layer fusion.

[0015] The RPN region proposal network includes: scanning the backbone feature map generated by the PANet intermediate layer through a sliding window, and proposing several rectangular candidate regions of different sizes.

[0016] The ROI Align network uses bilinear interpolation on the feature map output by the RPN network to output a 7×7 feature map.

[0017] The softmax function is:

[0018]

[0019] Among them, V represents the entire sample, i, j represents the serial number of the sample, n represents the total number of samples, V j represents the jth sample, S i Indicates V i The softmax value of .

[0020] The cross entropy loss function is:

[0021]

[0022] The ReCls reclassification network includes: first modifying the end of the category recognition branch after the feature map in the network structure, applying the mask branch branch to identify the mask corresponding to the category of one of the interlaced fibers in the fixed size feature map, then using this mask to screen out another fiber in the interlaced fibers, performing a new classification and recognition on this fiber, generating a mask corresponding to the category, and then applying these single fiber masks to the fixed size feature map to obtain the masked fixed feature map corresponding to each fiber, and finally sending these features to the full connected layers, outputting the reclassified fiber category, and forming the ReCls reclassification network.

[0023] The beneficial effects of the present invention are:

[0024] The present invention improves the Mask R-CNN neural network by replacing the original ResNet backbone network for extracting image features with a Transformer backbone network to better extract and transmit the features of wool and cashmere fibers and improve the classification accuracy of wool and cashmere fibers. The category loss function is changed to a softmax function and a cross-entropy loss function to accelerate the training of the neural network. The branch ends after the feature map in the network structure are modified, and a ReCls reclassification network is added after the category recognition branch to remove the influence of interlaced fibers and improve the category recognition accuracy. The present invention can quickly segment and recognize wool and cashmere fiber images, accurately classify images, and determine whether they are wool or cashmere, thereby realizing rapid and high-accuracy recognition of large quantities of wool and cashmere fibers. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of training and testing of the improved Mask R-CNN neural network of the present invention.

[0026] Figure 2 Schematic diagram of the improved Mask R-CNN neural network of the present invention.

[0027] Figure 3 Schematic diagram of the Transformer backbone network of the present invention.

[0028] Figure 4 Schematic diagram of the encoder layer of the present invention.

[0029] Figure 5 It is a schematic diagram of the decoder layer of the present invention.

[0030] Figure 6 Schematic diagram of the PANet middle layer of the present invention.

[0031] Figure 7 Schematic diagram of the ReCls reclassification network of the present invention.

[0032] Figure 8 This is a diagram of the annotation of the Labelme image annotation software.

[0033] Figure 9 This is a schematic diagram of the json file.

[0034] Figure 10 This is a schematic diagram of the json file in the COCO dataset format (images).

[0035] Figure 11 This is a schematic diagram of the json file in the COCO dataset format (categories).

[0036] Figure 12 This is a schematic diagram of the json file in the COCO dataset format (annotations).

[0037] Figure 13 It is a loss value curve graph of the Mask R-CNN neural network of the present invention.

[0038] Figure 14 2 is a fiber recognition accuracy curve of the Mask R-CNN neural network of the present invention.

[0039] Figure 15 This is a complex interlaced fiber recognition effect diagram of the Mask R-CNN neural network of the present invention.

[0040] Figure 16 It is a GUI interface made by Qt graphical interface design software. DETAILED DESCRIPTION

[0041] The present invention uses deep learning technology to train the model, automatically extracts and learns the features of the target, and continuously trains to update the weight coefficients of the network structure until the gradient is 0 and the value of the loss function reaches the minimum. At this time, the difference between the predicted value and the actual value reaches the minimum, and the accuracy of classification and recognition reaches the highest. There are many types of image segmentation networks based on deep learning. Here, an improved Mask R-CNN neural network is used, and Transformer is used as the backbone network to extract the feature maps of the image. The candidate ROI is then sent to the RPN region proposal network for label classification (foreground or background) and bounding box regression. On the basis of filtering out some candidate ROIs, the remaining ROIs are subjected to ROI Align operation, and a parallel branch for predicting the target mask (prediction mask branch) is added to the existing branch for bounding box recognition. A high-quality mask is generated for each target in the image, thereby improving the target detection accuracy.

[0042] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited to the following embodiments.

[0043] A wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network comprises the following steps:

[0044] 1) Collect 5,000 wool and cashmere fiber images to create a dataset, and annotate each fiber in each fiber image in the dataset to form training and test datasets in preparation for training the improved Mask R-CNN neural network;

[0045] 2) Input the data set into the improved Mask R-CNN neural network for training to obtain a trained improved Mask R-CNN neural network;

[0046] 3) Using the improved Mask R-CNN neural network to classify and identify wool and cashmere fibers

[0047] The annotation of step 1) includes: manually annotating each wool and cashmere fiber in each image in the dataset using Labelme image annotation software. After each image is annotated and saved, a corresponding json file will be generated, which stores the width, height, upper left corner coordinates and number of the bounding box of each wool and cashmere fiber in the image (such as Figure 9The annotation files corresponding to these wool and cashmere fiber images are converted into the COCO dataset format, and the improved Mask R-CNN neural network is trained using the COCO dataset format.

[0048] The COCO dataset format includes images, categories, and annotations, such as Figure 10 、 Figure 11 、 Figure 12 As shown in the figure, it is equivalent to concatenating the contents of many small json files of the training set and the test set into a large json file according to the requirements of the COCO dataset format. The large json file contains the annotation information corresponding to all images in the training set or the test set, which serves as the standard training data for the improved Mask R-CNN neural network. The hierarchical structure of the small json files is as follows Figure 9 As shown, the first is the curly brace of "shapes", which indicates the shape information of all objects. The content of the second curly brace is the shape information of each object, mainly consisting of "labels", "points" and "shape_type". "labels" indicates the label of each fiber; "points" indicates the anchor coordinates of the calibration area of ​​each fiber; "shape_type" indicates the shape of the calibration area. The hierarchical structure of the large json file (COCO dataset format) is as follows Figure 10 、 Figure 11 、 Figure 12 As shown in the figure, the large json file is actually a standard COCO dataset, which mainly consists of three parts: "images", "categories" and "annotations". "Images" represents the image information of all wool and cashmere fiber images, "categories" represents the hierarchical structure of the target category, where the parent class is fiber, subclass 1 is wool, and subclass 2 is cashmere. "Annotations" represents the annotation information of all individual fibers. The "segmentation" in the brackets represents the coordinate values ​​of the coordinate points of the calibration area of ​​each fiber, which are arranged in the order of x1 (the horizontal coordinate of the first calibration point), y1 (the vertical coordinate of the first calibration point), x2, y2, ..., and "bbox" represents the minimum horizontal rectangle that encloses the "segmentation".

[0049] The improved Mask R-CNN neural network includes the Transformer backbone network, the PANet intermediate layer, the RPN region proposal network, the ROI Align network, the head network, the ReCls reclassification network, and the category loss function. The improved Mask R-CNN neural network has made the following improvements:

[0050] 1. The backbone network used for feature extraction was changed from the original ResNet to the Transformer backbone network; 2. The intermediate layer used for feature fusion and enhancement was changed from the original ResNetFPN to the PANet intermediate layer; 3. The RPN region proposal network was adopted; 4. The ROI Align network (regional feature aggregation bilinear pooling network) was adopted; 5. The category loss function of the original Mask R-CNN network was changed to the softmax function plus the cross-entropy loss function; 6. The ReCls reclassification network was added after the category recognition branch in the original Mask R-CNN network.

[0051] The Transformer backbone network includes six encoder layers and six decoder layers, such as Figure 3 As shown, the six encoder layers are connected in sequence, the six decoder layers are also connected in sequence, and the last encoder layer is connected to all decoder layers. Each encoder layer includes a self-attention layer and a feed forward layer (feedforward neural network). The self-attention layer can help the current node focus not only on the current word, but also on the semantics of the context. Each decoder layer includes a self-attention layer, an encoder-decoder attention layer, and a feed forward layer (feedforward neural network). The encoder-decoder attention layer can help the current node obtain the key content that needs to be paid attention to. The processed image is input into the Transformer backbone network to generate feature maps of different sizes, thereby extracting the potential features of the wool and cashmere fiber images.

[0052] The Transformer backbone network not only uses a hierarchical Tokens-to-Token transformation mechanism to convert progressive image structures into Tokens, but also converts Tokens into Tokens through recursive integrated neighbors, which better constructs local structures and can further reduce the length of Tokens. The Transformer backbone network also uses an efficient deep-narrow backbone structure to enhance its feature richness, reduce information redundancy, significantly reduce the number of parameters, increase training speed, and improve recognition accuracy. It can better extract and transmit the features of wool and cashmere fibers, improve the classification accuracy of wool and cashmere fibers, and input the processed images into the Transformer backbone network of the improved Mask R-CNN neural network to generate feature maps of different sizes and extract the potential features of wool and cashmere fiber images.

[0053] The PANet middle layer (based on the path aggregation network under the proposed instance segmentation framework) is divided into four parts: 1. Feature Pyramid Network (FPN); 2. Bottom-up path enhancement, using accurate low-level positioning signals to enhance the entire feature hierarchy, thereby shortening the information propagation path; 3. Dynamic feature pooling, each proposal uses features from all levels of the pyramid to avoid arbitrary assignment of proposals; 4. Fully connected layer fusion, in order to further improve mask prediction, a complementary branch is created to capture different views for each proposal.

[0054] The low-level features of the feature pyramid network correspond to large targets, but the paths between high-level features and low-level features are long, which makes it difficult to obtain accurate positioning information. To shorten the path length between high-level and low-level features, the feature pyramid is enhanced with the position information of low-level features whenever possible, ensuring more complete feature fusion and more accurate and rapid feature location. Based on FPN, the middle layer of PANet innovatively adopts bottom-up path enhancement. This shortens the path between high-level and low-level features, utilizes the precise positioning signals stored in low-level features, and enhances the feature pyramid architecture. This allows for better extraction of feature maps at different scales for wool and cashmere fiber images, and further integrates the features to solve the problem of multi-scale detection.

[0055] The RPN region proposal network includes: scanning the backbone feature map generated by the PANet intermediate layer through a sliding window, proposing several rectangular candidate regions of different sizes, and the rectangular candidate region is represented by [x, y, w, h], where x is the horizontal coordinate of the upper left corner of the rectangular candidate region, y is the vertical coordinate of the upper left corner of the rectangular candidate region, w is the width of the rectangular candidate region (unit: pixel), and h is the height of the rectangular candidate region (unit: pixel).

[0056] The ROI Align network (regional feature aggregation bilinear pooling network) uses bilinear interpolation on the feature map output by the RPN network to output a fixed-size feature map of 7×7 with higher accuracy.

[0057] The head network includes a parallel category recognition branch, a prediction bounding box branch, and a prediction mask branch, and the category recognition branch and the prediction bounding box branch are arranged after the fully connected layer.

[0058] The category loss function in the category recognition branch is a softmax function and a cross entropy loss function.

[0059] The calculation formula of the softmax function is:

[0060]

[0061] Among them, V represents the entire sample, i, j represents the serial number of the sample, n represents the total number of samples, V i represents the jth sample, S i Indicates V i The softmax value of , that is, the probability of belonging to a certain category.

[0062] The cross entropy loss function mainly describes the distance between the expected output (probability) and the actual output (probability). That is, the smaller the cross entropy value, the closer the two probability distributions are. Assuming that probability distribution p is the expected output and probability distribution q is the actual output, H(p,q) is the cross entropy. Here, q(x) is replaced by S calculated by the softmax function. The calculation formula of the cross entropy loss function is:

[0063]

[0064] The cross entropy loss function is a more advanced loss function that uses logarithms. Compared with the original RPN classification loss function, it helps speed up the training of neural networks. This loss function is used to better guide the training of the network model and can more accurately classify each wool and cashmere fiber.

[0065] The ReCls reclassification network is set after the category recognition branch.

[0066] In the ReCls reclassification network: Figure 7As shown in the figure, the end of the category recognition branch after the feature map in the network structure is first modified, and the mask branch is applied to the fixed size feature map to identify the mask of the category corresponding to one of the interlaced fibers. This mask is then used to filter out the other fiber in the interlaced fibers, and a new classification and recognition of this fiber is performed. The mask of the corresponding category will also be generated, and then the masks of these single fibers are applied to the fixed size feature map to obtain the masked fixed feature map corresponding to each fiber. These features are sent to the fully connected layers, and the network will eventually output the reclassified fiber category (reclassification). This is equivalent to an improvement to the original Mask R-CNN neural network to form a ReCls reclassification network, which can improve the classification and recognition accuracy when the wool and cashmere fibers are complexly interlaced, and improve the detection stability.

[0067] Experimental verification

[0068] 5000 wool and cashmere fiber images with a size of 768×576 pixels were used as the training set, 600 images were used as the validation set, and 600 images were used as the test set. Because there is currently no public wool and cashmere fiber image library, Labelme image annotation software was used to manually annotate each wool and cashmere fiber in each image in the dataset, and the ROI area where each wool and cashmere fiber is located was marked.

[0069] Use the polygon annotation method in Labelme image annotation software to mark the anchor points of each fiber in the wool and cashmere fiber image, such as Figure 8 As shown in the figure. Every time a fiber is labeled, a set of information including multiple anchor coordinates and corresponding labels will be generated. This information is stored in the corresponding json file (generated by Labelme software after the fiber image is labeled). Figure 9 As shown in the figure, for an image, there are as many groups of information as there are fibers. The improved Mask R-CNN neural network will determine the fiber ROI area based on these different groups of information and remove redundant areas. The Transformer backbone network will extract features from the fiber ROI area. The PANet middle layer will further fuse the features. Finally, the loss value and accuracy of the bounding box are calculated based on the softmax function and the cross entropy loss function. The loss value will gradually decrease with the increase of the number of iterations, and the accuracy will gradually increase with the increase of the number of iterations. Figure 13 and Figure 14As shown in Figure 2, after training on 5,000 wool and cashmere fiber images, in the test set, the Mask R-CNN network can accurately predict the type, bounding box, and mask of each fiber in the image, as shown in Figure 2. Figure 15 shown.

[0070] The classification results are compared with those of the SVM (support vector machine) and the Mask R-CNN neural network before the improvement. The results are shown in Table 1. The improved Mask R-CNN neural network of the present invention has a very high accuracy rate in classifying wool and cashmere fibers, greatly improving the recognition efficiency of wool and cashmere fibers.

[0071] Table 1

[0072] Identification methods of wool and cashmere fibers Recognition accuracy SVM (Support Vector Machine) 80.5％ Mask R-CNN neural network before improvement 93.9％ Improved Mask R-CNN neural network of the present invention 96.6％

[0073] like Figure 16 To facilitate easy and convenient use of the trained network for wool and cashmere fiber identification, a GUI was created using Qt graphical interface design software, facilitating human-computer interaction and visualization. Within the designed GUI, clicking the Detect button selects a single wool and cashmere fiber image for classification and identification. The detection results are immediately output and displayed within the GUI. The displayed content includes: a fiber mask, a fiber bounding box, fiber type, and the probability of the fiber type. Using a color mask to select individual fibers for display provides excellent visualization.

Claims

1. A wool and cashmere fiber recognition method based on an improved Mask R-CNN neural network, comprising the following steps: 1) Collect several wool and cashmere fiber images to create a dataset, and annotate each fiber in each fiber image in the dataset; 2) Input the data set into the improved Mask R-CNN neural network for training to obtain a trained improved Mask R-CNN neural network; 3) Using the improved Mask R-CNN neural network to classify and identify wool and cashmere fibers; The improved Mask R-CNN neural network includes a Transformer backbone network, a PANet intermediate layer, an RPN region proposal network, a ROI Align network, a head network, a ReCls reclassification network, and a category loss function; the head network includes a fully connected layer, a category recognition branch, a prediction bounding box branch, and a prediction mask branch, and the category loss function in the category recognition branch is a softmax function and a cross entropy loss function; The head network includes a parallel category recognition branch, a prediction bounding box branch, and a prediction mask branch, and the category recognition branch and the prediction bounding box branch are set after the fully connected layer; The ReCls reclassification network is set after the category recognition branch; The Transformer backbone network consists of six encoder layers and six decoder layers. The encoder layer includes a self-attention layer and a feed forward layer. Each decoder The layers include a self-attention layer, an encoder-decoder attention layer, and a Feedforward layer; The PANet intermediate layer includes feature pyramid network, bottom-up path enhancement, dynamic feature pooling, and fully connected layer fusion; The ReCls reclassification network includes: first modifying the end of the category recognition branch after the feature map in the network structure, applying the mask branch branch to the fixed size feature map to identify the mask corresponding to the category of one of the interlaced fibers, then using this mask to screen out another fiber in the interlaced fibers, performing a new classification and recognition on this fiber to generate a mask corresponding to the category, then applying these single fiber masks to the fixed size feature map to obtain a masked fixed feature map corresponding to each fiber, and finally sending these features to the fully connected layers to output the reclassified fiber category to form the ReCls reclassification network; The RPN region proposal network includes: scanning the backbone feature map generated by the PANet intermediate layer through a sliding window, and proposing several rectangular candidate regions of different sizes; The ROI Align network uses bilinear interpolation on the feature map output by the RPN network to output a 7×7 feature map; The softmax function is: ； Among them, V represents the entire sample, i, j represents the serial number of the sample, n represents the total number of samples, V j represents the jth sample, S i Indicates V i The softmax value of .

2. The wool and cashmere fiber recognition method based on the improved Mask R-CNN neural network according to claim 1, characterized in that: The annotation in step 1) includes: manually annotating each wool and cashmere fiber in each image in the dataset using Labelme image annotation software and saving the result as a JSON file. The JSON file stores the width, height, upper left corner coordinates, and number information of each wool and cashmere fiber bounding box, and converting all JSON files into the COCO dataset format.

Images