Deep multimodal fiber recognition method based on image and near-infrared spectral signals

Abstract

The present invention discloses a deep multimodal fiber recognition method based on image and near-infrared spectral signals. The method comprises the following steps: (1) acquiring an image and near-infrared spectral signal of a textile; (2) processing the image data into local slices and extracting a visual sequence of sub-images using a deep self-attention network; (3) processing the near-infrared spectral data and obtaining hierarchical near-infrared features using a multi-branch deep convolutional network; (4) matching the sequential features of the image and near-infrared signal through a scoring module based on an image and signal-related attention module, and capturing the correlation between the signal-to-image and image-to-signal features in both directions through a parameter-separated bidirectional feature consistency module to obtain fused features; (5) encoding the fused features; and (6) performing classification and loss calculation. The present invention can identify textiles with similar texture features and similar near-infrared spectral curves with high recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, near-infrared signal processing, and multimodal data processing, and specifically relates to a deep multimodal fiber recognition method based on image and near-infrared signal data. Background Art

[0002] Textile fiber identification is one of the main indicators for evaluating the quality of fabrics and clothing in the textile industry. Traditionally, identification relies on physical or chemical methods, which have disadvantages such as destructiveness, environmental pollution, high cost, and long time consumption. In recent decades, near-infrared spectroscopy has been widely used to identify the composition of textile fibers because near-infrared light has different reflectance spectra on different textile fiber components. Different textile fiber components, as well as its fast, non-destructive, green and pollution-free characteristics. Based on near-infrared reflectance spectral signals, statistics [1] or machine learning methods [2,3,4] have been used for spectral quantitative or qualitative analysis to automatically detect the composition of textile fibers. However, these methods are usually based on small data sets with few components and are relatively easy to distinguish. When natural fibers have very similar near-infrared spectral curves, such as cotton and linen, the recognition accuracy of these models decreases.

[0003] In recent years, ultra-high-resolution images obtained from optical microscopes have been used to non-destructively detect textile fiber composition using computer vision techniques. Textile images typically contain rich texture information, so texture-based models are often used to achieve this task. In recent years, several deep computer vision-based methods have been proposed for textile fiber recognition [5, 6, 7, 8]. However, when examining fabrics with similar texture features, such as polyester and nylon, texture-based image classification models struggle to distinguish subtle differences.

[0004] References

[0005] [1]Xu-dong Sun, Ming-xing Zhou, and Yi-ze Sun, "Variables selection for quantitative determination of cotton content in textile blends by nearinfrared spectroscopy," Infrared Physics&Technology, vol.77, pp.65–72, 2016.

[0006] [2]Xiangrong Zhang,Yujia Sun,Jingyan Zhang,Peng Wu,and Licheng Jiao,“Hyperspectral unmixing via deep convolutional neural networks,”IEEEGeoscience and Remote Sensing Letters,vol.15,no.11,pp.1755–1759,2018.

[0007] [3]Burkni Palsson,Magnus O Ulfarsson,and Johannes R Sveinsson,“Convolutional autoencoder for spectral–spatial hyperspectral unmixing,”IEEETransactions on Geoscience and Remote Sensing,vol.59,no.1,pp.535–549,2020.

[0008] [4]Zhengdong Liu,Wenxia Li,and Zihan Wei,“Qualitative classificationof waste textiles based on near infrared spectroscopy and the convolutionalnetwork,”Textile Research Journal,vol.90,no.9-10,pp.1057–1066,2020.

[0009] [5]Zunlei Feng,Weixin Liang,Daocheng Tao,Li Sun,Anxiang Zeng,andMingli Song,“Cu-net:Component unmixing network for textilefiberidentification,”International Journal of Computer Vision,vol.127,no.10,pp.1443–1454,2019.

[0010] [6]Hang Zhang, Jia Xue, and Kristin J.Dana, "Deep TEN: texture encoding network," in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017.2017, pp.2896–2905, IEEE ComputerSociety.

[0011] [7] Zhile Chen, Feng Li, Yuhui Quan, Yong Xu, and Hui Ji, "Deep texturerecognition via exploiting cross-layer statistical self-similarity," in Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 2021, pp.5231–5240.

[0012] [8] Jia Xue, Hang Zhang, and Kristin J. Dana, "Deep texture manifold forground terrain recognition," in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018.2018, pp.558–567, IEEE Computer Society. Summary of the Invention

[0013] In response to the problems existing in the prior art, the present invention provides a deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data.

[0014] The present invention uses two data forms, near-infrared spectral signals and images, to solve the problem of textile fiber recognition. In order to enable the model to benefit from multimodal information, considering the correlation between near-infrared signals and textile images containing texture information, the present invention proposes a deep textile fiber fusion network. Specifically, the present invention first applies a multi-branch deep convolutional network to obtain hierarchical near-infrared features, and uses a deep self-attention network to extract the visual sequence of sub-images. Then, the present invention matches the sequential features of the image and near-infrared signal based on a hard scoring module, and a bidirectional feature consistency module based on parameter separation to capture the correlation between the signal-to-image and image-to-signal features in two directions. The present invention verifies the effectiveness of the proposed method by quantitative and qualitative analysis of textile fiber composition, compared with single and multimodal benchmarks.

[0015] The technical solution of the present invention is specifically described as follows.

[0016] A deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data comprises the following steps:

[0017] (1) Acquiring images and near-infrared spectral signals of textiles

[0018] The image is represented as x∈R H×W×C , the near-infrared spectral signal is expressed as: y∈R D×T , where H and W are the length and width of the image, and C is the number of channels of the image, which defaults to 3. D and T are the band dimension and number of channels of the near-infrared spectral data.

[0019] (2) The image data is processed into local slices and input into the visual feature encoding network to perform encoding calculations on the local slices of the image.

[0020] We take the source image x i ∈R H×W×C Convert to 2D patches Where W and H are the length and width of the feature map, C is the number of channels, and P is the size of the patch. We use a 1×1 convolutional embedding layer to flatten the patch and map into D dimensions, resulting in a size-invariant local feature.

[0021] We apply a standard self-attention network architecture as a block, which consists of alternating layers of multi-head self-attention layers and multi-layer perceptron layers. We apply a normalization layer before each block. The entire encoding can be expressed as:

[0022]

[0023] Q l =MSA(LN(Q l-1 ))+Q l-1 ,

[0024] Q l+1 =MLP(LN(Q l ))+Q l .

[0025] Represents a position embedding vector, which is added to the patch embedding to preserve the position information. MLP multi-layer perceptron layer, Q0 is the output after the first layer encoding, Q l is the output after encoding in the first layer, and MSA is the self-attention in the multi-head self-attention mechanism layer;

[0026] The self-attention calculation process in the MSA multi-head self-attention mechanism layer is as follows:

[0027]

[0028] Where: Q, K, V are the input vector image visual vector x and W respectively q 、W k 、W v The learnable variable is obtained by multiplying the three weights, and then h groups of self-attention are defined to concatenate the results to obtain the final vector.

[0029] (3) Processing and encoding of near-infrared spectral data

[0030] Spectral data is a long sequence of reflectance values sorted by wavelength. Furthermore, in most cases, reflectance values at adjacent wavelengths are correlated. Therefore, deep neural networks, especially those for time series, can extract features from spectral data and make relevant predictions when labeled spectral data is available.

[0031] First, two one-dimensional convolutional residual networks are used to extract features from near-infrared spectral data. Different convolution kernels are used in the two branches, and the resulting vectors containing multi-scale semantic information are concatenated.

[0032] In order to fully capture the local spatial features, we extract features at different levels and upsample the feature sequences at different levels to obtain the same resolution as the low-level feature maps through linear interpolation, and then obtain the hierarchical fusion features P∈R N×T , where N is the number of channels of the one-dimensional feature map, and T is the dimension of each fused feature; the convolution kernels of the two convolutional network branches are 3×1 and 5×1 respectively.

[0033] (4) Perform multimodal feature dimension processing and feature fusion

[0034] The hierarchical spectral features P∈R obtained using CNN N×T And the visual features Q∈R obtained by the self-attention encoder N×D, perform multimodal feature fusion calculation. First, convert the Q vector to the same feature dimension as P through a linear function projection layer to obtain The features of the two modalities are input into the feature fusion module for calculation. The details are as follows:

[0035] ① Design a hard scoring module to perform correlation mining and match the sequential features of images and near-infrared signals

[0036] For each near-infrared feature P i ∈R T (i=1..., N), the goal of the module is to mine the most relevant visual feature sequence by hard scoring

[0037] First define a correlation embedding matrix H∈R T×T , expressed as:

[0038]

[0039] Where i and j are the serial numbers of each near-infrared spectral feature and image visual feature, || || is the first-order norm, and H i，j is the correlation matrix obtained under the corresponding i, j numbers.

[0040] Next, design a hard scoring mechanism to calculate the most relevant index:

[0041]

[0042] Among them H i is the correlation matrix, r i is the obtained index. Based on the obtained index value, the near infrared feature P i The corresponding most relevant image visual features can be expressed as

[0043] ② Design a bidirectional feature consistency module to further match related features;

[0044] Although the hard scoring mechanism can perform cross-modal sequence matching, it may still lose relevant information at other embedding positions. Therefore, a bidirectional feature consistency layer is designed to optimize the feature fusion process, in which an attention weight is provided to each embedding, which includes two directions: P (near-infrared features) to Q (image visual features) and Q to P.

[0045] Two global similarity matrices are defined and can be expressed as:

[0046]

[0047]

[0048] H and U are expressed as:

[0049] H=[P i ;P i ⊙Q j ],

[0050] U=[Q j ;P i ⊙Q j ],

[0051] where w s , w m ∈R 2T are independent learnable parameters for feature transfer in two directions, p,q∈R T , [;] represents cross-row connection, and ⊙ is the multiplication of corresponding elements.

[0052] Calculate the soft weight from P to Q, and then pay attention to each position of Q. The processing process can be expressed as:

[0053]

[0054] a j =softmax(S i ),

[0055] Where S is the obtained weight matrix, the softmax function obtains the normalized result, and a is the probability value.

[0056] Similarly, we can get the attention weight from Q to P through the following process:

[0057]

[0058] b i =softmax(M j ).

[0059] Then we connect the weighted vectors of each position of P and Q and get a fusion feature

[0060] (5) Perform fusion feature encoding process

[0061] Define C = {c k ∈R D ,k=1,…,K} represents a set of learnable cluster center weights. By applying softmax to the residual error vector, the soft histogram H∈R K×D , the specific expression method is:

[0062]

[0063] where sk is the center of each cluster c k The smoothing coefficient, V is the fusion feature obtained in step 7

[0064] (6) Classification and loss calculation

[0065] The obtained encoded vector H is connected to the classifier, and the probability is output through the softmax function to determine which component category the textile belongs to. The cross entropy loss function is used to calculate the loss of the entire network.

[0066] Compared to existing technologies, the present invention is advantageous in that it can identify fabrics with similar texture features, such as polyester and nylon, and natural fibers with very similar near-infrared spectral profiles, such as cotton and linen, with high recognition accuracy. We collected a near-infrared microscopy image dataset containing over 10,000 textile samples with near-infrared signals and microscopy images. On this dataset, our network was compared with one-dimensional signal processing-based methods, including DBLSTM-WS, SSC-Net, CIS-Net, and MCN, as well as with state-of-the-art computer vision-based models for textile / material recognition tasks, including CU-Net, DEP, DeepTEN, and ViT. Our model outperformed all single-modal methods in classification and five principal component regression tasks. Our network was compared with several state-of-the-art multimodal models, including the Multimodal Transformer, Attention Bottlenecks for Multimodal Fusion (MBT), Sequential Learning AE, and BIDAF. The results show that our model outperforms all baselines in recognition accuracy, and the proposed model achieves state-of-the-art results in five regression tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The overall architecture of the deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data is proposed by fusing near-infrared signals and textile microscopic images. DETAILED DESCRIPTION

[0068] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0069] The present invention provides a deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data. The present invention first applies a multi-branch deep convolutional network (two-way convolutional network) to obtain hierarchical near-infrared features, and uses a deep self-attention network to extract the visual sequence of sub-images; then, the present invention uses a hard scoring module to match the sequential features of the image and near-infrared signal, and uses a parameter-separated bidirectional feature consistency module to capture the correlation between the near-infrared signal to the image and the image to the near-infrared signal features in two directions; then, the fused features are encoded for final classification and recognition. Its overall architecture is as follows Figure 1 shown.

[0070] A deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data is as follows:

[0071] 1. Obtain images and near-infrared spectral signals of textiles.

[0072] The image is represented as x∈R H×W×C , the near-infrared spectral signal is expressed as: y∈R D×T , where H = 224, W = 224, and C = 3, with a default value of 3. D and T are the band dimension and number of channels of the near-infrared spectral data. Band dimension D represents the average intercepted features from the 900nm-1700nm band, with a dimension of 912. T is the number of data points collected, expressed as the number of channels, with a dimension of 3.

[0073] Second, process the image data into local slices and input them into the encoding network. We transform the source image into two-dimensional slice patches. Where C = 3, P = 7 is the size of the patch. We use a 1 × 1 convolutional embedding layer to flatten the patch and map into D dimensions, D = 228, to obtain a size-invariant local feature.

[0074] Perform encoding calculations on local slices of an image. Figure 1 , we apply the standard self-attention network architecture as our block, which consists of alternating layers of multi-head self-attention layers and multi-layer perceptron layers. A normalization layer is applied before each block. The entire encoding can be expressed as:

[0075]

[0076] Q l =MSA(LN(Q l-1 ))+Q l-1 ,

[0077] Q l+1 =MLP(LN(Q l ))+Q l .

[0078] Represents a position embedding vector, which is added to the patch embedding to preserve the position information. In this embodiment, the number of layers of the multilayer perceptron is 2, Q is the output after encoding at each layer, where D = 228, the depth of the multi-head self-attention network in MSA is 6, and the number of multi-head attention is 8.

[0079] 3. Process and encode near-infrared spectral data.

[0080] First, we use two one-dimensional convolutions to form a residual network to extract features from the near-infrared spectral data. We use 3×1 and 5×1 convolution kernels in the two branches respectively and concatenate the resulting vectors containing multi-scale semantic information. Figure 1 As shown in , in order to fully capture the local spatial features, we extract features at different levels and upsample the feature sequences at different levels to obtain the same resolution as the low-level feature maps through linear interpolation. Then we get the hierarchical fusion feature P∈R N×T , where N is the number of channels of the one-dimensional feature map, and T is the dimension of each fused feature. The convolution kernels of the two convolutional network branches are 3×1 and 5×1, respectively, N=1024, and T=228.

[0081] 4. Perform multimodal feature dimension processing and feature fusion.

[0082] The hierarchical spectral features P∈R obtained using CNN 1024×228 And the visual features Q∈R obtained by the self-attention encoder 1024 ×228 , perform multimodal feature fusion calculation. First, convert the Q vector to the same feature dimension as P through a linear function projection layer to obtain The features of the two modalities are input into the feature fusion module for calculation.

[0083] For the multimodal feature dimension processing and feature fusion module, a hard scoring module is first designed to perform correlation mining. i ∈R T (i=1..., 1024), the goal of the module is to mine the most relevant visual feature sequence We define a correlation embedding matrix H∈R 228×228 , expressed as:

[0084]

[0085] Where i and j are the serial numbers of each near-infrared spectral feature and image visual feature, || || is the first-order norm, and H is the obtained correlation matrix.

[0086] Next, the most relevant index r is calculated through a hard scoring mechanism:

[0087]

[0088] The dimension of r is 228, where each value represents the correlation score of the two modal features at the corresponding position.

[0089] Although the hard scoring mechanism can perform cross-modal sequence matching, this approach may lose relevant information at other embedding positions. To this end, we design a bidirectional feature consistency layer to optimize the feature fusion process, in which an attention weight is provided to each embedding, which contains two directions, including P (near-infrared features) to Q (image visual features) and Q to P. We define two global similarity matrices, which can be expressed as:

[0090]

[0091]

[0092] H and U are expressed as:

[0093] H=[P i ;P i ⊙Q j ],

[0094] U=[Q j ;P i ⊙Q j ],

[0095] where w s , w m ∈R 456 are independent learnable parameters for feature transfer in two directions, p,q∈R 228

[0096] Then the soft weight from P to Q is calculated, and then attention is paid to each position of Q. The processing process can be expressed as:

[0097]

[0098] a j =softmax(S i ),

[0099] Get the normalized correlation probability value a j A floating point value.

[0100] Similarly, we can get the attention weights from Q to P through the following process:

[0101]

[0102] b i =softmax(M j ).

[0103] Then connect the weighted vectors of each position of P and Q to obtain a fusion feature

[0104] 5. Carry out the fusion feature encoding process.

[0105] Define C = {c k ∈R 64 ,k=1,…,K} represents a set of learnable cluster center weights. By applying softmax to the residual error vector, the soft histogram H∈R 16×64 , the specific expression method is:

[0106]

[0107] where s k is the center of each cluster c k The smoothing coefficient, V is the fusion feature obtained in step 7

[0108] 6. Carry out classification and loss calculation.

[0109] The resulting encoded vector H is connected to a classifier, and the softmax function outputs probabilities to determine which component category the textile belongs to. The cross-entropy loss function is then used to calculate the loss of the entire network. Specifically, the fused vector H is flattened into a 16*64 one-dimensional vector, and the classification result is output through a fully connected layer. Finally, a fully connected layer is used to obtain an output vector with a dimension equal to the number of categories. The network training process uses SGD as the optimizer, with an input image size of 224*224 and a training data batch size of 64. The learning rate starts at 0.004 and is divided by 10 when the error stabilizes. The decay rate is set to 0.0001, and the momentum is set to 0.9.

[0110] We collected a near-infrared microscopy image dataset containing more than 10,000 textile samples with near-infrared signals and microscopy images.

[0111] On this dataset, we first compare our network with one-dimensional signal processing-based methods, including DBLSTM-WS, SSC-Net, CIS-Net, and MCN, as well as with state-of-the-art models for textile / material recognition tasks based on computer vision, including CU-Net, DEP, DeepTEN, and ViT. The experimental results in Table 1 show that our model outperforms all single-modal methods in classification and five principal component regression tasks. We compare our network with several state-of-the-art multimodal models, including the Multimodal Transformer, Attention Bottlenecks for Multimodal Fusion (MBT), Sequential Learning AE, and BIDAF. We also utilize a widely used feature fusion solution as a baseline. The experimental results shown in Table 1 show that our model outperforms all baselines in recognition accuracy, and the proposed model achieves state-of-the-art results in the five regression tasks.

[0112] Table 1 Comparison of the proposed method with single and multimodal baselines in terms of overall accuracy (ACC) and root mean square error (RMSE) for each major textile fiber component

[0113]

Claims

1. A deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data, characterized in that: The steps include: (1) Acquire images and near-infrared spectral signals of textiles; (2) Processing the image data into local slices and performing encoding calculations on the local slices of the image; (3) Processing and encoding of near-infrared spectral data; (4) Perform multimodal feature dimension processing and feature fusion; (5) Perform fusion feature encoding process; (6) Classification and loss calculation; including: In step (4), the image visual feature vector is first projected through a linear function layer. Convert to near-infrared spectrum hierarchical feature vector The same feature dimension, we get ; Where R represents the dimension of the feature, N is the number of channels of the one-dimensional feature map, T is the dimension of each fused feature, and D is the transformation dimension of the patch and map; Then the features of the two modalities are fused; the details are as follows: ① Perform correlation mining based on the hard scoring module to match the sequential features of images and near-infrared signals; For each near-infrared feature The goal of the hard scoring module is to mine the most relevant visual feature sequences by hard scoring. ; First define a correlation embedding matrix , expressed as: ， Where i and j are the serial numbers of each near-infrared spectral feature and image visual feature, || || is the first-order norm, is the correlation matrix obtained under the corresponding i, j numbers; Next, a hard scoring mechanism is used to calculate the most relevant index: ， in is the correlation matrix, is the obtained index; based on the obtained index value, the near infrared feature The corresponding most relevant image visual feature is expressed as ; ② Further matching related features based on the bidirectional feature consistency module; The bidirectional feature consistency module is used to optimize the feature fusion process. It provides an attention weight for each embedding, which includes two directions: the near-infrared spectrum hierarchical feature vector P to the image visual feature vector Q, and the image visual feature vector Q to the near-infrared spectrum hierarchical feature vector P; Define two global similarity matrices, expressed as: ， H and U are expressed as: ， in are independent learnable parameters for feature transfer in two directions, are near-infrared spectral features and image visual features, respectively. [;] indicates cross-row connection. It is the element-wise multiplication; Calculate the soft weight from P to Q, and then pay attention to each position of Q to obtain the attention weight from P to Q. The processing process is expressed as: ， in To obtain the weight matrix, the softmax function obtains the normalized result. is the probability value; Similarly, the attention weight from Q to P is obtained, and the processing process is expressed as: ； Finally, the weighted vectors of each position of P and Q are connected to obtain the fusion feature .

2. The deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data according to claim 1 is characterized in that: In step (2), the source image Convert to 2D patches , where W and H are the length and width of the feature map, C is the number of channels, and P is the size of the patch; Use a 1×1 convolutional embedding layer to flatten the patch and map into D dimensions, thus obtaining a local feature of constant size; We apply the standard self-attention network architecture as a block, which consists of alternating layers of multi-head self-attention layers and multi-layer perceptron layers. A normalization layer is applied before each block. The entire encoding is represented as: ， represents a position embedding vector, which is added to the patch embedding to preserve the position information, MLP is a multi-layer perceptron layer, is the output after first layer encoding, is the output after encoding at layer l; MSA is the self-attention in the multi-head self-attention mechanism layer, and the calculation process is as follows: ， Where: Q, K, V are the input vector image visual vector x and W respectively q 、W k 、W v The learnable variable is obtained by multiplying the three weights, and then h groups of self-attention are defined to concatenate the results to obtain the final vector.

3. The deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data according to claim 1 is characterized in that: In step (3), a multi-branch deep convolutional network is used to obtain the near-infrared spectrum hierarchical features, that is, the hierarchical fusion features .

4. The deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data according to claim 1 is characterized in that: In step (3), two convolutional residual networks are used to extract features from the near-infrared spectral data, and the convolution kernels of the two convolutional residual networks are 3×1 and 5×1 respectively.

5. The deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data according to claim 1 is characterized in that: In step (5), define Represents a set of learnable cluster center weights, and the soft histogram is calculated by applying softmax to the residual error vector , the specific expression method is: ， in Each cluster center The smoothing coefficient, V is the fusion feature, .

6. The deep multimodal textile fiber recognition method based on image and near-infrared spectral signal data according to claim 1, characterized in that: In step (6), the obtained encoded vector H is connected to the classifier, and the probability is output through the softmax function to determine which component category the textile belongs to, and the cross entropy loss function is used to calculate the loss of the entire network.

Images