Computer-aided leather coating detection method based on dual U-Net neural network

Abstract

The present invention relates to the field of leather coating detection; specifically, it relates to a computer-aided leather coating detection method based on a dual U-Net neural network, comprising the following steps: 1. collecting original leather longitudinal section image data and calibrating the coating area; 2. pre-processing the original leather longitudinal section image data and intercepting the spatial area of ​​the coating; 3. constructing and training a convolutional neural network model based on a dual U-Net and a leather coating detection objective function based on an adaptive loss function weight; 4. using the trained model to measure the leather coating of the detection task, and outputting the results of the thickness and area prediction of the leather coating area. The present invention realizes the automatic measurement of the thickness and area of ​​the leather coating with high accuracy.

Description

Technical Field

[0001] The invention relates to the field of leather coating detection, and in particular to a computer-aided leather coating detection method based on a dual U-Net neural network. Background Art

[0002] During the processing of leather, coating or transfer film is usually used to give the leather surface coating, which can bring excellent properties such as waterproof, anti-fouling and wear resistance to the leather. The thickness of the surface coating is a key factor affecting these properties, and it is also a key indicator in my country's leather standard system for evaluating physical properties such as friction color fastness of different types of leather. Therefore, it is very necessary to measure the thickness of the leather coating. The existing national standard "Determination of the Thickness of Surface Coatings in Physical and Mechanical Tests of Leather" (GB / T22889-2021) is to cut the leather to be tested into small sections, place it under a microscope lens, and image it through a camera. The coating part in the image is measured at discrete points by artificial point measurement, and then the average value is calculated as the thickness of the coating. It is worth noting that the surface of natural leather is usually uneven, which causes the coating in close contact with the leather to be uneven. Then, different testers will get different average thickness results due to the different positions of the collected discrete points. On the other hand, the difference between some coatings and leather is small, and the accuracy requirements for testers to identify the coating are also relatively high. Therefore, the existing leather coating thickness measurement method has large human errors.

[0003] In recent years, deep learning has made rapid progress in various tasks of natural image recognition. Since AlexNet reduced the image recognition error rate to 16.4% on the ImageNet dataset in 2012, deep neural networks have been widely used. In 2014, VGGNet, which uses a 19-layer deep network, reduced the error rate to 7.3%. In 2015, ResNet proposed adding a shortcut connection structure to the traditional convolutional network, which can reduce the gradient disappearance during model training and reduce the recognition rate of ImageNet to 3.57% in a 152-layer ResNet. This study uses a dual U-Net neural network model to identify and accurately measure leather coatings. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a computer-aided leather coating detection method based on a dual U-Net neural network, which realizes automatic measurement of leather coating thickness and area with high accuracy.

[0005] The computer-aided leather coating detection method based on dual U-Net neural network of the present invention comprises the following steps:

[0006] Step 1: Collect the original leather longitudinal section image data and calibrate the coating area of ​​the leather coating data used for training;

[0007] Step 2: pre-process the original leather longitudinal section image data and intercept the spatial area of ​​the coating;

[0008] Step 3: Build and train a convolutional neural network model based on dual U-Net and a leather coating detection objective function based on adaptive loss function weights;

[0009] Step 4: Use the trained model to measure the leather coating of the inspection task and output the predicted results of the thickness and area of ​​the leather coating area.

[0010] Preferably, in step 1, when collecting the original leather longitudinal section image data from the actual leather sample, the collected image data includes at least 1000 types of longitudinal section image data of leather samples with coating.

[0011] Preferably, in step 1, the coating area is calibrated on the leather longitudinal section image data using annotation software, and then the contour of the leather coating is marked on the surface layer to obtain the coating to be learned by the training model, and finally the influencing features of the coating performance are cross-calibrated. Each leather longitudinal section image contains the spatial position, thickness and calibrated coating of the coating.

[0012] Preferably, in cross-calibration, at least two training model parameter configuration strategies are used to train the same data set, and the training effects of different parameter strategies are compared with manual measurement results to select a more accurate parameter combination.

[0013] Preferably, in step 2, the collected original leather longitudinal section image is cropped by a multiple of 16 to ensure that the length of the image used for coating calibration and training is divisible by 16 to meet the needs of subsequent processing.

[0014] Preferably, in step 3, the dual U-Net convolutional neural network model is composed of two U-Nets, namely a simple U-Net in the first stage and a complex U-Net in the second stage. The original leather longitudinal section image data is encoded and decoded by the simple U-Net in the first stage, and the leather image background and target mask are output; the input and output of the simple U-Net in the first stage are element-wise multiplied, and the result of the multiplication is used as the input of the complex U-Net in the second stage. After encoding and decoding by the complex U-Net in the second stage, the coating and leather mask are output.

[0015] The so-called element-wise multiplication is to multiply the corresponding elements of tensor1 and tensor2 element by element to produce a new tensor with the same shape as the input tensor. This element-by-element multiplication operation is usually used to fuse feature maps. In this network, tensor1 is: the input of the first-stage simple U-Net, and tensor2 is: the output of the first-stage simple U-Net. For example, the number of images (grayscale images) involved in training at one time is n. The input of the first-stage simple U-Net is n two-dimensional tensors, that is, a 3D tensor composed of n original image matrices. The first dimension represents the number of images, and the second and third dimensions represent the height and width of the image respectively. The output of the first-stage simple U-Net has two parts, namely the background mask (background area) and the target mask (target area). Each part is a 3D tensor composed of n two-dimensional tensors.

[0016] Preferably, the first-stage simple U-Net includes an encoder, an ASPP, and four decoder blocks arranged in sequence, and the second-stage complex U-Net includes four encoder blocks, an ASPP, and four decoder blocks.

[0017] Preferably, the first-stage simple U-Net and the second-stage complex U-Net network structures both include a feature extraction network and a feature fusion network, the feature extraction network is applied to the downsampling layer, and the feature fusion network is applied to the upsampling layer.

[0018] In the first stage of simple U-Net, the VGG-19 network is used for encoding, and then ASPP (Atrous Spatial Pyramid Pooling) is used to extract the multi-scale features of the image, which is then passed through the decoding network (4 decoder blocks). In the decoding network, the weight of each input feature is adjusted through convolution operations, so that the model can learn the important features of the input data more finely, thereby helping the model to predict the results more accurately.

[0019] After the input image is downsampled for image feature extraction, it is necessary to restore the image to its original size in order to further classify the pixels (that is, semantic segmentation). This process is called upsampling. Generally speaking, the common upsampling methods are: bilinear interpolation, transposed convolution, and unpooling.

[0020] Preferably, the feature extraction network uses Relu, conv 3×3 and MaxPool, the downsampling layer input is The output is Represents the number of downsampling layers. The feature extraction process on each layer is as follows:

[0021]

[0022] Where: conv 3×3 represents a 3×3 convolution operation, Relu represents an activation function, and MaxPool represents a maximum pooling operation with a step size of 2. represents the downsampled i-th layer input, They represent the results of the first and second convolutions respectively, W1 and w2 represent the weight parameters of the convolution kernel in the first and second convolution operations respectively, b1 and b2 represent the bias parameters in the first and second convolution operations respectively, Stride represents the step size, Padding represents the padding method, and Pool_size represents the pooling window size.

[0023] That is to say, on each layer of the network, two convolution operations and one maximum pooling operation are performed.

[0024] Preferably, the second-stage complex U-Net structure introduces the following boundary attention mechanism:

[0025]

[0026] Among them: Sigmoid (S n ) represents the Sigmoid activation function, -Sigmoid(S n ) represents the inversion operation, MaxPool(S n ) represents the maximum pooling operation with a step size of 2, * represents the pixel-by-pixel product operation, S n It represents the nth decoder. It represents the nth decoder output, S n-1 It represents the input of the n-1th decoder, n=1, 2, 3, 4.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention firstly utilizes the simple U-Net structure in the first stage to accurately and automatically detect the overall area of ​​the cross-section of the leather product, and then the complex U-Net structure in the second stage distinguishes the coating part from the leather part, thereby realizing automatic identification of the leather coating area and greatly reducing human errors and computing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Simplified diagram of the dual U-Net neural network structure of the present invention.

[0030] Figure 2The overall structure of the dual U-Net neural network of the present invention.

[0031] Figure 3 The feature extraction and feature fusion network diagram of the present invention.

[0032] Figure 4 Schematic diagram of coating and leather segmentation image of the present invention. In the figure: 1. Image background; 2. Coating; 3. Leather. DETAILED DESCRIPTION

[0033] Embodiment 1:

[0034] like Figure 1-2 As shown, the computer-aided leather coating detection method based on the dual U-Net neural network described in this embodiment includes the following steps:

[0035] Step 1: Collect the original leather longitudinal section image data and calibrate the coating area of ​​the leather coating data used for training;

[0036] Step 2: pre-process the original leather longitudinal section image data and intercept the spatial area of ​​the coating;

[0037] Step 3: Build and train a convolutional neural network model based on dual U-Net and a leather coating detection objective function based on adaptive loss function weights;

[0038] Step 4: Use the trained model to measure the leather coating of the inspection task and output the predicted results of the thickness and area of ​​the leather coating area.

[0039] In step 1, the original leather section image data is directly collected using the existing microscopic digital image acquisition system. When the original leather longitudinal section image data is collected from the actual leather sample, the collected image data contains at least 1,000 types of leather sample longitudinal section image data with coating; the leather longitudinal section image data is calibrated with the coating area using the annotation software, and the outline of the leather coating is marked on the surface by manual delineation to obtain the coating to be learned by the training model. The annotation software can use one of labelme, labelimg, and Vott. After review, the influencing features of the coating performance are cross-calibrated, that is, at least two training model parameter configuration strategies are used to train the same data set, and the training effects of different parameter strategies are compared with the manual measurement results to select a more accurate parameter combination. Each leather longitudinal section image contains the spatial position, thickness and calibrated coating of the coating.

[0040] This embodiment adopts three training model parameter configuration strategies: Strategy 1, the loss function is the binary cross entropy loss function, the optimizer is Adam, the learning rate is 0.0001, the batch size is 16, the number of iterations is 150, and the data enhancement method is random horizontal flipping, random vertical flipping or random cropping of the image, and Dropout regularization is used to prevent overfitting; Strategy 2, the loss function is the Dice loss function, the optimizer is Adam, the learning rate is 0.001, the batch size is 8, the number of iterations is 100, the data enhancement method is random rotation, elastic deformation or color enhancement of the image, and Dropout regularization is used to prevent overfitting; Configuration strategy 3, the loss function is Dice loss and edge loss, the optimizer is Adam, the learning rate is 0.0005, the batch size is 16, the number of iterations is 75, the data enhancement method is random image scaling, random brightness / contrast adjustment, and Dropout regularization is used to prevent overfitting. Different parameter configuration strategies used during training will affect the image features learned by the final model, and these features will affect the final model prediction effect. Comparing the above three configuration strategies with the manual measurement results, configuration strategy 1 has higher training effect accuracy, and configuration strategy 1 is finally adopted.

[0041] To meet the needs of subsequent processing, the original leather longitudinal section image collected in step 2 is cropped by a multiple of 16 to ensure that the length of the image used for coating calibration and training is divisible by 16.

[0042] In step 3, the dual U-Net convolutional neural network model consists of two U-Nets, namely a simple U-Net in the first stage and a complex U-Net in the second stage. The original leather longitudinal section image data is encoded and decoded by the simple U-Net in the first stage, and the leather image background and target mask are output; the input and output of the simple U-Net in the first stage are element-wise multiplied, and the result of the multiplication is used as the input of the complex U-Net in the second stage. After encoding and decoding by the complex U-Net in the second stage, the coating and leather mask are output.

[0043] The so-called element-wise multiplication is to multiply the corresponding elements of tensor1 and tensor2 element by element to produce a new tensor with the same shape as the input tensor. This element-by-element multiplication operation is usually used to fuse feature maps. In this network, tensor1 is: the input of the first-stage simple U-Net, and tensor2 is: the output of the first-stage simple U-Net. For example, the number of images (grayscale images) involved in training at one time is n. The input of the first-stage simple U-Net is n two-dimensional tensors, that is, a 3D tensor composed of n original image matrices. The first dimension represents the number of images, and the second and third dimensions represent the height and width of the image respectively. The output of the first-stage simple U-Net has two parts, namely the background mask (background area) and the target mask (target area). Each part is a 3D tensor composed of n two-dimensional tensors.

[0044] The first-stage simple U-Net includes an encoder, ASPP, and 4 decoder blocks set in sequence, and the second-stage complex U-Net includes 4 encoder blocks, ASPP, and 4 decoder blocks.

[0045] The first-stage simple U-Net and the second-stage complex U-Net network structures both include a feature extraction network and a feature fusion network. The feature extraction network is applied to the downsampling layer, and the feature fusion network is applied to the upsampling layer.

[0046] In the first stage of simple U-Net, the VGG-19 network is used for encoding, and then ASPP (Atrous Spatial Pyramid Pooling) is used to extract the multi-scale features of the image, which is then passed through the decoding network (4 decoder blocks). In the decoding network, the weight of each input feature is adjusted through convolution operations, so that the model can learn the important features of the input data more finely, thereby helping the model to predict the results more accurately.

[0047] After the input image is downsampled for image feature extraction, it is necessary to restore the image to its original size in order to further classify the pixels (i.e., semantic segmentation). This process is called upsampling. Generally speaking, common upsampling methods include: bilinear interpolation, transposed convolution, and unpooling. In this embodiment, upsampling is achieved through deconvolution.

[0048] The upsampling layer mainly realizes the function of image feature fusion. Each step involves upsampling the feature map. First, the feature map of the corresponding downsampled part in the same layer network is connected to obtain more complete feature information, and then convolution operations are performed twice with 3×3 convolution kernels, and the ReLU activation function is used for both. Then, a 2×2 convolution kernel is used for convolution operations (up-convolution) to halve the number of feature channels.

[0049] There are two U-Net structures in the model. The expected output of the simple U-Net in the first stage is the mask of the target and the background, and the expected output of the complex U-Net in the second stage is the mask of the coating and leather. Since the gap between the background and the target is still relatively large, the simple U-Net in the first stage is just a U-Net model. Since the gap between the coating and the leather is still relatively small, the complex U-Net in the second stage is a U-Net model with boundary attention.

[0050] Feature extraction network uses Relu, conv 3×3 and MaxPool, the downsampling layer input is The output is Represents the number of downsampling layers. The feature extraction process on each layer is as follows:

[0051]

[0052] Where: conv 3×3 represents a 3×3 convolution operation, Relu represents an activation function, and MaxPool represents a maximum pooling operation with a step size of 2. represents the downsampled i-th layer input, They represent the results of the first and second convolutions respectively, W1 and W2 represent the weight parameters of the convolution kernel in the first and second convolution operations respectively, b1 and b2 represent the bias parameters in the first and second convolution operations respectively, Stride represents the step size, Padding represents the padding method, and Pool_size represents the pooling window size.

[0053] In the convolution operation, the weight parameters define how each convolution kernel is convolved with the input data, that is, how to perform weighted summation on each local area of ​​the input data through the weights. The size and shape of the weight parameters are usually determined by the size of the convolution kernel. For example, a 3x3 convolution kernel means that each weight matrix will have 3x3 parameters. The bias parameter provides an adjustable offset (i.e. a constant added after the weighted summation) for the output of the convolution operation, which is used to adjust the output of the network. The weight parameters W1, W2 and the bias parameters b1, b2 are automatically learned through the network training process so that the network can effectively extract the features of the input data.

[0054] The downsampling network has 4 layers. The input of the first layer is the original image, the input of the second layer is the output of the first layer, and so on. The input of the fourth layer is the output of the third layer. On each layer of the network, two convolution operations and one maximum pooling operation are performed.

[0055] In short, the downsampling layer mainly realizes the function of image feature extraction. It consists of two repeated 3×3 convolution operations, both of which use the rectified linear unit (ReLU) activation function and a maximum pooling operation with a step size of 2 for downsampling, and in each downsampling step, the number of feature channels will double.

[0056] Specifically, the number of feature channels will double, which means that in each layer of the downsampling step, the number of feature channels will double. For example, if the initial number of channels of the first layer input image is C, then after the first layer of downsampling steps, the number of channels becomes 2C. This is because in the two convolutions of each layer, the number of convolution kernels in the first convolution is 2, while the number of convolution kernels in the second convolution is 1.

[0057] The second stage complex U-Net structure introduces the following boundary attention mechanism:

[0058]

[0059] Among them: Sigmoid (S n ) represents the Sigmoid activation function, -Sigmoid(S n ) represents the inversion operation, MaxPool(S n ) represents the maximum pooling operation with a step size of 2, * represents the pixel-by-pixel product operation, S n It represents the nth decoder. It represents the nth decoder output, S n-1 It represents the input of the n-1th decoder, n=1, 2, 3, 4.

[0060] like Figure 3-Figure 4 In this embodiment, the size of the leather coating image is set to 1880×1160. Therefore, the size of the leather coating image training set, the leather coating image in the validation set, and the leather coating image to be identified in this embodiment are all 1880×1160 to ensure that the image length is divisible by 16. We first crop the image by a multiple of 16, and the resulting image size is 1872×1152.

[0061] Since we get the mask of the image in stages, we need to detect the output of the segmented detection here, so we need to adjust the loss function on the model, so we use an adaptive binary classification loss function. In addition, the model uses the Adam algorithm to calculate the gradient to ensure that the optimal solution of the gradient is found.

[0062] The adaptive binary classification loss function is usually used to deal with the problem of class imbalance and automatically adjust the loss weight according to the importance of each sample. In the adaptive binary classification loss function, the commonly used loss function is the weighted binary cross-entropy loss (Weighted Binary Cross-Entropy Loss). For each sample x, its calculation formula is as follows:

[0063] L x =-[y x ·log(p x )+(1-y x )·log(1-p x )]

[0064] in:

[0065] y x is the true label (0 or 1) of sample x.

[0066] p x is the probability that the model predicts that sample i is a positive class.

[0067] L x is the loss of the sample.

[0068] The total loss L is the weighted average of all samples. The weight of each sample is adjusted according to the importance of the sample. When the sample categories are unbalanced, giving a higher weight to the minority category can better handle the imbalance problem.

[0069] The specific process is: collect leather coating images through special devices (microscope, leather sample fixture, image acquisition system), such as Figure 2 As shown in the figure, the cropped image is used as the input of the first stage simple U-Net, that is, the first stage simple U-Net downsampling layer part, and the convolution operation is performed with a 3×3 convolution kernel and the number of channels is increased to 64, and then the convolution operation is performed with a 3×3 convolution kernel, and the number of channels remains unchanged. Then a maximum pooling operation with a stride of 2 is used. The situation is similar afterwards, increasing the number of channels of the feature map also reduces the area of ​​the image. There are 4 such operations in total. So the size of the image changes from 1872×1152 to 117×72.

[0070] Next, a 3×3 convolution kernel is used for convolution operation to further expand the number of channels of the image, that is, to enlarge the image size through deconvolution operation.

[0071] Then, it is cascaded with the feature map of the same size just obtained to get the input of the current upsampling layer. The so-called "cascade" refers to splicing two feature maps of different sizes together in a certain dimension, usually in the channel dimension, to create a larger feature map. It is a common operation in deep neural networks and is usually used to fuse high-resolution features with low-resolution features to obtain a better combination of semantic information and spatial information. Figure 2 The connection method of the encoder and decoder of the second stage of the complex U-Net is the "cascade" operation. In the U-Net network, the cascade operation is used to connect the feature map in the downsampling and the feature map in the upsampling so that the high-level features from the encoder can be used in the decoder to restore the details of the image.

[0072] The cascade operation helps to combine feature information from different resolutions to improve the performance of the model. Because in the image segmentation task, it is necessary to consider both the global and local information of the image. The "cascade" method allows the model to retain both fine details and high-level semantic information to complete the segmentation task more accurately.

[0073] Then, a convolution operation is performed with a 3×3 convolution kernel, and the number of channels is halved, and then a convolution operation is performed with a 3×3 convolution kernel again, and the number of channels remains unchanged. Then, the image size is enlarged through the deconvolution operation. Finally, the downsampling layer can be connected to the upsampling layer.

[0074] In this way, we can get a [0,1] feature map, and multiply this feature map with the original image as the network input of the second stage complex U-Net. The specific steps are similar to the first stage simple U-Net, but we add a boundary attention mechanism to it to get better results.

Claims

1. A computer-aided leather coating detection method based on a dual U-Net neural network, characterized in that: The steps include: Step 1: Collect the original leather longitudinal section image data and calibrate the coating area of ​​the leather coating data used for training; Step 2: pre-process the original leather longitudinal section image data and intercept the spatial area of ​​the coating; Step 3: Build and train a convolutional neural network model based on dual U-Net and a leather coating detection objective function based on adaptive loss function weights; Step 4: Use the trained model to measure the leather coating of the inspection task and output the results of the thickness and area prediction of the leather coating area; In step 3, the convolutional neural network model of the dual U-Net is composed of two U-Nets, namely, a simple U-Net in the first stage and a complex U-Net in the second stage. The original leather longitudinal section image data is encoded and decoded by the simple U-Net in the first stage, and the leather image background and target mask are output; the input and output of the simple U-Net in the first stage are element-wise multiplied, and the result of the multiplication is used as the input of the complex U-Net in the second stage, and the coating and leather mask are output after encoding and decoding by the complex U-Net in the second stage; the simple U-Net in the first stage includes an encoder, an ASPP, and 4 decoder blocks arranged in sequence, and the complex U-Net in the second stage includes 4 encoder blocks, an ASPP, and 4 decoder blocks; the network structures of the simple U-Net in the first stage and the complex U-Net in the second stage both include a feature extraction network and a feature fusion network, the feature extraction network is applied to the downsampling layer, and the feature fusion network is applied to the upsampling layer; There are two U-Net structures in the model. The expected output of the simple U-Net in the first stage is the mask of the target and the background, and the expected output of the complex U-Net in the second stage is the mask of the coating and leather. The complex U-Net in the second stage is a U-Net model with boundary attention.

2. A computer-aided leather coating detection method based on a dual U-Net neural network according to claim 1, characterized in that: In step 1, when collecting original leather longitudinal section image data from actual leather samples, the collected image data includes at least 1000 types of longitudinal section image data of leather samples with coatings.

3. A computer-aided leather coating detection method based on a dual U-Net neural network according to claim 1, characterized in that: In step 1, the coating area of ​​the leather longitudinal section image data is calibrated using annotation software, and then the outline of the leather coating is marked on the surface to obtain the coating to be learned by the training model, and finally the influencing features of the coating performance are cross-calibrated.

4. A computer-aided leather coating detection method based on a dual U-Net neural network according to claim 3, characterized in that: In cross-calibration, at least two training model parameter configuration strategies are used to train the same data set, and the training effects of different parameter strategies are compared with the manual measurement results to select a more accurate parameter combination.

5. According to the computer-aided leather coating detection method based on dual U-Net neural network described in claim 1, in step 2, the collected original leather longitudinal section image is cropped by a multiple of 16.

6. A computer-aided leather coating detection method based on a dual U-Net neural network according to claim 1, characterized in that: The feature extraction network uses Relu, and , the downsampling layer input is , the output is Represents the number of downsampling layers. The feature extraction process on each layer is as follows: ; in: represents a 3 x 3 convolution operation, Relu represents the activation function, represents a maximum pooling operation with a step size of 2, represents the downsampled i-th layer input, They represent the results of the first and second convolutions respectively, W1 and W2 represent the weight parameters of the convolution kernel in the first and second convolution operations respectively, b1 and b2 represent the bias parameters in the first and second convolution operations respectively, Stride represents the step size, Padding represents the padding method, and Pool_size represents the pooling window size.

Images