A convolutional neural network ground material image classification method based on LBP features

Abstract

The present invention provides a convolutional neural network ground material image classification method based on LBP features, including extracting a surface material data set, and also including the following steps: generating a ground material recognition model based on the LBP operator; using the images in the training set to train the ground material recognition model to generate an optimized ground material recognition model; using the images in the test set to test the optimized ground material recognition model, and outputting the test results. The present invention proposes a classification method that combines the traditional LBP feature descriptor and the deep learning model method based on prior knowledge, that is, when training the convolutional neural network model, in addition to using the collected original pictures for training, the LBP descriptor is also used to perform feature engineering on the ground material image to extract the texture features of the image and add them to the model, so as to achieve the purpose of making the model quickly fit the actual road surface data.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a convolutional neural network ground material image classification method based on LBP features. Background Art

[0002] At present, the mainstream ground material recognition method is based on ground image classification. After extracting the main features of the ground image obtained by the camera, a suitable classification method is selected for judgment. Most of the decisions in the field of unmanned driving control are based on a single road condition, but the actual application scenarios are often more complex than the ideal scenarios. At this time, the error caused by the difference in ground material will affect the control accuracy of the vehicle. In recent years, with the development of deep learning algorithm application research in the field of computer vision, various neural network models based on back propagation have emerged, and more and more new models have shown greater potential in image recognition than traditional feature engineering. In 2012, Krizhevsky et al. proposed the AlexNet model, and first proposed the ReLU activation function, Dropout sparse connection method, etc., which greatly promoted the development of deep learning in the field of image. The subsequent VGGNet, GoogleLeNet, etc. have continuously improved the accuracy and reliability of deep learning models in image recognition, and gradually began to replace the feature engineering methods that have been developed for decades. Ground material recognition methods also have new possibilities.

[0003] When a car is driving, in addition to factors such as road conditions and obstacles, the material, texture, and smoothness of the ground will also affect the friction coefficient between the vehicle and the ground. This will affect the vehicle's start-up, operation, and braking. At this time, the error caused by the difference in ground material will affect the vehicle's control accuracy. When the amount of data is sufficient, the use of mainstream neural network models in the current deep learning field, such as VGG and ResNet, can achieve a good accuracy rate for ground material recognition. However, due to the complex road conditions, even the same type of ground has certain differences in different regions. The model is often different from the training verification when it is actually used for unmanned driving road detection. At this time, the use of transfer learning strategies and the use of real-time collected data to continue training the model can solve this problem. Since transfer learning requires a certain amount of new data to fine-tune the model, and unmanned vehicle driving has a certain degree of real-time nature, transfer learning cannot collect enough data on site to fine-tune the model. Therefore, it brings difficulties and challenges to real-time road detection.

[0004] Image material recognition methods based on deep learning methods have been very popular in recent years. Material recognition aims to identify the main objects in natural material images and the material categories to which they belong.

[0005] The invention patent application with publication number CN107341518A discloses an image classification method based on a convolutional neural network, which includes the following steps: construction of a deep convolutional neural network, improvement of the deep convolutional neural network, training and testing of the deep convolutional neural network, and optimization of network parameters. The disadvantage of this method is that the network depth is not enough, and it is generally suitable for image classification with large differences in the color and shape of the main target in the image, but it is not sensitive to features such as ground material and texture, and the recognition effect is poor. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a convolutional neural network ground material image classification method based on LBP features. According to prior knowledge, a classification method combining traditional LBP feature descriptors and deep learning model methods is proposed. That is, when training the convolutional neural network model, in addition to using the collected original pictures for training, the LBP descriptor is also used to perform feature engineering on the ground material image - the texture features of the extracted image are added to the model, so as to achieve the purpose of making the model quickly fit the actual road surface data.

[0007] The present invention provides a convolutional neural network ground material image classification method based on LBP features, comprising extracting a ground surface material data set, and further comprising the following steps:

[0008] Step 1: Generate a ground material recognition model based on the LBP operator;

[0009] Step 2: Use the images in the training set to train the ground material recognition model to generate an optimized ground material recognition model;

[0010] Step 3: Use the images in the test set to test the optimized ground material recognition model and output the test results.

[0011] Preferably, the surface material data set includes 5 types of terrain, N images are selected from each type of terrain as a training set, and the remaining images are selected as a test set.

[0012] In any of the above solutions, preferably, the ground material recognition model is based on the MobileNet v2 network and is a model that combines the LBP operator with the CNN network.

[0013] In any of the above solutions, it is preferred that an attention module is added to the ground material recognition model, and the attention module serves as an auxiliary classification module of the neural network.

[0014] In any of the above schemes, it is preferred that the attention module uses the LBP operator to extract the texture features of the image from the original image, and after the convolution layer for extracting the texture features, a convolution kernel of the size of the feature map is used to perform convolution calculation to spatially integrate the feature map, and then a feature vector is obtained through an activation layer of a gate structure.

[0015] In any of the above schemes, it is preferred that the working method of the attention module is to use the convolution layer of MobileNet in the backbone network to extract image features, obtain a feature map of size 7*7, and then use deep convolution to activate the feature map of the input image to obtain an attention feature map, which is then input into the subsequent classification layer for image classification.

[0016] In any of the above solutions, preferably, the model training in step 2 includes the following sub-steps:

[0017] Step 21: Calculate the average pixel value and variance;

[0018] Step 22: Standardize the images before training.

[0019] In any of the above solutions, preferably, step 21 includes normalizing all image data to between 0 and 1, and calculating the size of pixel values ​​and image variance according to the three RGB channels respectively.

[0020] In any of the above solutions, preferably, step 22 includes subtracting the mean from the image data and dividing it by the variance, the formula is:

[0021] X=X-mean_value

[0022]

[0023] Among them, X represents image data, mean_calue represents the mean of the image, and std_value represents the variance of the image.

[0024] In any of the above solutions, preferably, the model test in step 3 is divided into data performance test and deployment online test.

[0025] In any of the above schemes, preferably, the data performance test includes inputting multiple images and corresponding labels as a test set into the network, and the model outputs the recognition accuracy, which represents the proportion of the number of images recognized correctly to the total number.

[0026] In any of the above schemes, preferably, the deployment of online testing includes capturing images with a camera and inputting them into an embedded device for preprocessing, and then outputting a recognition result through network calculation, where the recognition result is the type of ground material to which the image belongs.

[0027] The present invention proposes a convolutional neural network ground material image classification method based on LBP features, which adopts the Mobilenet network which is more suitable for industrial edge computing devices as the main method. The inference speed of edge devices is faster and more suitable for the application scenario of assisted unmanned driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flowchart of a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features according to the present invention.

[0029] Figure 2 Schematic diagram of different neighborhoods of several LBP operators according to an embodiment of the convolutional neural network ground material image classification method based on LBP features of the present invention.

[0030] Figure 3 The figure is a schematic diagram of an example of a surface material data set according to a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features of the present invention.

[0031] Figure 4 The figure is a schematic diagram of the LBP branch structure of a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features according to the present invention.

[0032] Figure 5 The figure is a schematic diagram of the MobileNet-LBP model structure according to a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features of the present invention.

[0033] Figure 6 It is a schematic diagram of the model evaluation results of a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features according to the present invention.

[0034] Figure 7 A schematic diagram of a road crack image data set according to a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features of the present invention.

[0035] Figure 8 The present invention is a flow chart of road material detection according to a preferred embodiment of the convolutional neural network ground material image classification method based on LBP features. DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0037] Embodiment 1

[0038] like Figure 1 As shown, step 100 is executed to extract a surface material dataset.

[0039] Execute step 110 to generate a ground material recognition model based on the LBP operator. The ground material data set includes 5 types of terrain, and N images are selected from each type of terrain as a training set, and the remaining images are used as a test set.

[0040] Execute step 120, use the images in the training set to train the ground material recognition model, and generate an optimized ground material recognition model. The ground material recognition model is based on the MobileNet v2 network, and uses the LBP operator combined with the CNN network. An attention module is added to the ground material recognition model, and the attention module serves as an auxiliary classification module of the neural network. The attention module uses the LBP operator to extract the texture features of the image from the original image. After the convolution layer for extracting the texture features, the convolution kernel of the feature map size is used to perform convolution calculations to spatially integrate the feature map, and then the feature vector is obtained through the activation layer of the gate structure. The working method of the attention module is to use the convolution layer of MobileNet to extract image features in the backbone network to obtain a feature map of size 7*7, and then use deep convolution to activate the feature map of the input image to obtain an attention feature map, which is then input into the subsequent classification layer for image classification.

[0041] Model training consists of the following sub-steps:

[0042] Step 21: Calculate the average pixel value and variance; normalize all image data to between 0 and 1, and calculate the pixel value and image variance according to the RGB three channels.

[0043] Step 22: The image is normalized before training. The image data is subtracted from the mean and divided by the variance. The formula is:

[0044] X=X-mean_value

[0045]

[0046] Among them, X represents image data, mean_calue represents the mean of the image, and std_value represents the variance of the image.

[0047] Execute step 130 to test the optimized ground material recognition model using the images in the test set and output the test results.

[0048] Model testing is divided into data performance testing and deployment online testing. The data performance test includes inputting multiple images and corresponding labels as test sets into the network, and the model outputs the recognition accuracy, which represents the proportion of the number of correct image recognitions to the total number. The deployment online test includes the camera capturing the image and inputting it into the embedded device for preprocessing, and then outputting the recognition result through network calculation. The recognition result is the ground material type to which the image belongs.

[0049] Embodiment 2

[0050] Aiming at the problem of low material recognition accuracy caused by the small amount of data in material image datasets and the difficulty in manually labeling local texture areas, the present invention proposes a material recognition method based on attention mechanism and deep convolutional neural network, the core of which is the material recognition deep convolutional neural network (MaterialNet). MaterialNet uses a deep residual network to extract features from images, and introduces an attention mechanism using the proposed cascaded atrous spatial pyramid pooling method, so that the network can adaptively focus on key areas containing texture features through end-to-end training, thereby effectively identifying local texture features of materials. Experiments were conducted on the FMD material dataset, and the overall recognition accuracy of MaterialNet can reach 82.3%.

[0051] 1. LBP features

[0052] LBP (local binary patterns) was first proposed by T.Ojala, M. It was proposed by D. Harwood in 1994. It is a binary description of the size relationship between a certain pixel of a grayscale image and the surrounding pixels. It is an operator used to describe the local texture features of an image. It has significant advantages such as rotation invariance and grayscale invariance. The present invention uses LBP feature sub-collection to collect local texture features of the image.

[0053] The original LBP operator is defined as taking the central pixel of the window as the threshold within a 3*3 window, comparing the grayscale values ​​of the 8 adjacent pixels with it, and if the surrounding pixel values ​​are greater than the central pixel value, the position of the pixel is marked as 1, otherwise it is 0. In this way, the 8 points in the 3*3 neighborhood can generate an 8-bit binary number (usually converted to a decimal number, namely the LBP code, a total of 256 types) after comparison, that is, the LBP value of the central pixel of the window is obtained, and this value is used to reflect the texture information of the area.

[0054] In order to adapt the LBP operator to texture features of different scales, Ojala et al. extended the 3×3 neighborhood to an arbitrary neighborhood, replaced the square neighborhood with a circular neighborhood, and used a bilinear interpolation algorithm to calculate the grayscale values ​​that did not completely fall on the pixel position. The radius and number of pixels of the improved LBP operator can be arbitrary. The radius is R and the sampling point is P to represent the sampling method. Different neighborhoods such as Figure 2 As shown, the parameters on the left are R=1, P=8, in the middle are R=2, P=16, and on the right are R=2, P=8.

[0055] 2. Ground material recognition model based on LBP operator

[0056] The convolutional neural network method performs well for recognition tasks in complex scenarios, but a large amount of labeled data is required to train a model with better performance. In addition, due to the differences in terrain in different regions, the distribution of features in the training set and the measured images is different. To solve this problem, transfer learning and domain adaptation methods can be used. This requires some calibration data and additional calculations when facing new scenes, while the unsupervised domain adaptation method requires a larger amount of calculations. These conditions restrict the speed and accuracy of recognition in the actual operation scenarios of unmanned vehicles. After some research and testing, we found that the model using the traditional LBP operator combined with CNN is not very effective in terrain recognition (data examples such as Figure 3 As shown in the figure, from left to right they are stone ground, cement ground, swamp ground, mud ground, and grass ground) tasks, and less computation is required using the transfer learning method under the same conditions.

[0057] 3. MobileNet v2

[0058] The MobileNet network series has been widely used in recent years and is a typical representative of lightweight networks. The MobileNet network structure is shown in Table 1. Based on the classic CNN network structure, the pooling layer and some fully connected layers are replaced by convolution layers. Its outstanding feature is the introduction of the convolution form of depthwise separable convolution. Standard convolution is to combine a set of convolution kernels with the input data to output a single channel feature. Depthwise separable convolution is a factorized convolution form, which decomposes the standard convolution into depthwise convolution and 1×1 pointwise convolution. Depthwise convolution is to apply a single fixed-size convolution kernel to each input channel, and then pointwise convolution is performed through a 1×1 convolution kernel to fuse channel information and output it. When using 3×3 depthwise separable convolution, the amount of calculation is 8 to 9 times less than that of standard convolution, with only a small loss in accuracy. MobileNetV2 is further optimized on the basis of MobileNetV1, integrating the residual connection idea of ​​resnet and improving the model structure, which further improves the model performance while reducing the number of parameters. When using MobileNetV2 to classify images, the amount of calculation and computing time are greatly reduced compared to networks such as vgg and resnet. Even if the CPU is used for calculation, it can reach 15FPS, which can meet the recognition task requirements of unmanned driving portable devices.

[0059]

[0060]

[0061] Table 1 MobileNetV2 network structure

[0062] 4. LBP-Attention branch

[0063] The Attention mechanism is derived from the human visual attention mechanism: focusing limited attention on key information, from paying attention to everything to paying attention to key points, thereby saving resources and quickly obtaining the most effective information. For Attention, it is a weight parameter allocation mechanism, the goal is to assist the model in capturing important information. The calculation process in the attention mechanism can be understood as a weighted summation, which will give different incentives to the image in different contexts, so that the network can focus on the features of interest during classification. Based on prior knowledge, this method uses the LBP features of the image through the Attention mechanism to assist in material image classification.

[0064] Based on the CNN model, we construct an attention module (LBP-Attention) by adding the LBP operator to extract the texture features of the image, as shown in Figure 4As shown in the figure, this module consists of a simple fully convolutional network, which filters the extracted featuremap to improve the classification effect of the model.

[0065] The LBP-Attention structure is shown in the figure. This branch is used as an auxiliary classification module of the network. It uses the LBP operator to extract the texture features of the image from the original image. After the convolution layer that extracts the texture features, it uses the convolution kernel of the feature map size to perform convolution calculations and spatially integrate the feature map. Then, it obtains the feature vector through the activation layer of the gate structure. The backbone network uses the convolution layer of MobileNet to extract image features and obtain a 7*7 feature map. Then, it uses depthwise convolution to activate the feature map of the input image to obtain an attention feature map, which is then input into the subsequent classification layer for image classification. The overall structure is as follows: Figure 5 shown.

[0066] 5. Specific implementation plan

[0067] Model Training

[0068] Training data preparation: Collect several images of different ground materials, classify them by folder, put the same images into the same folder, and the image size can be arbitrary. If the color difference of images of different categories is too large, data standardization is required. The method is as follows:

[0069] Step 1: Normalize all image data to between 0 and 1, calculate the pixel value and image variance according to the three RGB channels, and find the average pixel value and variance;

[0070] Step 2: Standardize the image before training, that is, subtract the mean from the image data and divide it by the variance:

[0071] X=X-mean_value

[0072]

[0073] This method is based on the improvement of mobilenet v2 image classification network, and the training and testing methods are carried out according to the normal classification network. Mobilenet v2 and LBP-MobileNet improved based on prior knowledge are used to test on the surface material dataset. The surface material dataset contains 676 images of 5 types of terrain in total. In the experiment, 20 images of each type are taken as the training set, and the remaining images are taken as the test set. The model input image uses a scaled RGB image of size 224*224 after whitening; the LBP branch input is the grayscale image after the texture features are extracted by the LBP operator. Different sampling strategies are used in the training and inference stages: random cropping is used during model training to crop the grayscale image to an image of size 56*56, and the center cropping strategy is used during inference verification. The learning rate in the initial stage of training is 0.001, and the regularization term attenuation coefficient is 0.0001; the stochastic gradient descent method is used for optimization during training, and the learning rate is decayed once every 50 iterations, with an attenuation coefficient of 0.9, so that the model parameters converge further. We use a variety of mainstream neural networks to classify and predict the data. The prediction results are as follows: Figure 6 shown.

[0074] The training software environment is: operating system is Ubuntu 20.04; deep learning framework is Pytorch 1.7.0;. The hardware information can be adjusted according to the existing environment. The following is a reference: CPU is Xeon (R) Silver 4114, memory is 64G, GPU is NVIDIA Quadro P5000.

[0075] The model testing phase is divided into data performance testing and deployment online testing. Model performance testing: multiple images and corresponding labels are input into the network as test sets, and the model outputs the recognition accuracy, which represents the proportion of the number of correct image recognitions to the total number. Online testing: After the camera captures the image, it is input into the embedded device for preprocessing, and then the recognition result is output through network calculation. The recognition result is the ground material type to which the image belongs.

[0076] The model evaluation results are as follows Figure 6 As shown in Figure 2, compared with MobileNet v2, the test accuracy of the improved LBP-MobileNet model on the dataset is improved from 86.2% to 89.1%, and the inference speed is only reduced by about 0.5%.

[0077] In addition to testing the effectiveness of the LBP-MobileNet model on the ground material dataset, we also verified it on the road crack image dataset. The road crack image dataset contains 155 road images (such as Figure 7As shown in the figure, some of them are ground images with cracks. On this basis, we collected and sorted 200 normal road images to expand the dataset. The accuracy of the MobileNet-v2 model on the dataset is 91.7%, and the LBP-MobileNet model improved based on prior knowledge has a test accuracy of 94.3%.

[0078] 6. Results Analysis

[0079] MobileNet has the advantages of simple structure, small amount of calculation and high accuracy in the field of mobile device image processing. Compared with networks such as VGG and GoogleNet, it has fewer parameters and is easier to fit data with a small number of samples, and the test effect is relatively good. From Table 2, it can be seen that the recognition accuracy of MobileNet on the ground material dataset is higher than that of models such as VGG and ResNet. Therefore, the proposed improved MobileNet algorithm is compared with the MobileNet-v2 classification algorithm, and the experimental results are shown below Table 2. After adding the Attention module with the LBP operator, the recognition error rate of the improved LBP-MobileNet algorithm on the ground material dataset is reduced to 10.9%, which is about 20% lower than before the improvement.

[0080]

[0081]

[0082] Table 2 Statistics of error rate and inference speed of six network structures on the surface material dataset

[0083] In order to verify the effectiveness of the LBP branch added based on prior knowledge in improving the model classification effect, we also replaced the basic model with ResNet50 and VGG-19, and added the LBP branch after some adjustments to the network for model fine-tuning and verification. The test accuracy on the ground material dataset increased by 4.1% and 1.6% respectively. This proves that the method of adding LBP branches to assist model classification based on prior knowledge is robust, and the backbone network used by the neural network to extract features is not limited to MobileNet-v2.

[0084] The experiment uses the circular LBP operator. This algorithm improves the limitations of the radius coverage based on the traditional LBP and expands the 3×3 domain to any neighborhood. The circular LBP operator uses a circular coverage with a radius of R as the neighborhood and selects different numbers of sampling points for actual application scenarios. The LBP feature descriptor is tested with different radii and sampling points, and the experimental results are shown in Table 3. The experimental results show that the highest test accuracy is obtained when the LBP radius is 2 and 8 sampling points are taken around. In actual application scenarios, fine-tuning can be performed according to the ground material and image size. While increasing the sampling points to a certain extent improves the accuracy, it will increase the inference time, which needs to be weighed when adjusting the model.

[0085] P＝8 P＝16 R＝1 12.1 11.4 R＝2 10.9 11.7

[0086] Table 3 Experimental results of LBP features with different radii and sampling points

[0087] like Figure 8 As shown in the figure, in the inference stage, the camera collects images and sends them to the edge computing device, the images are preprocessed, and the recognition results are output after passing through the network. After obtaining the recognition results, subsequent development can be carried out, and the network structure is simple and easy to expand.

[0088] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A convolutional neural network ground material image classification method based on LBP features, comprising extracting a ground surface material dataset, characterized in that: The following steps are also included: Step 1: Generate a ground material recognition model based on the LBP operator; Step 2: Use the images in the training set to train the ground material recognition model to generate an optimized ground material recognition model, which is based on the MobileNet v2 network and uses the LBP operator combined with the CNN network; add an attention module to the ground material recognition model, and the attention module serves as an auxiliary classification module of the neural network; the attention module uses the LBP operator to extract the texture features of the image from the original image, and after the convolution layer for extracting the texture features, uses the convolution kernel of the feature map size to perform convolution calculation and spatially integrate the feature map, and then obtains the feature vector through the activation layer of the gate structure; Step 3: Use the images in the test set to test the optimized ground material recognition model and output the test results.

2. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 1, characterized in that: The working method of the attention module is to use the convolution layer of MobileNet in the backbone network to extract image features, obtain a feature map of size 7*7, and then use deep convolution to activate the feature map of the input image to obtain an attention feature map, which is then input into the subsequent classification layer for image classification.

3. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 2, characterized in that: The model training in step 2 includes the following sub-steps: Step 21: Calculate the average pixel value and variance; Step 22: Standardize the images before training.

4. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 3 is characterized in that: The step 21 includes normalizing all image data to between 0 and 1, and calculating the size of pixel values ​​and image variance according to the three RGB channels.

5. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 4, characterized in that: The step 21 includes subtracting the mean from the image data and dividing it by the variance, the formula is: X=X-mean_value Among them, X represents image data, mean_calue represents the mean of the image, and std_value represents the variance of the image.

6. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 5, characterized in that: The model test in step 3 is divided into data performance test and deployment online test.

7. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 6, characterized in that: The data performance test includes inputting multiple images and corresponding labels as test sets into the network, and the model outputs the recognition accuracy, which represents the proportion of the number of images recognized correctly to the total number.

8. The convolutional neural network ground material image classification method based on LBP features as claimed in claim 7, characterized in that: The deployment online test includes collecting images with a camera and inputting them into an embedded device for preprocessing, and then outputting a recognition result through network calculation, where the recognition result is the type of ground material to which the image belongs.

Images