Road crack disease classification method and device based on semantic perception Mixup

Abstract

The invention discloses a road crack disease classification method and device based on semantic perception Mixup, and the method comprises the steps: carrying out the sampling of a road through a road condition rapid detection system, obtaining a road crack disease original image, and carrying out the image cleaning and image enhancement; marking images, and constructing a road crack disease multi-label data set to obtain a training set, a verification set and a test set; the training set is enhanced by fusing label semantics and a Mixup method; constructing a deep neural network architecture, and performing iterative training on the constructed deep neural network by using the enhanced training set and the verification set to obtain a road crack disease classification model; and testing the test set data by using the trained road crack disease classification model to obtain a predicted road disease category. According to the method, automatic classification of road disease cracks is facilitated, the problem of data imbalance is relieved, the effect of data enhancement is optimized, and the recognition accuracy of a small number of categories is improved.

Description

Technical Field

[0001] The present invention relates to the field of traffic disease detection, and in particular to a road crack disease classification method and device based on semantic perception Mixup. Background Art

[0002] Highways are the most populous and frequently used mode of transportation, and are also important infrastructure. As the use time increases, under the influence of multiple factors such as high temperature, freezing, heavy vehicle rolling, and water erosion, roads are prone to diseases. The most direct apparent disease is road cracks. Long-term and regular inspection and maintenance of roads are necessary and cannot be ignored. Collecting road image data through a rapid road condition detection system and identifying cracks have become an important part of road inspection and maintenance. However, the analysis and interpretation of road crack images after collection is challenging and requires human experts with relevant field knowledge and certain experience, which leads to low efficiency, high cost, and inconsistent standards due to subjectivity.

[0003] In order to solve the above problems, with the development of technology, researchers gradually use automatic classification to replace manual work. Early automatic crack detection used machine learning methods, which often performed poorly when faced with complex image data. In recent years, crack classification technology based on deep learning has developed rapidly. A series of models based on CNN have been widely used in crack classification. However, the main research is on single-label classification of small-scale images, and there is little research on large-scale images and multi-label classification. There are also many challenges: (1) Due to the distribution of cracks in reality, there is a class imbalance in the training data, which increases the difficulty of classifying some crack categories. (2) It is difficult to label the multi-label classification data of road cracks, and there are some samples with incorrect labels, which are easy to introduce noise.

[0004] Data enhancement uses existing data to generate equivalent data, enrich data distribution, and optimize problems such as insufficient data and data imbalance. Mixup is a representative method of data enhancement. It trains neural networks through linear interpolation between images, improves the robustness and generalization ability of the model, and better handles labels. However, in the application scenario of road crack classification, there are problems with introducing the traditional Mixup method. Mixup that does not consider semantic information will increase the proportion of noise. At the same time, the method of mixing labels and multi-label classification according to the image interpolation ratio has label exclusion, which ultimately reduces the classification effect. Therefore, how to use semantic information and data enhancement to classify road cracks, alleviate data imbalance and avoid increasing noise is an urgent problem to be solved. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned deficiencies in the prior art and provides a road crack disease classification method and device based on semantic perception Mixup.

[0006] The object of the present invention is achieved through the following technical solution: a road crack disease classification method based on semantic perception Mixup, comprising the following steps:

[0007] Step 1: Use the road condition rapid detection system to sample the original image of road crack disease from the road and perform image cleaning and image enhancement.

[0008] Step 2: Annotate the images, construct a multi-label dataset of road crack diseases, and obtain training set, validation set and test set;

[0009] Step 3: Fusion label semantics and Mixup method to enhance the training set.

[0010] Step 4: Construct a deep neural network architecture, iteratively train the constructed deep neural network with the enhanced training set and validation set, and update the parameters to obtain a road crack disease classification model.

[0011] Step 5: Use the trained road crack disease classification model to test the test set data to obtain the predicted road disease category.

[0012] Furthermore, in step 1, the image cleaning process removes a large number of disease-free samples and distorted samples, and the image enhancement methods include bilateral filtering and grayscale correction.

[0013] Furthermore, the steps for constructing the multi-label dataset of road crack diseases in step 2 are as follows: first, the road images are marked with crack categories, and the marked categories are network cracks, transverse cracks, longitudinal cracks, repairs and normal; secondly, each image is preprocessed to enhance the crack features.

[0014] Furthermore, the preprocessing of each image to enhance the crack features includes: scaling and cropping the image to 224×224 pixels, and performing random horizontal flipping, grayscale and normalization; finally, dividing the road disease crack data from the total data set into a training set, a test set and a validation set, with a division ratio of 0.7, 0.2, and 0.1.

[0015] Furthermore, the fusion label semantics and Mixup method in step 3, namely semantic-aware Mixup, has the following specific process:

[0016] Firstly, the Mixup categories are divided according to the label semantics, and a batch of data Mixup is divided into two types: a mixture of normal samples and crack samples, and a mixture of crack samples and crack samples.

[0017] Secondly, different mixing strategies are adopted for the two mixtures; for normal samples (x normal ,Y normal ) and cracked samples (x crack ,Y crack ) is a mixture of The image mixing ratio is λ, and the mixing result is x mix =λ·x crack +(1-λ)·x normal , the corresponding sample label is Y mix =Y crack ; For cracked samples (x crack1 ,Y crack1 ) and cracked samples (x crack2 ,Y crack2 ), the image mixing ratio λ is sampled from beta(2.0,2.0), and the image result is x mix =λ·x crack1 +(1-λ)·x crack2 , the corresponding sample label is Y mix =Y crack1 ∪Y crack2 ;

[0018] Finally, the mixed batch data and the initial batch data are merged to form the enhanced batch training data.

[0019] Furthermore, the deep neural network settings constructed in step 4 are as follows:

[0020] The deep neural network consists of a feature extraction module and a classification module.

[0021] The feature extraction module consists of a convolutional pooling block, a convolutional block, and three dilated convolutional blocks. The convolutional pooling block consists of a 7×7 convolutional layer and a maximum pooling layer; the convolutional block consists of a 1×1 and 3×3 convolutional layer, a batch normalization layer, and an activation function layer; the dilated convolutional block is based on the convolutional block and is connected to a dilated spatial convolutional pooling pyramid layer, which consists of a dilated convolution, a pooling pyramid, and a 1×1 convolution. Residual connections are used in the convolutional layer and the dilated convolutional block.

[0022] The output of the convolution pooling block is connected to the input of the convolution block, and the number of output channels is 64; the output of the convolution block is connected to the input of the first dilated convolution block, and the number of output channels is 256; the output of the first convolution block is connected to the input of the second dilated convolution block, and the number of output channels is 512; the output of the second dilated convolution block is connected to the input of the third dilated convolution block, and the number of output channels is 1024; the output of the third dilated convolution block is connected to the input of the classification module, and the number of output channels is 2048.

[0023] The classification module consists of an average pooling layer and a linear layer. The average pooling layer obtains a 2048-dimensional vector, which is input into the linear layer. The linear layer outputs the category logit of five categories.

[0024] The second aspect of the present invention relates to a road crack disease classification device based on semantic perception Mixup, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the road crack disease classification method based on semantic perception Mixup of the present invention.

[0025] The third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the road crack disease classification method based on semantic perception Mixup of the present invention.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1. A road crack disease classification method based on semantic perception Mixup was invented, which improved the shortcomings of the existing Mixup method in multi-label classification of road cracks, improved the stability of data enhancement, alleviated data imbalance, and improved the classification accuracy of a small number of categories.

[0028] 2. The road crack disease classification method based on semantic perception Mixup constructed by the present invention is easy to implement compared with other multi-label classification methods, and can be combined with model structure improvement to improve the classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall flow chart of the method of the present invention.

[0030] Figure 2 This is a graph of prediction results of the road crack disease classification network of the present invention.

[0031] Figure 3 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0032] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0033] Example 1

[0034] like Figure 1 As shown, this embodiment relates to a road crack disease classification method based on semantic perception Mixup, comprising the following steps:

[0035] Step 1: Use the road condition rapid detection system to sample the original image of road crack disease from the road and perform image cleaning and image enhancement.

[0036] Specifically, the data used in Example 1 are collected from urban roads and highways in Zhejiang Province. The hardware used for the collection is the CICS road condition rapid detection system. After the collection is completed, a large number of disease-free samples and distorted samples are excluded through cleaning. Then, image filtering and image enhancement are used to process the remaining samples, and finally a data set of 13,807 images of three highways is obtained.

[0037] Step 2: Annotate the images, construct a multi-label dataset of road crack diseases, and obtain training set, validation set and test set;

[0038] Specifically, the images were annotated by an expert team and reviewed by the Zhejiang Provincial Transportation Science and Technology Institute to obtain the final labels. The annotated dataset was processed by scaling and cropping to obtain 224×224 pixels, and further randomly flipped horizontally, grayed and normalized. According to the number of samples, the dataset was divided into training set, validation set and test set in a ratio of 0.7, 0.1 and 0.2.

[0039] Step 3: Fusion label semantics and Mixup method to enhance the training set.

[0040] This step is the core of the present invention, integrating label semantics and Mixup method, namely semantic-aware Mixup. The specific process is as follows: First, the Mixup categories are divided according to label semantics, and a batch of data Mixup is divided into two types: a mixture of normal samples and crack samples, and a mixture of crack samples and crack samples. Second, different mixing strategies are adopted for the two mixtures; for normal samples (x normal ,Y normal ) and cracked samples (x crack ,Y crack ) is a mixture of The image mixing ratio is λ, and the mixing result is x mix =λ·x crack +(1-λ)·x normal , the corresponding sample label is Y mix =Y crack ; For cracked samples (x crack1 ,Y crack1 ) and cracked samples (x crack2 ,Y crack2 ), the image mixing ratio λ is sampled from beta(2.0,2.0), and the image result is x mix =λ·x crack1 +(1-λ)·x crack2 , the corresponding sample label is Y mix =Ycrack1 ∪Y crack2 ; Finally, the mixed batch data and the initial batch data are merged to form the enhanced batch training data.

[0041] Step 4: Construct a deep neural network architecture, iteratively train the constructed deep neural network with the enhanced training set and validation set, and update the parameters to obtain a road crack disease classification model.

[0042] Further explanation: In this example, ResNet50 is used as the basic network structure to extract image features, and the classification logic value uses a threshold of 0.5 to determine whether to take the disease. The cross entropy loss is used as the loss function, and the function is expressed as follows:

[0043] All experiments were conducted on a Linux machine with NVIDIA 3090. The Pytorch architecture was used to implement the model and train the model. The Adam optimizer was used to optimize the model, with a learning rate of 0.0001, a weight decay coefficient of 0.0001, and 150 epochs of training. Mean Average Precision was used as the main evaluation indicator.

[0044] Step 5: Use the trained road crack disease classification model to test the test set data to obtain the predicted road disease category.

[0045] like Figure 2 As shown, the crack classification effect of the present invention on four road images is shown.

[0046] In summary, the road crack disease classification method based on semantic perception Mixup provided by the present invention uses data enhancement to alleviate the data imbalance and improve the effect of multi-label classification of road crack diseases.

[0047] Example 2

[0048] like Figure 3 This embodiment relates to a road crack disease classification device based on semantic perception Mixup, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the road crack disease classification method based on semantic perception Mixup of Example 1.

[0049] Example 3

[0050] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the road crack disease classification method based on semantic perception Mixup of embodiment 1 is implemented.

[0051] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A road crack disease classification method based on semantic perception Mixup, characterized in that: The following steps are involved: Step 1: Use the road condition rapid detection system to sample the original image of road crack disease from the road and perform image cleaning and image enhancement. Step 2: Annotate the images, construct a multi-label dataset of road crack diseases, and obtain training set, validation set and test set; Step 3: Fusion label semantics and Mixup method to enhance the training set. Step 4: Construct a deep neural network architecture, iteratively train the constructed deep neural network with the enhanced training set and validation set, and update the parameters to obtain a road crack disease classification model. Step 5: Use the trained road crack disease classification model to test the test set data to obtain the predicted road disease category.

2. The road crack disease classification method based on semantic perception Mixup according to claim 1 is characterized in that: In step 1, the image cleaning process removes a large number of disease-free samples and distorted samples, and the image enhancement methods include bilateral filtering and grayscale correction.

3. The road crack disease classification method based on semantic perception Mixup according to claim 1 is characterized in that: The specific steps for constructing a multi-label dataset of road crack diseases in step 2 are as follows: first, the road images are labeled with crack categories, which are network cracks, transverse cracks, longitudinal cracks, repairs, and normal; secondly, each image is preprocessed to enhance the crack features.

4. The road crack disease classification method based on semantic perception Mixup according to claim 3 is characterized in that: The preprocessing of each image to enhance the crack features specifically includes: scaling and cropping the image to 224×224 pixels, and performing random horizontal flipping, graying and normalization; finally, dividing the road disease crack data from the total data set into a training set, a test set and a validation set, with a division ratio of 0.7, 0.2, and 0.

1.

5. The road crack disease classification method based on semantic perception Mixup according to claim 1 is characterized in that: The fusion label semantics and Mixup method in step three is semantic-aware Mixup. The specific process is as follows: First, the Mixup categories are divided according to the label semantics, and a batch of data Mixups are divided into two types: a mixture of normal samples and crack samples, and a mixture of crack samples and crack samples. Secondly, different mixing strategies are adopted for the two mixtures; for normal samples (x normal , Y normal ) and cracked samples (x crack , Y crack ) is a mixture of The image mixing ratio is λ, and the mixing result is x mix =λ·x crack +(1-λ)·x normal , the corresponding sample label is Y mix =Y crack ; For cracked samples (x crack1 , Y crack1 ) and cracked samples (x crack2 , Y crack2 ), the image mixing ratio λ is sampled from beta(2.0, 2.0), and the image result is x mix =λ·x crack1 +(1-λ)·x crack2 , the corresponding sample label is Y mix =Y crack1 ∪Y crack2 ; Finally, the mixed batch data and the initial batch data are merged to form the enhanced batch training data.

6. The road crack disease classification method based on semantic perception Mixup according to claim 1 is characterized in that: The deep neural network built in step 4 is set up as follows: The deep neural network consists of a feature extraction module and a classification module. The feature extraction module consists of a convolutional pooling block, a convolutional block, and three dilated convolutional blocks. The convolutional pooling block consists of a 7×7 convolutional layer and a maximum pooling layer; the convolutional block consists of a 1×1 and 3×3 convolutional layer, a batch normalization layer, and an activation function layer; the dilated convolutional block is based on the convolutional block and is connected to a dilated spatial convolutional pooling pyramid layer, which consists of a dilated convolution, a pooling pyramid, and a 1×1 convolution. Residual connections are used in the convolutional layer and the dilated convolutional block. The output of the convolution pooling block is connected to the input of the convolution block, and the number of output channels is 64; the output of the convolution block is connected to the input of the first dilated convolution block, and the number of output channels is 256; the output of the first convolution block is connected to the input of the second dilated convolution block, and the number of output channels is 512; the output of the second dilated convolution block is connected to the input of the third dilated convolution block, and the number of output channels is 1024; the output of the third dilated convolution block is connected to the input of the classification module, and the number of output channels is 2048. The classification module consists of an average pooling layer and a linear layer. The average pooling layer obtains a 2048-dimensional vector, which is input into the linear layer. The linear layer outputs the category logit of five categories.

7. The road crack disease classification device based on semantic perception Mixup is characterized by: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the road crack disease classification method based on semantic perception Mixup as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the road crack disease classification method based on semantic perception Mixup described in any one of claims 1 to 6 is implemented.

Images