Pavement crack segmentation method and device based on classification activation map bootstrapping

Abstract

The present invention discloses a road crack segmentation method and device based on classification activation mapping bootstrapping, which relates to the fields of artificial intelligence and machine vision. The method includes: training a normal and crack road classification model based on a deep network; generating an activation map of a road image using a class activation mapping method, filtering out a category high activation mask through a high activation threshold and performing an enhancement operation, and then adding it to a road image training set; repeating the above steps until a set condition is met; generating an activation map of a road image to be inferred based on the trained normal and crack road classification model, and obtaining a crack mask of the road image to be inferred through a crack mask threshold as a crack segmentation result. The present invention uses a classification model and activation mapping to find a category high activation mask to update a road image training set, and continuously iterates and optimizes the mask effect to improve the road crack segmentation effect, without pixel-level annotation of cracks, which greatly reduces the annotation cost.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and machine vision, and in particular to a road crack segmentation method and device based on classification activation mapping bootstrapping. Background Art

[0002] Pavement crack segmentation is a necessary means in municipal maintenance. The commonly used pavement crack segmentation method is to use pavement images with crack annotations to drive the deep learning-based segmentation model. However, crack annotation information is usually pixel-level annotation, that is, people need to annotate all pixels of each image, and the data annotation cost is huge. Therefore, how to avoid the reliance on crack segmentation masks in semantic segmentation and reduce the data annotation cost is a problem that needs to be solved. Summary of the invention

[0003] The purpose of the present invention is to provide a road surface crack segmentation method and device based on classification activation mapping bootstrapping, which uses an image classification algorithm to classify normal road surface data and crack data, obtain image data with a high activation response to the crack category, and achieve road surface crack segmentation without the need for crack space annotation information, thereby reducing data annotation costs.

[0004] The present invention adopts the following technical solution:

[0005] On the one hand, a road crack segmentation method based on classification activation map bootstrapping includes:

[0006] Normal and cracked road surface classification model training process:

[0007] A normal and cracked road surface classification model based on a deep network is trained; based on the normal and cracked road surface classification model, a class activation mapping diagram of a road surface image selected from a road surface image training set is generated using a class activation mapping method, and a normal class high activation image and a crack class high activation image are obtained after filtering out a class high activation mask through a high activation threshold and performing an enhancement operation; the normal class high activation image and the crack class high activation image are added to the normal class image set and the crack class image set of the road surface image training set, respectively; the above training process is repeated until the maximum number of iterations is reached or the normal and cracked road surface classification model converges;

[0008] Reasoning process:

[0009] The trained normal and cracked road classification model is used to generate a class activation map of the road image to be inferred, and the crack mask of the road image to be inferred is obtained through the crack mask threshold. The obtained crack mask is used as the crack segmentation result.

[0010] Preferably, a normal and cracked road surface classification model based on a deep network is trained as follows:

[0011] ;

[0012] in, is the road image data, each training iteration contains B samples, and the height and width of each sample are H and W respectively; the road image data in a training iteration are randomly selected from the road image training set, and the road image training set contains a normal class image set and a crack class image set. Before the training starts, the total number of road image training sets is fixed. After the training starts, the high activation mask of the category is filtered out by a high activation threshold and the high activation images of the normal class and the high activation images of the crack class obtained after the enhancement operation will be sent to the normal class image set and the crack class image set respectively to increase the total number of road image training sets; Net is a feature extraction network, which is a normal and crack road classification model based on a deep network, used to extract classification features in x , C is the number of channels of the classification feature; the classification loss calculation includes the fully connected layer FC, the probability mapping function Softmax and the cross entropy loss function CEL. The fully connected layer FC is used to map the number of channels of the classification feature to the same as the number of categories. The fully connected layer FC outputs a feature with a channel number of 2. The probability mapping function Softmax is used to map the classification feature to a category probability value. The cross entropy loss function CEL is used to calculate the loss of the category probability value and the image category label GT. L is the loss value of the calculated classification feature and the image category label; the loss value L is gradient descended to realize the training of the normal and cracked pavement classification model based on the deep network.

[0013] Preferably, based on the normal and cracked road surface classification model, a class activation mapping method is used to generate a class activation map of the road surface image selected from the road surface image training set, as shown below:

[0014] ;

[0015] Among them, CAM is a class activation mapping method, which is used to map the activation intensity of the road image data x in the road image training set for the specified category in the feature extraction network Net in the spatial position, that is, after a piece of image data x with a height and width of H and W respectively passes through the class activation mapping method CAM, two class activation mapping images with a height and width of H and W respectively will be obtained; is a class activation map. Among the B pieces of data, the 0th class activation map with a height and width of H and W respectively belongs to the normal class, and the 1st class activation map with a height and width of H and W respectively belongs to the crack class.

[0016] Preferably, after filtering out the category high activation mask through the high activation threshold and performing the enhancement operation, a normal class high activation image and a crack class high activation image are obtained, as shown below:

[0017] ;

[0018] in, and They are the class activation map values ​​of class activation map g in the kth channel and 1-k channels, with height number i and width number j; δ is the indicator function used for threshold screening. When the condition is met 1 when satisfied, 0 when not satisfied; t m is the high activation threshold, and its value range is t m >2; is the high activation mask of the category after threshold filtering, is the mask value of the channel number k, height number i, and width number j in the category high activation mask m. When k is 0, it belongs to the normal class, and when k is 1, it belongs to the crack class. is the value of the road surface image x in the road surface image training set in the cth channel, with the height number i and the width number j; the normal / crack category high activation mask is multiplied with the road surface image to achieve the cropping of the normal / crack high activation images in the road surface image; is the high activation image of the kth class, is a high activation image of the normal class, is a highly activated image of the crack class, is the image value of the k-th high activation image with channel number c, height number i, and width number j.

[0019] Preferably, during the inference process, the trained normal and cracked road surface classification model is used to generate a class activation map of the road surface image to be inferred, the crack mask of the road surface image to be inferred is obtained by the crack mask threshold, and the obtained crack mask is used as the crack segmentation result, as shown below:

[0020] ;

[0021] in, and They are the class activation maps of the road images to be inferred. In the 0th channel and the 1st channel, the class activation map value with height number i and width number j; δ is the indicator function used for threshold screening, which is 1 when the condition is met and 0 when it is not met; t s is the crack mask threshold, ranging from t m >t s >1,t m is the high activation threshold; is the crack mask after threshold filtering, is the mask value of the spatial position with height number i and width number j in the crack mask s. The value of 1 indicates the crack class, and the value of 0 indicates the normal class. When it is 1, it means that the crack mask s belongs to the crack class at the spatial position with height number i and width number j. When it is 0, it means that the crack mask s at the spatial position with height index i and width index j belongs to the normal class; the crack mask is output as the crack segmentation result.

[0022] On the other hand, a road crack segmentation device based on classification activation map bootstrapping includes:

[0023] Normal and cracked pavement classification model training module, specifically used for:

[0024] Training a normal and cracked road surface classification model based on a deep network; based on the normal and cracked road surface classification model, using a class activation mapping method to generate a class activation map of a road surface image selected from a road surface image training set, filtering out a class high activation mask through a high activation threshold and performing an enhancement operation, thereby obtaining a normal class high activation image and a crack class high activation image; adding the normal class high activation image and the crack class high activation image to the normal class image set and the crack class image set of the road surface image training set, respectively; repeating the normal and cracked road surface classification model training module until the maximum number of iterations is reached or the normal and cracked road surface classification model converges;

[0025] The reasoning module is specifically used for:

[0026] The trained normal and cracked road classification model is used to generate a class activation map of the road image to be inferred, and the crack mask of the road image to be inferred is obtained through the crack mask threshold. The obtained crack mask is used as the crack segmentation result.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention utilizes the obvious difference between crack images and normal images, and realizes the segmentation of road cracks through classification method and class activation mapping method, and feeds back the high-confidence segmentation results to the data set, avoiding the dependence on the segmentation mask annotation information of the road cracks, thereby reducing the data annotation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the road crack segmentation method based on classification activation map bootstrapping of the present invention;

[0030] Figure 2 A schematic diagram of the normal and cracked pavement classification model training process of the present invention;

[0031] Figure 3 It is a structural block diagram of the road crack segmentation device based on classification activation map bootstrapping of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0033] See also Figure 1 As shown, the present invention provides a road surface crack segmentation method based on classification activation mapping bootstrapping, including: a normal and cracked road surface classification model training process S1 and an inference process S2.

[0034] For details, see Figure 2 As shown, the normal and cracked pavement classification model training process S1 is implemented as follows.

[0035] S11, obtaining a road surface image training set as an image classification training set, where the image classification training set includes a normal class image set and a crack class image set.

[0036] S12, select the ResNet-50 model as the feature extraction network to build a normal and cracked road surface classification model based on a deep network, select the last convolutional layer of the fourth stage of ResNet-50 as the target layer of the class activation mapping method; select GradCAM as the class activation mapping method.

[0037] S13, data preprocessing, uniformly adjusts the size of the road images in the road image training set to H×W, where H and W are the height and width of the image respectively, and then applies random flipping as a data enhancement method.

[0038] S14, using a class activation mapping method to train a normal and cracked road surface classification model, specifically including the following steps.

[0039] S141, training a normal and cracked road surface classification model based on a deep network, as shown in formula (1):

[0040] ; (1)

[0041] in, is the road image data in the road image training set. Each training iteration contains B samples, and the height and width of each sample are H and W respectively. The road image data in a training iteration are randomly selected from the road image training set. The road image training set contains a normal class image set and a crack class image set. Before the training starts, the total number of road image training sets is fixed. After the training starts, the normal class high activation images and crack class high activation images obtained by the subsequent image cropping steps will be sent to the normal class image set and the crack class image set respectively, thereby gradually increasing the total number of road image training sets. Net is a feature extraction network, which is a normal and crack road classification model based on a deep network, and is used to extract classification features in the road image x. , C is the number of channels of the classification feature; the classification loss calculation includes a fully connected layer FC, a probability mapping function Softmax and a cross entropy loss function CEL, the fully connected layer FC is used to map the number of channels of the classification feature to be the same as the number of categories, since the present invention is for the classification of cracks and normal roads, the fully connected layer FC finally outputs a feature with a channel number of 2, the probability mapping function Softmax is used to map the classification feature to a category probability value, the cross entropy loss function CEL is used to calculate the loss of the category probability value and the image category label GT, L is the calculated loss value of the classification feature and the image category label; the loss value L is gradient descended to realize the training of the normal and crack road classification model based on the deep network.

[0042] S142, based on the normal and cracked road surface classification model, a class activation mapping method is used to generate a class activation map of the road surface image selected from the road surface image training set, as shown in formula (2):

[0043] ; (2)

[0044] Among them, CAM is a class activation mapping method, which is used to map the activation intensity of the road image data x for the specified category in the feature extraction network Net to the spatial position, that is, after a piece of image data x with a height and width of H and W respectively passes through the class activation mapping method CAM, two class activation mapping images with a height and width of H and W respectively will be obtained; is the class activation map. In the B data, the 0th and 1st class activation maps with height and width H and W respectively belong to the normal class and the crack class respectively.

[0045] S143, filter out the high activation mask of the category through the high activation threshold and perform enhancement operations such as cropping to obtain the normal class high activation image and the crack class high activation image; add the normal class high activation image and the crack class high activation image to the normal class image set and the crack class image set of the road image training set respectively. Repeat the above process until the maximum number of iterations is reached or the model converges, as shown in formula (3):

[0046] ; (3)

[0047] in, and are the class activation map values ​​of class activation map g in the kth channel and 1-kth channel, with height number i and width number j; δ is the indicator function used for threshold screening, which is 1 when the condition is met and 0 when it is not met; t m is the high activation threshold, which is t m =3; is the high activation mask of the category after threshold filtering, is the mask value of the channel number k, height number i, and width number j in the category high activation mask m. When k is 0, it belongs to the normal class, and when k is 1, it belongs to the crack class. is the value of the road surface image x in the cth channel with the height number i and the width number j; the normal / crack category high activation mask is multiplied with the road surface image to achieve the clipping of the normal / crack high activation image in the road surface image, that is, the original value of the high activation image data is retained, and the low activation image data is set to 0; is the high activation image of the kth class, where is a high activation image of the normal class, is a highly activated image of the crack class, is the image value of the k-th class high activation image with channel number c, height number i, and width number j; and They are sent to the normal image set and crack image set of the road image training set respectively to enhance the understanding of normal class data and crack class data during training.

[0048] Specifically, the reasoning process S2 is implemented as follows.

[0049] During the inference process, the trained normal and crack road classification model is used to generate the activation map of the road image to be inferred, and the crack mask is obtained by threshold selection as the crack segmentation result, as shown in formula (4):

[0050] ; (4)

[0051] in, and They are the class activation maps of the road images to be inferred. In the 0th channel and the 1st channel, the class activation map value with height number i and width number j; δ is the indicator function used for threshold screening, which is 1 when the condition is met and 0 when it is not met; t s is the crack mask threshold, and its value is t s =1.5; is the crack mask after threshold filtering, is the mask value of the spatial position with height number i and width number j in the crack mask s. The value of 1 indicates the crack class, and the value of 0 indicates the normal class. When it is 1, it means that the crack mask s belongs to the crack class at the spatial position with height number i and width number j. When it is 0, it means that the crack mask s at the spatial position with height index i and width index j belongs to the normal class; the crack mask is output as the crack segmentation result. Since the whole process only requires the category labels of crack class images and normal class images, it avoids the dependence on the crack segmentation mask in semantic segmentation and reduces the data annotation cost.

[0052] In this implementation case, a road crack segmentation method based on classification activation mapping bootstrapping is proposed, and its reasoning stage is as follows: the crack classification model obtained in the training stage and the road surface image to be inferred are obtained, and both are sent to the class activation mapping method to obtain a class activation map, and a threshold is applied to the class activation map to obtain a crack mask, and the crack mask is used as the crack segmentation result.

[0053] The key of the present invention is to use the image classification algorithm to classify and learn the normal road surface data and crack data, obtain image data with high activation response to the crack category, and segment the road surface cracks without the need for crack space annotation information, thereby reducing the data annotation cost. Therefore, the present invention can be widely used in crack segmentation applications for municipal maintenance.

[0054] See also Figure 3 As shown, the present invention also discloses a road crack segmentation device based on classification activation mapping bootstrapping, comprising:

[0055] The normal and cracked pavement classification model training module 301 is specifically used for:

[0056] Training a normal and cracked road surface classification model based on a deep network; based on the normal and cracked road surface classification model, using a class activation mapping method to generate a class activation map of a road surface image selected from a road surface image training set, filtering out a class high activation mask through a high activation threshold and performing an enhancement operation, thereby obtaining a normal class high activation image and a crack class high activation image; adding the normal class high activation image and the crack class high activation image to the normal class image set and the crack class image set of the road surface image training set, respectively; repeating the normal and cracked road surface classification model training module until the maximum number of iterations is reached or the normal and cracked road surface classification model converges;

[0057] The reasoning module 302 is specifically used for:

[0058] The trained normal and cracked road classification model is used to generate a class activation map of the road image to be inferred, and the crack mask of the road image to be inferred is obtained through the crack mask threshold. The obtained crack mask is used as the crack segmentation result.

[0059] The specific implementation of each module of a road crack segmentation device based on classification activation mapping bootstrapping is the same as the road crack segmentation method based on classification activation mapping bootstrapping, and the present embodiment will not repeat the description.

[0060] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described in this application are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A road crack segmentation method based on classification activation map bootstrapping, characterized in that: include: Normal and cracked road surface classification model training process: Train a deep network-based model for normal and cracked road surface classification; Based on the normal and cracked road classification model, the class activation mapping method is used to generate the class activation map of the road image selected from the road image training set. The high activation mask of the category is filtered out by the high activation threshold and enhanced to obtain the normal class high activation image and the crack class high activation image. The normal class high activation images and the crack class high activation images are added to the normal class image set and the crack class image set of the road image training set respectively; the above training process is repeated until the maximum number of iterations is reached or the normal and crack road classification models converge; Reasoning process: The trained normal and cracked road surface classification model is used to generate a class activation map of the road surface image to be inferred, the crack mask of the road surface image to be inferred is obtained through the crack mask threshold, and the obtained crack mask is used as the crack segmentation result; After filtering out the category high activation mask through the high activation threshold and performing the enhancement operation, the normal class high activation image and the crack class high activation image are obtained, as shown below: Among them, g k,i,j With g 1-k,i,j They are the class activation map values ​​of class activation map g in the kth channel and 1-k channels, with height number i and width number j; δ is the indicator function used for threshold screening. When the condition is met 1 when satisfied, 0 when not satisfied; t m is the high activation threshold, and its value range is t m >2; is the high activation mask of the category after threshold filtering, B is the number of data copies, the height and width of each data copy are H and W respectively, m k,i,j is the mask value of the channel number k, height number i, and width number j in the category high activation mask m. When k is 0, it belongs to the normal class, and when k is 1, it belongs to the crack class; x c,i,j is the value of the road surface image x in the road surface image training set in the cth channel, with the height number i and the width number j; the normal / crack category high activation mask is multiplied with the road surface image to achieve the cropping of the normal / crack high activation image in the road surface image; img k is the high activation image of the kth class, img 0 For normal class high activation images, img 1 is a highly activated image of the crack class, is the image value of the k-th class high activation image with channel number c, height number i, and width number j; During the inference process, the trained normal and cracked road classification model is used to generate the class activation map of the road image to be inferred, and the crack mask of the road image to be inferred is obtained through the crack mask threshold. The obtained crack mask is used as the crack segmentation result, as shown below: in, and are the class activation maps g of the road images to be inferred b In the 0th channel and the 1st channel, the class activation map value with height number i and width number j; δ is the indicator function used for threshold screening, which is 1 when the condition is met and 0 when it is not met; t s is the crack mask threshold, ranging from t m >t s >1,t m is the high activation threshold; is the crack mask after threshold filtering, s i,j is the mask value of the spatial position with height number i and width number j in the crack mask s. The value 1 indicates the crack class and the value 0 indicates the normal class. i,j When it is 1, it means that the crack mask s at the spatial position with height number i and width number j belongs to the crack class, s i,j When it is 0, it means that the crack mask s at the spatial position with height index i and width index j belongs to the normal class; the crack mask is output as the crack segmentation result.

2. The road crack segmentation method based on classification activation map bootstrapping according to claim 1 is characterized in that: Train a deep network-based normal and cracked road classification model as follows: in, is the road image data, each training iteration contains B samples, and the height and width of each sample are H and W respectively; the road image data in a training iteration are randomly selected from the road image training set, and the road image training set contains a normal class image set and a crack class image set. Before the training starts, the total number of road image training sets is fixed. After the training starts, the high activation mask of the category is filtered out by a high activation threshold and the high activation images of the normal class and the high activation images of the crack class obtained after the enhancement operation will be sent to the normal class image set and the crack class image set respectively to increase the total number of road image training sets; Net is a feature extraction network, which is a normal and crack road classification model based on a deep network, used to extract classification features in x C is the number of channels of the classification feature; the classification loss calculation includes the fully connected layer FC, the probability mapping function Softmax and the cross entropy loss function CEL. The fully connected layer FC is used to map the number of channels of the classification feature to the same as the number of categories. The output of the fully connected layer FC is a feature with a channel number of 2. The probability mapping function Softmax is used to map the classification feature to the category probability value. The cross entropy loss function CEL is used to calculate the loss of the category probability value and the image category label GT. L is the calculated loss value of the classification feature and the image category label; the loss value L is gradient descended to realize the training of the normal and cracked pavement classification model based on the deep network.

3. The road crack segmentation method based on classification activation map bootstrapping according to claim 1 is characterized in that: Based on the normal and cracked road surface classification model, the class activation mapping method is used to generate the class activation map of the road surface image selected from the road surface image training set, as shown below: g = CAM (Net, x); Among them, CAM is a class activation mapping method, which is used to map the activation intensity of the road image data x in the road image training set for the specified category in the feature extraction network Net in the spatial position, that is, after a piece of image data x with a height and width of H and W respectively passes through the class activation mapping method CAM, two class activation mapping images with a height and width of H and W respectively will be obtained; is a class activation map. Among the B pieces of data, the 0th class activation map with a height and width of H and W respectively belongs to the normal class, and the 1st class activation map with a height and width of H and W respectively belongs to the crack class.

4. A road crack segmentation device based on classification activation map bootstrapping, characterized in that: include: Normal and cracked pavement classification model training module, specifically used for: A normal and cracked road surface classification model based on a deep network is trained; based on the normal and cracked road surface classification model, a class activation mapping method is used to generate a class activation map of a road surface image selected from a road surface image training set, and a class high activation mask is screened out by a high activation threshold and enhanced to obtain a normal class high activation image and a crack class high activation image; The normal class high activation images and the crack class high activation images are added to the normal class image set and the crack class image set of the road surface image training set respectively; the normal and crack road surface classification model training module is repeated until the maximum number of iterations is reached or the normal and crack road surface classification model converges; The reasoning module is specifically used for: The trained normal and cracked road surface classification model is used to generate a class activation map of the road surface image to be inferred, the crack mask of the road surface image to be inferred is obtained through the crack mask threshold, and the obtained crack mask is used as the crack segmentation result; After filtering out the category high activation mask through the high activation threshold and performing the enhancement operation, the normal class high activation image and the crack class high activation image are obtained, as shown below: Among them, g k,i,j With g 1-k,i,j They are the class activation map values ​​of class activation map g in the kth channel and 1-k channels, with height number i and width number j; δ is the indicator function used for threshold screening. When the condition is met 1 when satisfied, 0 when not satisfied; t m is the high activation threshold, and its value range is t m >2; is the high activation mask of the category after threshold filtering, B is the number of data copies, the height and width of each data copy are H and W respectively, m k,i,j is the mask value of the channel number k, height number i, and width number j in the category high activation mask m. When k is 0, it belongs to the normal class, and when k is 1, it belongs to the crack class; x c,i,j is the value of the road surface image x in the road surface image training set in the cth channel, with the height number i and the width number j; the normal / crack category high activation mask is multiplied with the road surface image to achieve the cropping of the normal / crack high activation image in the road surface image; img k is the high activation image of the kth class, img 0 For normal class high activation images, img 1 is a highly activated image of the crack class, is the image value of the k-th class high activation image with channel number c, height number i, and width number j; During the inference process, the trained normal and cracked road classification model is used to generate the class activation map of the road image to be inferred, and the crack mask of the road image to be inferred is obtained through the crack mask threshold. The obtained crack mask is used as the crack segmentation result, as shown below: in, and are the class activation maps g of the road images to be inferred b In the 0th channel and the 1st channel, the class activation map value with height number i and width number j; δ is the indicator function used for threshold screening, which is 1 when the condition is met and 0 when it is not met; t s is the crack mask threshold, ranging from t m >t s >1,t m is the high activation threshold; is the crack mask after threshold filtering, s i,j is the mask value of the spatial position with height number i and width number j in the crack mask s. The value 1 indicates the crack class and the value 0 indicates the normal class. i,j When it is 1, it means that the crack mask s at the spatial position with height number i and width number j belongs to the crack class, s i,j When it is 0, it means that the crack mask s at the spatial position with height index i and width index j belongs to the normal class; the crack mask is output as the crack segmentation result.

Images