A method and system for identifying cracks in highway asphalt pavement

Abstract

The present invention discloses a method and system for identifying cracks in a highway asphalt pavement, comprising: obtaining a highway asphalt pavement image; obtaining a highway asphalt pavement crack identification result according to the highway asphalt pavement image and a trained pavement crack identification model, wherein the pavement crack identification model comprises a backbone feature network and a feature fusion network, the backbone feature network comprises a multi-layer feature extraction layer, the multi-layer feature extraction layer continuously extracts multi-layer features from the highway asphalt pavement image, and the features extracted from each layer are input into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the features of the two adjacent layers to obtain multiple fused features, fuses all fused features to obtain enhanced features, and obtains a highway asphalt pavement crack identification result according to the enhanced features. The accuracy of identifying cracks in a highway asphalt pavement is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and related technologies, and in particular to a method and system for identifying cracks in a highway asphalt pavement. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The quality of highways is becoming increasingly important for traffic safety and efficiency. However, the influence of traffic load, design and construction quality, and environmental factors have led to pavement cracks becoming one of the main forms of pavement damage. The appearance of cracks accelerates the destruction of the pavement, seriously affecting the bearing capacity, driving comfort and safety of the road, and also causing increasingly serious pavement maintenance and management problems. Pavement crack detection is a prerequisite for road maintenance decisions. Timely detection, evaluation and treatment of pavement cracks can not only extend the service life of the road, but also improve driving comfort and reduce annual road maintenance costs. Therefore, it is crucial to carry out high-quality pavement crack detection.

[0004] Traditional pavement crack detection methods mainly rely on manual inspection, that is, workers visually inspect the road surface and mark cracks. However, this method has many limitations, including low efficiency, low detection accuracy, and being affected by factors such as weather conditions. In addition, in the process of collecting pavement parameter information, there are problems such as poor versatility, inaccurate accuracy, and low standardization. To solve the above problems, scholars began to explore the application of automation technology in pavement crack recognition. Early pavement crack recognition algorithms mainly used image processing algorithms. Although these algorithms can realize the automatic recognition of pavement cracks, in practical applications, their recognition results are greatly affected by manually set parameters, and different pavement crack data require different processing methods. With the deepening of research, scholars began to apply machine learning to pavement crack recognition. These methods have a higher degree of automation, but their processes are relatively cumbersome, and their accuracy and generalization ability are poor. In recent years, with the development of deep learning, it has provided a new research direction for the recognition of cracks in asphalt pavements on highways. The detection of pavement cracks using deep learning can generally be divided into two ways. One way is to perform target detection through bounding boxes to obtain the location of cracks in the original image. However, this method can only frame the location of the cracks, but cannot provide detailed information such as crack length and area ratio required for maintenance work. Another method is to perform semantic segmentation on the pavement image to obtain a binary image of the pavement cracks. By post-processing the binary image, fine parameters such as the pavement crack length and area ratio can be obtained. In order to provide a scientific decision-making basis for road maintenance, the highway asphalt pavement crack segmentation task has high requirements on the precision and integrity of the segmentation results. At the same time, the annotation cost of the highway asphalt pavement crack semantic segmentation is high, the quality of the pavement image varies greatly, and the contrast between positive and negative samples is different, which greatly affects the segmentation effect of the model. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a highway asphalt pavement crack recognition method and system, which is suitable for the highway asphalt pavement crack segmentation task of different scales and different positive and negative sample contrasts, and ensures the accuracy of highway asphalt pavement crack segmentation.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] In the first aspect, a method for identifying cracks in a highway asphalt pavement is proposed, comprising:

[0008] Acquire highway asphalt pavement images;

[0009] According to the highway asphalt pavement image and the trained pavement crack recognition model, the highway asphalt pavement crack recognition result is obtained, wherein the pavement crack recognition model includes a backbone feature network and a feature fusion network, the backbone feature network includes a multi-layer feature extraction layer, the multi-layer feature extraction layer performs continuous multi-layer feature extraction on the highway asphalt pavement image, and inputs the features extracted from each layer into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, all the fused features are fused to obtain enhanced features, and the highway asphalt pavement crack recognition result is obtained according to the enhanced features.

[0010] In the second aspect, a highway asphalt pavement crack identification system is proposed, comprising:

[0011] An image acquisition module, used for acquiring an image of a highway asphalt pavement;

[0012] The image recognition module is used to obtain the highway asphalt pavement crack recognition result based on the highway asphalt pavement image and the trained pavement crack recognition model, wherein the pavement crack recognition model includes a backbone feature network and a feature fusion network, the backbone feature network includes a multi-layer feature extraction layer, the multi-layer feature extraction layer performs continuous multi-layer feature extraction on the highway asphalt pavement image, and inputs the features extracted from each layer into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, all fused features are fused to obtain enhanced features, and the highway asphalt pavement crack recognition result is obtained based on the enhanced features.

[0013] In a third aspect, an electronic device is proposed, comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps described in a method for identifying cracks in a highway asphalt pavement are completed.

[0014] In a fourth aspect, a computer-readable storage medium is proposed for storing computer instructions. When the computer instructions are executed by a processor, the steps of a method for identifying cracks in a highway asphalt pavement are completed.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. The present invention uses a backbone feature network to extract multi-layer features of road surface images, fuses multi-layer features through a feature fusion network, and then fuses the fused features again to obtain enhanced features, so that more image information is retained in the enhanced features. When the enhanced features are used to identify cracks in highway asphalt pavement, the accuracy of the highway asphalt pavement crack identification results is guaranteed.

[0017] 2. When the backbone feature network of the present invention extracts image features through the first feature extraction module, the second feature extraction module and upsampling, it can capture broader contextual information while retaining low-dimensional feature information, thereby improving the model's ability to express multi-scale features; an EMA attention mechanism is introduced into the backbone feature network to evenly distribute feature information in feature groups, so that the model can accurately capture the spatial features of the image and improve the ability to model the relationship between features; thereby improving the accuracy of highway asphalt pavement crack recognition results.

[0018] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0020] Figure 1 A flow chart of the method disclosed in the embodiment;

[0021] Figure 2 A structural diagram of a pavement crack identification model disclosed in an embodiment;

[0022] Figure 3 A diagram of the backbone feature network structure disclosed in the embodiment;

[0023] Figure 4 The network structure diagram of MobileOne Block disclosed in the embodiment;

[0024] Figure 5 The network structure diagram of GAMC Block disclosed in the embodiment;

[0025] Figure 6 Schematic diagram of the EMA attention mechanism disclosed in the embodiment;

[0026] Figure 7 It is a schematic diagram of the Bottleneck structure disclosed in the embodiment;

[0027] Figure 8 It is a schematic diagram of the AIFI structure disclosed in the embodiment;

[0028] Fig. 9 It is a schematic diagram of a dense upsampling structure disclosed in an embodiment. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] Example 1

[0033] In this embodiment, a method for identifying cracks in a highway asphalt pavement is disclosed. Figure 1-Figure 9 As shown, including:

[0034] Acquire highway asphalt pavement images;

[0035] According to the highway asphalt pavement image and the trained pavement crack recognition model, the highway asphalt pavement crack recognition result is obtained, wherein the pavement crack recognition model includes a backbone feature network and a feature fusion network, the backbone feature network includes a multi-layer feature extraction layer, the multi-layer feature extraction layer performs continuous multi-layer feature extraction on the highway asphalt pavement image, and inputs the features extracted from each layer into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, all the fused features are fused to obtain enhanced features, and the highway asphalt pavement crack recognition result is obtained according to the enhanced features.

[0036] The pavement crack recognition model constructed in this embodiment takes the highway asphalt pavement image as input and the highway asphalt pavement crack recognition result as output, including a backbone feature network and a feature fusion network, wherein the backbone feature network includes a multi-layer feature extraction layer, through which the highway asphalt pavement image is continuously subjected to multi-layer feature extraction, and the features extracted from each layer are output, and the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, and fuses all fused features to obtain enhanced features, and obtains the highway asphalt pavement crack recognition result based on the enhanced features. The overall framework of the pavement crack recognition model is shown in the figure. Figure 2 As shown, the calculation process includes:

[0037] F1=EMA(M(M(X)))

[0038] F2=G(G(M(SPD(F1))))

[0039] F3=G(G(G(G(M(SPD(F2))))))

[0040] F4=G(G(G(M(SPD(F3)))))

[0041] F5=M(SPD(F4))

[0042] O5=AIFI(F5)

[0043] O4=ASFF(O5,F4,F3)

[0044] O3=ASFF(F4,F3,F2)

[0045] O2=ASFF(F3,F2,F1)

[0046] O1=ASFF(O4,O3,O2)

[0047] Y=C k＝1 (O1)

[0048] Where: X is the highway asphalt pavement image, F i is the feature extracted from the i-th feature extraction layer of the backbone feature network. The backbone feature network is represented by the Backbone network, i=1, 2, 3, 4 or 5, M is the feature extraction operation of the first feature extraction module, the first feature extraction module adopts MobileOne Block, EMA is the EMA attention mechanism, G is the feature extraction operation of the second feature extraction module, the second feature extraction module adopts GAMC Block, SPD is the SPDConv (downsampling) operation, AIFI is the position encoding module, the position encoding module adopts the AIFI module, O i is the i-th fusion feature, i = 1, 2, 3 or 4, O5 is the enhanced feature, ASFF is the feature fusion operation, C is the convolution operation, k is the convolution kernel size, and Y is the highway asphalt pavement crack recognition result.

[0049] like Figure 3 As shown in the figure, the backbone feature network uses the first feature extraction module and the second feature extraction module to extract feature information. SPDConv is used to downsample feature information. Before each downsampling, the feature map of the previous level is output. At the same time, the EMA attention mechanism is introduced to improve the network's ability to extract cross-space features. The calculation process of the backbone feature network is as follows:

[0050] F1=EMA(M(M(X)))

[0051] F2=G(G(M(SPD(F1))))

[0052] F3=G(G(G(G(M(SPD(F2))))))

[0053] F4=G(G(G(M(SPD(F3)))))

[0054] F5=M(SPD(F4))

[0055] Wherein: X is the asphalt pavement image of the highway, Fi is the feature extracted by the i-th feature extraction layer of the backbone feature network, i = 1, 2, 3, 4 or 5, M is the feature extraction operation of the first feature extraction module, the first feature extraction module adopts MobileOneBlock, EMA is the EMA attention mechanism, G is the feature extraction operation of the second feature extraction module, the second feature extraction module adopts GAMC Block, and SPD is the SPDConv (downsampling) operation.

[0056] The backbone feature network includes multiple layers of feature extraction layers. The first feature extraction layer performs two consecutive feature extractions on the highway asphalt pavement image through two first feature extraction modules to obtain image features, and performs attention feature extraction on the image features to obtain the first layer of features extracted by the first feature extraction layer. Except for the first feature extraction layer, each of the remaining feature extraction layers takes the features extracted by the previous feature as input. The last feature extraction layer downsamples the input features, and then extracts features from the downsampled features through a first feature extraction module to obtain the features extracted by the last feature extraction layer. Except for the first feature extraction layer and the last feature extraction layer, the remaining feature extraction layers downsample the input features, and then perform feature extraction on the downsampled features through the first feature extraction module. The extracted features are then subjected to feature extraction through multiple consecutive second feature extraction modules to obtain the features extracted by the corresponding feature extraction layer.

[0057] Taking the backbone feature network including five layers of feature extraction layers as an example, the structure of the backbone feature network is explained. The backbone feature network includes five layers of feature extraction layers, among which the first feature extraction layer includes two first feature extraction modules and an EMA attention mechanism. After the first first feature extraction module extracts features from the highway asphalt pavement image, the extracted features are input into the second first feature extraction module for feature extraction. The second first feature extraction module extracts the input features, and the obtained image features are extracted through the EMA attention mechanism to obtain the features extracted by the first feature extraction layer; the features extracted by the first feature extraction layer are input into the second feature extraction layer, and the second feature extraction layer includes a downsampling operation, a first A feature extraction module and two second feature extraction modules, after the second feature extraction layer performs a downsampling operation on the input features, the features are input into the first feature extraction module for feature extraction, the features extracted by the first feature extraction module are input into one of the second feature extraction modules, the second feature extraction module performs feature extraction on the input features, and the extracted features are input into another second feature extraction module, the second feature extraction module performs feature extraction on the input features, and the extracted features are used as the features extracted by the second feature extraction layer; the features extracted by the second feature extraction layer are input into the third feature extraction layer, the third feature extraction layer includes a downsampling operation, a first feature extraction module and four second feature extraction modules, the third feature extraction layer performs ... After the input features are downsampled, they are input into the first feature extraction module for feature extraction. The features extracted by the first feature extraction module are input into the first second feature extraction module. The second feature extraction module extracts features from the input features and inputs the extracted features into the second second feature extraction module. The second feature extraction module extracts features from the input features and inputs the extracted features into the third second feature extraction module. The second feature extraction module extracts features from the input features and inputs the extracted features into the fourth second feature extraction module. The fourth second feature extraction module extracts features from the input features and extracts features as the features extracted by the third feature extraction layer. The features extracted by the third feature extraction layer The fourth feature extraction layer is input, and the fourth feature extraction layer includes a downsampling operation, a first feature extraction module and three second feature extraction modules. After the third feature extraction layer performs a downsampling operation on the input features, the first feature extraction module is input for feature extraction. The features extracted by the first feature extraction module are input into the first second feature extraction module. The second feature extraction module extracts features from the input features and inputs the extracted features into the second second feature extraction module. The second feature extraction module extracts features from the input features and inputs the extracted features into the third second feature extraction module. The second feature extraction module extracts features from the input features and extracts features as the features extracted by the fourth feature extraction layer.The features extracted by the fourth feature extraction layer are input into the fifth feature extraction layer. The fifth feature extraction layer includes a downsampling operation and a first feature extraction module. After the third feature extraction layer performs a downsampling operation on the input features, the features are input into the first feature extraction module for feature extraction. The features extracted by the first feature extraction module are used as the features extracted by the fifth feature extraction layer. The features extracted by the first feature extraction layer, the features extracted by the second feature extraction layer, the features extracted by the third feature extraction layer, the features extracted by the fourth feature extraction layer, and the features extracted by the fifth feature extraction layer are all used as the output of the backbone feature network and input into the feature fusion network. ;

[0058] like Figure 4 As shown, the first feature extraction module includes a depth convolution part and a point-by-point convolution part. The depth convolution part includes three branches, namely, a first convolution layer, a second convolution layer, and a batch normalization layer. The first convolution layer, the second convolution layer, and the batch normalization layer respectively extract features from the features input into the first feature extraction module to obtain the first feature, the second feature, and the third feature. After fusing the first feature, the second feature, and the third feature, the features are input into the point-by-point convolution part. The point-by-point convolution part includes two branches, namely, a point-by-point convolution layer and a batch normalization layer. The point-by-point convolution layer and the batch normalization layer respectively extract the fourth feature and the fifth feature from the features input into the point-by-point convolution part, and fuse the fourth feature and the fifth feature as the output of the first feature extraction module.

[0059] Preferably, the first convolution layer is a depthwise convolution with a convolution kernel size of 1, and the second convolution layer is a depthwise convolution with a convolution kernel size of 3.

[0060] The point-by-point convolution layer is a point-by-point convolution with a convolution kernel size of 1.

[0061] In the inference state, the first feature extraction module (MobileOne Block) uses the re-parameterization technology to fuse the deep convolution part into a deep convolution with a convolution kernel size of 3, and the point-by-point convolution part into a point-by-point convolution with a convolution kernel size of 1. Taking the first feature extraction layer as an example, the calculation process is as follows:

[0062]

[0063]

[0064] Where: X is the highway asphalt pavement image, C d is the depth convolution, C p is the point-by-point convolution, k is the convolution kernel size, BN is the batch normalization operation, M1 is the result of the depth convolution part, and M2 is the result of the point-by-point convolution part.

[0065] like Figure 6As shown in the figure, the EMA attention mechanism is used to extract the attention features of the features. The process is as follows: the EMA attention mechanism divides the input features into different feature groups, and performs convolution, average pooling and convolution operations in the X and Y directions on each group of features. The average pooling results in the X and Y directions are stacked on the channel dimension, and the weight of the feature group is obtained through the Sigmoid operation, which is multiplied with the features of the corresponding group. The multiplication result and the convolution result of the features of the corresponding group are respectively average pooled and Softmax operations are performed to obtain the first convolution feature and the second convolution feature. The first convolution feature and the second convolution feature are cross-multiplied and then added. Then, the final feature weight is obtained through the Sigmoid operation, and the final feature weight is applied to the feature group as the output of the EMA attention mechanism.

[0066] The EMA attention mechanism converts some channels into batch dimensions so that feature information can be evenly distributed in feature groups. First, the input features are divided into different feature groups, and average pooling and convolution operations are performed in the X and Y directions respectively. Subsequently, the average pooling results in the X and Y directions are stacked on the channel dimension, and the weight of the feature group is obtained through the Sigmoid operation and multiplied with the feature group. Next, the feature group is convolved, and the multiplication result and the convolution result of the feature group are average pooled and Softmax operations are performed respectively to obtain two results. The two results are cross-multiplied and then added, and finally the final feature weight is obtained through the Sigmoid operation. The final feature weight is applied to the feature group to obtain the output feature. Taking the first feature extraction layer as an example, the calculation process of the EMA attention mechanism is as follows:

[0067] E1=G(M(M2))

[0068] E2=C k＝1 (Cat(X A (E1),Y A (E1)

[0069] E3=GN(Sg(E2)*E1)

[0070] E4=C k＝3 (E1)

[0071] E5=Sm(Avg(E3))*E4

[0072] E6=Sm(Avg(E4))*E3

[0073] F1=Sg(E5+E6)*E1

[0074] Where: M2 is the feature extracted from the highway asphalt pavement image by the first feature extraction module, M is the MobileOne Block operation, G is the grouping operation, and E is the iis the value in the process, X A ,Y A They are the average pooling operations in the X and Y directions, Cat is the channel stacking operation, C is the convolution operation, k is the convolution kernel size, Sg is the Sigmoid normalization, GN is the group normalization, Sm is the Softmax operation, Avg is the average pooling operation, and F1 is the feature extracted by the first feature extraction layer.

[0075] The downsampling operation (SPDConv) consists of two parts: the SPD layer (downsampling layer) and the non-stride layer. The SPD layer is responsible for downsampling the features of the input downsampling operation while retaining its low-dimensional information, thereby avoiding the loss of detail information caused by traditional downsampling; the non-stride layer is composed of a convolution layer with a convolution size of 1×1, which is responsible for adjusting the number of channels of the features after the SPD layer is downsampled, and using learnable parameters to process the image features.

[0076] like Figure 5 As shown, the second feature extraction module (GAMC Block) includes a parallel hole convolution, a first feature extraction module (MobileOne Block), a global pooling branch and a convolutional pre-feedback network. The parallel hole convolution performs a parallel hole convolution on the features input to the second feature extraction module, and adds the hole convolution results to obtain the added features, so as to realize the fusion of feature information of different receptive fields. The first feature extraction module performs feature extraction on the added features, extracts the features and adds the features input to the second feature extraction module to form a residual structure, and obtains the enhanced extracted features. The global pooling branch performs global average pooling and two full connections on the features input to the second feature extraction module, and then performs normalization processing to obtain the weights on the feature channels input to the second feature extraction module, and multiplies the enhanced extracted features with the weights on the feature channels to obtain the channel information enhanced features; the channel information enhanced features are subjected to attention feature extraction to obtain the attention features of the enhanced features, and the attention features of the enhanced features are processed by the convolutional pre-feedback network and added to the attention features of the enhanced features, and the attention feature extraction is performed again to obtain the output features of the second feature extraction module.

[0077] Preferably, the parallel dilated convolution includes parallel dilated convolution operations with expansion rates of 1, 2, 5 and 7. The global pooling branch performs global average pooling and two full connections on the features input to the second feature extraction module, and then uses the Sigmoid function to perform normalization to obtain the weights on the feature channels of the second feature extraction module. The second feature extraction module uses the EMA attention mechanism to perform attention feature extraction.

[0078] The convolutional feedback network is used to perform nonlinear changes on the features of the input convolutional feedback network. It includes a convolutional layer with a convolution kernel size of 7 and two fully connected layers. The convolutional feedback network can improve the expressiveness of the features.

[0079] Taking the second feature extraction layer as an example, the calculation process is as follows:

[0080] MS=M(SPD(F1))

[0081]

[0082] G2=EMA(FC(FC(C k＝7 EMA((G1)))))+G1

[0083] Where: F1 is the feature extracted by the first feature extraction layer, SPD is the SPD operation, M is the feature extraction operation of the first feature extraction module, MS is the feature extracted by the first feature extraction module, C is the convolution operation, d is the dilation rate, k is the convolution kernel size, FC is the full connection, Sg is the Sigmoid operation, EMA is the EMA attention mechanism, G1 is the channel information enhancement feature, and G2 is the output feature of the first feature extraction module.

[0084] The process of using the feature fusion network to fuse the features output by the backbone feature network in this embodiment is as follows:

[0085] The process of the feature fusion network fusing the features extracted by the last three feature extraction layers of the backbone feature network is as follows: adding position coding information to the features extracted by the last feature extraction layer of the backbone feature network, and fusing the features with the added position coding information with the features extracted by the penultimate and penultimate feature extraction layers to obtain fused features.

[0086] like Figure 7As shown, this embodiment uses the Attention-based Intra-scale Feature Interaction (AIFI) module as a bottleneck to add position coding information to the features extracted by the last feature extraction layer of the backbone feature network to realize the interaction of high-dimensional feature information within the scale; the process of adding position coding information to the features extracted by the last feature extraction layer of the backbone feature network is as follows: flattening the features extracted by the last feature extraction layer of the backbone feature network in the third dimension to obtain flattened feature information, swapping the positions of the first dimension and the second dimension of the flattened feature information to obtain transformed feature information, position coding the flattened feature information to obtain position coding information, adding the position coding information to the transformed feature information, and performing feature enhancement processing through a multi-head self-attention mechanism and a fully connected layer to obtain feature information with position coding added, swapping the positions of the first dimension and the second dimension of the feature information with position coding added, and reshaping in the second dimension to obtain features consistent with the feature dimensions extracted by the last feature extraction layer, which are features with added position coding information.

[0087] Preferably, the position coding information and the transformed feature information are sent to an encoder of a transformer, so that the position coding information is added to the transformed feature information. The calculation process of AIFI is described with the fifth feature extraction layer as the last feature extraction layer. The calculation process of AIFI is as follows:

[0088] O 5-1 =MHA(N(P(F5)))+P(F5)

[0089] O5=MLP(N(O 5-1 ))+O 5-1

[0090] Where: O 5-1 is the process output, F5 is the feature extracted by the fifth feature extraction layer, P is the position encoding operation, N is the normalization operation, MHA is the multi-head self-attention mechanism, and MLP is the multi-layer perceptron operation.

[0091] The decoder is used to perform feature fusion on the features extracted from the output of the backbone feature network. When the backbone feature network includes a five-layer feature extraction network, the decoder fuses the features extracted from the first feature extraction layer, the second feature extraction layer, and the third feature extraction layer to obtain a second fused feature, fuses the features extracted from the second feature extraction layer, the third feature extraction layer, and the fourth feature extraction layer to obtain a third fused feature, fuses the features extracted from the third feature extraction layer, the fourth feature extraction layer, and the fifth feature extraction layer with position coding information added to obtain a fourth fused feature; and fuses the second fused feature, the third fused feature, and the fourth fused feature to obtain an enhanced feature.

[0092] like Figure 8 As shown in the figure, the decoder is constructed using the ASFF structure. In the neural network, the low-dimensional feature layer is responsible for extracting local detail features, while the high-dimensional feature layer is responsible for extracting global abstract features. The ASFF structure first uses different convolution and pooling layers to obtain features of different sizes. Then, through adaptive weight learning, the contribution weights of information of different scales are adjusted, so that the network automatically learns the importance of each scale feature, thereby better capturing multi-scale semantic information. Taking the fusion of the features extracted from the last three feature extraction layers of the backbone feature network output containing five feature extraction layers as an example, the calculation process is as follows:

[0093] O4=F3×w3+DUC s＝2 (F4)×w4+DUC s＝4 (O5)×w5

[0094] Where: F i is the feature extracted by the i-th feature extraction layer, O5 is the feature after AIFI, that is, the feature extracted by the fifth feature extraction layer with added position encoding information, DUC is continuous dense upsampling, s is the upsampling multiple, and w i is the adaptive weight, and O4 is the fourth fusion feature.

[0095] In the feature fusion network of this embodiment, dense upsampling convolution is used instead of traditional interpolation upsampling to avoid the loss of detail information caused by the upsampling process. Fig. 9 As shown in the figure, assuming that the feature size before upsampling is H×W, the feature size before upsampling becomes h×w×c after convolution, where h=H / r, w=W / r, and r is the dimensionality reduction ratio. Dense upsampling convolution first convolves the feature before upsampling from h×w×c to h×w×r2 to obtain the convolution feature, and then reshapes the convolution feature to the same size H×W as the feature before upsampling. Taking the fifth level as an example, the calculation process is as follows:

[0096] O3-5 =R(C r (O5))

[0097] Where: O 3-5 It is the result of upsampling the output features of the fifth layer to the third layer, that is, the feature extracted by the fifth feature extraction layer with added position coding information is upsampled twice. R is the reshape operation, C is the convolution operation, r is the dimensionality reduction ratio, and O5 is the feature after AIFI, that is, the feature extracted by the fifth feature extraction layer with added position coding information.

[0098] In order to solve the problem that the uneven positive and negative samples of road images make the model more likely to predict negative samples, FocalDice Loss is constructed as the loss function of the road crack recognition model to verify the loss of the road crack recognition model. The specific loss function of the road crack recognition model is:

[0099]

[0100] Where DiceLoss is the Dice loss value, |P| and |T| are the number of elements in the segmentation result P and the true label T respectively, |P∩T| represents the number of their intersection elements, and Smooth is the smoothness index.

[0101]

[0102] In the formula, FocalLoss is the Focal loss value, p is the predicted probability of the model, α is the weight of the sample, and γ is the hyperparameter of FocalLoss.

[0103] Since the target pixels in the pavement disease image account for a small proportion, the network is more likely to predict negative samples. Therefore, the above two loss functions are combined to construct Focal Dice Loss for verifying the model prediction results in the pavement crack segmentation task. The specific loss function of the pavement crack recognition model is:

[0104] FocalDiceLoss=FocalLoss-βDiceLoss

[0105] Among them, FocalDiceLoss is the loss value of the pavement crack identification model; β is the weight of Dice Loss.

[0106] After constructing the pavement crack recognition model, this embodiment obtains an existing pavement crack image, and labels the position and shape of the existing pavement crack image to obtain the labeled pavement crack image as a training image, and trains the pavement crack recognition model. After the training is completed, a trained pavement crack recognition model is obtained.

[0107] The specific process of obtaining the trained pavement crack recognition model is as follows:

[0108] Step 1: The training data set is prepared. Based on the two-dimensional images of highway asphalt pavement of different scales, qualities, and positive and negative sample contrasts obtained by the multifunctional pavement inspection vehicle, the obtained pavement crack images are annotated using Labelme software to obtain a JSON label file containing detailed information such as the crack location and shape.

[0109] Step 2: Dataset processing and image enhancement: The JSON file is processed using Python programming to generate a binary crack label map in VOC format, that is, the background pixel is 0 and the crack target pixel is 1. All data are uniformly adapted to a size of 512×512, and data enhancement is performed using operations such as flipping, symmetry, scaling, length and width twisting, and random adjustment of brightness and contrast.

[0110] Step 3: Divide the data set into training set, validation set and test set in a ratio of 4:1. Divide the data set into training set and validation set in a ratio of 4:1.

[0111] Step 4: Model training: Use the divided training set and validation set to train and validate the pavement crack recognition model.

[0112] Step 6: Model verification: Use the test dataset to evaluate the trained pavement crack recognition model.

[0113] This embodiment uses a semantic segmentation task evaluation system to calculate the performance of the trained pavement crack recognition model. The evaluation indicators include mIoU (Mean Intersection over Union), mPA (Mean Pixel Accuracy), mPrecision (Mean Precision), mRecall (Mean Recall) and F1 score.

[0114] IoU is the intersection of the predicted positive class and the actual positive class divided by their union. mIoU is the sum of the intersection of all classes and the average. The calculation formula is as follows:

[0115]

[0116] Among them, TP i is the number of pixels of the true positives of the i-th category, FP i is the number of pixels of false positives of the i-th category, FN iis the number of pixels of false negatives of the i-th category, and N is the total number of categories.

[0117] mPA is used to measure the average proportion of correctly classified pixels in the entire dataset.

[0118] The calculation formula is as follows.

[0119]

[0120] Among them, TP i is the number of pixels of the true positives of the i-th category, TN i is the number of pixels of the true negatives of the i-th category, FP i is the number of pixels of false positives of the i-th category, FN i is the number of pixels of false negatives of the i-th category, and N is the total number of categories.

[0121] mPrecision represents the average value of the correct classification ratio among all samples classified as positive categories. Its calculation formula is as follows.

[0122]

[0123] Among them, TP i is the number of pixels of the true positives of the i-th category, FP i is the number of pixels of false positives of the i-th category, and N is the total number of categories.

[0124] mRecall represents the average value of the proportion of correctly classified samples in all actual positive categories. Its calculation formula is as follows.

[0125]

[0126] Among them, TP i is the number of pixels of the true positives of the i-th category, FN i is the number of pixels of false negatives of the i-th category, and N is the total number of categories.

[0127] F1 Score takes mPrecision and mRecall into consideration and is the harmonic mean of mPrecision and mRecall. Its calculation formula is as follows.

[0128]

[0129] The acquired highway asphalt pavement image is input into the trained pavement crack recognition model to obtain the highway asphalt pavement crack recognition result.

[0130] The present embodiment discloses a method for identifying cracks in asphalt pavement of highways, which can realize accurate identification of cracks in asphalt pavement of highways by using a pavement crack identification model. A new backbone feature network is constructed based on MobileOne Block, dilated convolution and SPDConv, which can capture broader context information while retaining low-dimensional feature information, thereby improving the model's ability to express multi-scale features. The EMA attention mechanism is introduced to evenly distribute feature information in feature groups, so that the model can accurately capture the spatial features of the image and improve the modeling ability of the relationship between features. The AIFI module constructed based on Transformer is used as the feature fusion network of the model to perform intra-scale interaction on high-dimensional global information, so that the model can better understand the relationship between different regions in the high-dimensional global information of the image. The decoder is constructed using the ASFF structure of spatial scale information adaptive fusion technology, and DUC is used instead of traditional interpolation upsampling to fully integrate high-dimensional global information with low-dimensional detail information through adaptive weights, thereby enhancing the model's learning ability for detail features. The FocalDice Loss loss function is introduced to perform loss verification of the model, thereby avoiding the problem that the model is more likely to predict negative samples due to uneven distribution of positive and negative samples of pavement images.

[0131] Example 2

[0132] In this embodiment, a highway asphalt pavement crack identification system is disclosed, comprising:

[0133] An image acquisition module, used for acquiring an image of a highway asphalt pavement;

[0134] The image recognition module is used to obtain the highway asphalt pavement crack recognition result based on the highway asphalt pavement image and the trained pavement crack recognition model, wherein the pavement crack recognition model includes a backbone feature network and a feature fusion network, the backbone feature network includes a multi-layer feature extraction layer, the multi-layer feature extraction layer performs continuous multi-layer feature extraction on the highway asphalt pavement image, and inputs the features extracted from each layer into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, all fused features are fused to obtain enhanced features, and the highway asphalt pavement crack recognition result is obtained based on the enhanced features.

[0135] Example 3

[0136] In this embodiment, an electronic device is disclosed, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps described in the method for identifying cracks in a highway asphalt pavement disclosed in Example 1 are completed.

[0137] Example 4

[0138] In this embodiment, a computer-readable storage medium is disclosed for storing computer instructions. When the computer instructions are executed by a processor, the steps of a method for identifying cracks in a highway asphalt pavement disclosed in Embodiment 1 are completed.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying cracks in a highway asphalt pavement, characterized in that: include: Acquire highway asphalt pavement images; According to the highway asphalt pavement image and the trained pavement crack recognition model, the highway asphalt pavement crack recognition result is obtained, wherein the pavement crack recognition model includes a backbone feature network and a feature fusion network, the backbone feature network includes a multi-layer feature extraction layer, the multi-layer feature extraction layer performs continuous multi-layer feature extraction on the highway asphalt pavement image, and inputs the features extracted from each layer into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, all fused features are fused to obtain enhanced features, and based on the enhanced features, a Highway asphalt pavement crack recognition results; The backbone feature network includes multiple feature extraction layers. The first feature extraction layer performs two consecutive feature extractions on the highway asphalt pavement image through two first feature extraction modules to obtain image features, and performs attention feature extraction on the image features to obtain the first layer of features extracted by the first feature extraction layer. Except for the first feature extraction layer, each of the remaining feature extraction layers uses the features extracted by the previous feature as input. The last feature extraction layer downsamples the input features, and then extracts features from the downsampled features through a first feature extraction module to obtain the features extracted by the last feature extraction layer. Except for the first feature extraction layer, The remaining feature extraction layers except the first and last feature extraction layers all downsample the input features, and then perform feature extraction on the downsampled features through the first feature extraction module, and then perform feature extraction through multiple consecutive second feature extraction modules to obtain the features extracted by the corresponding feature extraction layer; the second feature extraction module includes parallel hole convolution, the first feature extraction module, the global pooling branch and the convolution pre-feedback network, the parallel hole convolution performs parallel hole convolution on the features input to the second feature extraction module, and adds the hole convolution results to obtain the added features, and the first feature extraction module performs feature extraction on the added features to extract the features. The enhanced extracted features are added to the features input to the second feature extraction module to obtain the enhanced extracted features. The global pooling branch performs global average pooling and two full connections on the features input to the second feature extraction module, and then performs normalization to obtain the weights on the feature channels of the second feature extraction module. The enhanced extracted features are multiplied by the weights on the feature channels to obtain the channel information enhanced features. Attention feature extraction is performed on the channel information enhanced features to obtain the attention features of the enhanced features. The attention features of the enhanced features are processed by the convolutional pre-feedback network and added to the attention features of the enhanced features. Attention feature extraction is performed again to obtain the output features of the second feature extraction module.

2. A method for identifying cracks in a highway asphalt pavement as claimed in claim 1, characterized in that: The first feature extraction module includes a depth convolution part and a point-by-point convolution part. The depth convolution part includes three branches, namely, a first convolution layer, a second convolution layer and a batch normalization layer. The first convolution layer, the second convolution layer and the batch normalization layer respectively extract features from the features input into the first feature extraction module to obtain the first feature, the second feature and the third feature. After fusing the first feature, the second feature and the third feature, the first feature, the second feature and the third feature are input into the point-by-point convolution part. The point-by-point convolution part includes two branches, namely, a point-by-point convolution layer and a batch normalization layer. The point-by-point convolution layer and the batch normalization layer respectively extract the fourth feature and the fifth feature from the features input into the point-by-point convolution part, and fuse the fourth feature and the fifth feature as the output of the first feature extraction module.

3. A method for identifying cracks in a highway asphalt pavement as claimed in claim 1, characterized in that: The EMA attention mechanism is used to extract attention features from features. The process is as follows: the EMA attention mechanism divides the input features into different feature groups, and performs convolution, average pooling and convolution operations in the X and Y directions on each group of features. The average pooling results in the X and Y directions are stacked on the channel dimension, and the weight of the feature group is obtained through the Sigmoid operation, which is multiplied with the features of the corresponding group. The multiplication result and the convolution result of the features of the corresponding group are average pooled and Softmax operations are performed respectively to obtain the first convolution feature and the second convolution feature. The first convolution feature and the second convolution feature are cross-multiplied and then added. Then, the final feature weight is obtained through the Sigmoid operation, and the final feature weight is applied to the feature group as the output of the EMA attention mechanism.

4. A method for identifying cracks in a highway asphalt pavement as claimed in claim 1, characterized in that: The process of the feature fusion network fusing the features extracted by the last three feature extraction layers of the backbone feature network is as follows: adding position coding information to the features extracted by the last feature extraction layer of the backbone feature network, and fusing the features with the added position coding information with the features extracted by the penultimate and penultimate feature extraction layers to obtain fused features.

5. A method for identifying cracks in a highway asphalt pavement as claimed in claim 4, characterized in that: The process of adding position coding information to the features extracted by the last feature extraction layer of the backbone feature network is as follows: flattening the features extracted by the last feature extraction layer of the backbone feature network in the third dimension to obtain the flattened feature information, swapping the positions of the first dimension and the second dimension of the flattened feature information to obtain the transformed feature information, position encoding the flattened feature information to obtain the position coding information, adding the position coding information to the transformed feature information, performing feature enhancement processing through a multi-head self-attention mechanism and a fully connected layer to obtain feature information with added position coding, swapping the positions of the first dimension and the second dimension of the feature information with added position coding, and reshaping it in the second dimension to obtain features consistent with the feature dimensions extracted by the last feature extraction layer, which are features with added position coding information.

6. A highway asphalt pavement crack identification system, characterized in that: include: An image acquisition module, used for acquiring an image of a highway asphalt pavement; The image recognition module is used to obtain the highway asphalt pavement crack recognition result according to the highway asphalt pavement image and the trained pavement crack recognition model, wherein the pavement crack recognition model includes a backbone feature network and a feature fusion network, the backbone feature network includes a multi-layer feature extraction layer, the multi-layer feature extraction layer performs continuous multi-layer feature extraction on the highway asphalt pavement image, and inputs the features extracted from each layer into the feature fusion network, the feature fusion network fuses each layer of features between the first layer and the last layer of features extracted by the backbone feature network with the two adjacent layers of features to obtain multiple fused features, and fuses all the fused features to obtain enhanced features, according to Strengthen the features to obtain the results of highway asphalt pavement crack recognition; the backbone feature network includes multiple feature extraction layers, the first feature extraction layer performs two consecutive feature extractions on the highway asphalt pavement image through two first feature extraction modules to obtain image features, and performs attention feature extraction on the image features to obtain the first layer of features extracted by the first feature extraction layer, and each feature extraction layer except the first feature extraction layer uses the features extracted by the previous feature as input, and the last feature extraction layer downsamples the input features, and then extracts features from the downsampled features through a first feature extraction module to obtain the features extracted by the last feature extraction layer, except the first The remaining feature extraction layers except the first feature extraction layer and the last feature extraction layer all downsample the input features, and then perform feature extraction on the downsampled features through the first feature extraction module, and then perform feature extraction through multiple consecutive second feature extraction modules to obtain the features extracted by the corresponding feature extraction layer; the second feature extraction module includes parallel hole convolution, the first feature extraction module, the global pooling branch and the convolution pre-feedback network, the parallel hole convolution performs parallel hole convolution on the features input to the second feature extraction module, and adds the hole convolution results to obtain the added features, and performs feature extraction on the added features through the first feature extraction module to extract The features are added to the features of the second feature extraction module input to obtain enhanced extracted features. The global pooling branch performs global average pooling and two full connections on the features of the second feature extraction module input to obtain the weights on the feature channels of the second feature extraction module. The enhanced extracted features are multiplied by the weights on the feature channels to obtain channel information enhanced features. Attention feature extraction is performed on the channel information enhanced features to obtain attention features of the enhanced features. The attention features of the enhanced features are processed by the convolutional pre-feedback network and added to the attention features of the enhanced features. Attention feature extraction is performed again to obtain the output features of the second feature extraction module.

7. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the method for identifying cracks in a highway asphalt pavement as claimed in any one of claims 1 to 5 are completed.

8. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a method for identifying cracks in a highway asphalt pavement as described in any one of claims 1 to 5.

Images