A multi-scale road surface crack detection method based on deep and shallow layer attention feature fusion

Abstract

The application provides a multi-scale road surface crack detection method based on deep and shallow layer attention feature fusion, collects a road surface crack image and pre-processes the image; the image is input into a backbone network to extract hierarchical features P2, P3, P4 and P5; P5 is input into a scale adaptive SA module to obtain multi-scale information; P2, P3 and P4 are input into a neck to refine features; the neck comprises a deep and shallow layer attention feature fusion DSAFF module and a feature refinement FR module; the DSAFF module uses channel attention to obtain channel information, uses a block decoupling attention BDAM module to obtain spatial information of deep and shallow layer fusion, and the FR is used to refine edge information; the detection head is input respectively to obtain a final detection result; a SimOTA label allocation scheme during training is introduced. The application can capture more sufficient multi-scale feature information under the condition of increasing a small amount of calculation, and can strengthen feature extraction of a damage area by the network, thereby relieving interference of a background area.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision road crack detection, and particularly relates to a multi-scale road crack detection method based on deep and shallow layer attention feature fusion. BACKGROUND

[0002] Road crack detection can be mainly divided into two categories: a method based on traditional image processing technology and a method based on deep learning. Compared with the traditional image processing method, deep learning has the advantages of strong learning ability and high robustness, and is favored by more and more researchers.

[0003] The road crack detection based on the target detection framework mainly uses the vehicle-mounted camera or the camera carried by the unmanned aerial vehicle to obtain the road surface image. In order to enhance the performance of the model, some methods study the backbone network. Wan et al. change the backbone network of YOLOv5s into a lighter backbone network, and use the BiFPN structure in the Neck layer. Although good results are achieved under the condition of meeting real-time, the factors of weak crack semantics and irregular shape are ignored, and the multi-scale features cannot be fully extracted. Xu et al. propose a backbone network LrNet which reuses low-level features and mixes appropriate range-dependent features, aiming at the weak semantic information and abnormal geometric properties of road damage. The problem of weak semantics is relieved through multi-level feature complementation. However, their design is too complex to meet the real-time detection. Some other methods increase the feature acquisition ability through a larger and stronger backbone network, but greatly increase the calculation amount.

[0004] In addition, in order to suppress background noise, many pavement crack detection methods pay more attention to the damage area by introducing attention mechanisms to improve the accuracy of detection. Li et al. based on the YOLOv4 detection framework, by introducing the CBAM (Convolutional Block Attention Module) module, and explored the influence of the embedding position of the CBAM module on the accuracy. Guo et al. based on the YOLOv5s, introduced the SE (Squeeze-and-Excitation) module and other operations, so that the detection effect is better. Yao et al. based on the YOLOv5 model, explored the optimization of adding attention mechanism modules, tried two adding ways and three adding positions, and finally realized good performance. However, these methods are to introduce general attention mechanisms to alleviate the background influence, which is suboptimal. Zhang et al. obtained the edge information of the picture by edge detection, and processed these information to realize the attention to the crack area. However, these methods all ignore the slender morphological characteristics of the cracks, and cannot obtain the long-distance dependence relationship of the long strip-shaped cracks. Ren et al. introduced CoordAttention into the backbone network to capture long-distance dependence relationship, but ignored the influence of background factors on the acquisition of long-distance dependence relationship. In general, these methods do not adequately consider the subtle differences between the damage area and the background area, resulting in the model's inability to effectively strengthen the crack features and alleviate the interference of background information. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, provide a multi-scale pavement crack detection method based on deep and shallow layer attention feature fusion, so as to realize more rapid and accurate pavement crack detection. The present application proposes a multi-scale pavement crack detection network based on deep and shallow layer attention feature fusion based on YOLOv5s detector, named SDFNet. First, a novel scale adaptive module is proposed, deformable convolution DCN is introduced, DCN is combined in series by suitable inflation coefficient, and jump connection is adopted to relieve the information loss caused by too large inflation coefficient, so as to fully fuse multiple scale information, realize scale self-adaptation of damage area under the condition of meeting real-time, and replace the SPP module in YOLOv5s. Secondly, the present application proposes a deep and shallow layer attention feature fusion module, uses channel attention mechanism to capture channel information in deep layer of network, obtains spatial information in shallow layer and deep layer of network through the proposed block decoupling attention mechanism, fuses channel information and spatial information, strengthens crack feature and weakens background information, so that the model pays more attention to damage area and relieves the interference of background. Replace the ordinary splicing operation in the Neck part of YOLOv5s. Then, the proposed feature refinement module replaces the C3 structure in the Neck part of YOLOv5s, so as to fuse the features. Finally, the dynamic label assignment scheme SimOTA is introduced, so as to replace the static label assignment of YOLOv5s, so that the positive and negative samples are more reasonable during training.

[0006] In order to achieve the above purpose, the present application provides the following technical scheme: a multi-scale pavement crack detection method based on deep and shallow layer attention feature fusion, comprising the following steps:

[0007] S1, collecting pavement crack image and preprocessing the image;

[0008] S2, inputting the image into the backbone network to extract hierarchical features P2, P3, P4 and P5;

[0009] S3, inputting P5 into the scale adaptive SA module to obtain multi-scale information;

[0010] S4, inputting P2, P3 and P4 into the Neck to refine the features; the Neck comprises a deep and shallow layer attention feature fusion DSAFF module and a feature refinement FR module; the DSAFF module uses channel attention to obtain channel information, uses a block decoupling attention BDAM module to obtain spatial information of deep and shallow layer fusion, and the FR is used to refine edge information;

[0011] S5, inputting into the detection head respectively to obtain the final detection result;

[0012] S6, introducing SimOTA to optimize the label assignment scheme during training.

[0013] Further, step S3 comprises:

[0014] The SA module is distinguished from the parallel max-pooling operation used in SPP, and selects to use a serial hollow deformable convolution to obtain larger receptive field information, and the dilation coefficient used is combined as (3, 6, 12); at the same time, a jump connection operation is adopted;

[0015] First, a 1x1 convolution is added to reduce the input feature map dimension to C out / 2, to obtain a feature map f1, which is then input into a 3x3 hollow deformable convolution with a dilation coefficient of 3, and the obtained result is added to f1 to obtain a feature map f2; the dilation coefficient of the 3x3 hollow deformable convolution is changed to 6 and 12 respectively to obtain feature maps f3 and f4; finally, f i ,i∈(1,2,3,4) are spliced and input into a 1x1 convolution to obtain an output feature map f5; the calculation is as shown in the following formula;

[0016]

[0017] x out =f 1×1 (C(x0,x1,x2,x3))

[0018] Where x in , x out represent input and output feature maps respectively, C() and f 1×1 () represent splicing and convolution operations respectively, represents a 3x3 hollow deformable convolution with a convolution kernel size, where d represents the dilation coefficient.

[0019] Further, in step S4, the deep and shallow layer attention feature fusion DSAFF module comprises:

[0020] The DSAFF module includes two branches, one is an attention branch and the other is a shallow layer spatial information acquisition branch; the attention branch uses a channel attention CA module and a BDAM module to obtain channel and spatial information;

[0021] The DSAFF module first uses high-level features and low-level features to do simple splicing, and then uses a channel attention CA to obtain channel information;

[0022] The features outputted by the shallow spatial information acquisition branch and the CA module are fused after processing and inputted into the BDAM module to acquire spatial information fused in deep and shallow layers; the shallow spatial information acquisition branch is down-sampled by a 3x3 maximum pooling with a step of 2, and then the channel number is changed to 1 by using global maximum pooling to acquire shallow spatial information; the CA module is compressed to the dimension of Cx1x1 by using global average pooling, scaled by using 1x1 convolution, and the importance of the channel is acquired by using Sigmoid activation;

[0023] The BDAM module firstly performs deep and shallow spatial information fusion, and the process is as shown in the following formula;

[0024]

[0025] For the fused feature, is the cth channel of the spliced feature, C is the channel number, F q is the shallow feature;

[0026] F r is processed by MN, and the formula is as shown below;

[0027] F R = y*F r / (max(abs(F r )))

[0028] wherein max() represents the maximum value in F r , abs() represents the absolute value of all data in F r , and y is a self-defined value;

[0029] F R is divided into several regions, and then horizontal and vertical pooling operations are performed on each region, and the process is as shown in the following formula;

[0030]

[0031]

[0032] wherein F′ R,P , F″ R,P , F R,P respectively represent the output features of horizontal and vertical pooling of the Pth patch and F R , c, i, j respectively represent the dimension, row and column, wherein

[0033] The high-frequency region and the low-frequency region of the feature are decoupled, as shown in the following formula;

[0034] f low,H(i,j) = min(f patch (i,j), F R ′ ,P (i,j)

[0035] f low,V (i,j) = min(f patch (i,j), F″ R,P (i,j)

[0036] f high,H (i,j) = max(f patch (i,j), F′ R,P (i,j)

[0037] f high,V (i,j) = max(f patch (i,j), F″ R,P (i,j)

[0038] where f high,H , f high,V、 f low,H , f low,V represent decoupled high and low frequency horizontal and vertical pooling feature maps respectively, f patch represents region feature map, min represents taking small value at corresponding position, and max represents taking large value;

[0039] Sigmoid activation is performed on the features, as shown in the following formula.

[0040]

[0041] where f i,j p represents the value of the ith row and jth column of the pth patch, represents the value of the ith row and jth column of f high,H after Sigmoid, Similarly, the above formula is f i,j p greater than and greater than , represents the value of the first row and jth column of F′ R,P If the former is less than, the corresponding f is changed to The latter is the same.

[0042] Further, in step S4, the feature refinement FR module comprises:

[0043] First, two 1x1 convolutions are performed to adjust the dimensions, so that the number of channels is adjusted to C out / 2, and forms two branches; replace the common 3*3 convolution by cascaded 3*1 and 1*3 convolution, the output dimension of the first 3*1 convolution is C out , and the output dimension of the following 1*3 convolution is C out / 2; one branch is subjected to two cascaded 3*1 and 1*3 convolution, the features obtained in this process are spliced with the features of the initial input of the other branch, and then a 1*1 convolution is used to fuse the features to form the final refined features.

[0044] Further, in step S6, the step flow of the SimOTA is as follows:

[0045] Firstly, the positive sample candidate region is determined, and the sample point within the range of the center point 5*5 of the real frame is defined as the positive sample candidate region; secondly, the sum of the classloss, iouloss and the loss of the non-candidate region ~IsInBoxCenter of each anchor frame with the real frame is calculated as the cost, and the specific cost formula is as follows:

[0046] Cost=ClsLoss CE +α*RegLoss IOU +β*(~IsInBoxCenter)

[0047] Wherein, α and β are weight coefficients; then, the 10 anchor frames with the minimum cost for each real frame are selected, and the sum of the IOU of the 10 anchor frames is calculated to obtain the dynamic k value of each real frame; finally, the first k anchor frames with the minimum cost for each real frame are taken as positive samples, and the rest are taken as negative samples, and the loss is calculated for training, and the loss function during training is as follows: the loss is composed of ClsLoss CE , RegLoss IOU and ObjLoss BCE , and λ, α and β are weight coefficients.

[0048] Cost=λ*ClsLoss CE +α*RegLoss IOU +β*ObjLoss BCE .

[0049] Compared with the prior art, the beneficial effects of the present application are:

[0050] 1) The present application proposes a multi-scale road crack detection framework with deep and shallow feature attention mechanism, which can maintain a good balance between speed and accuracy.

[0051] 2) The application proposes a novel scale adaptive module, which obtains adaptive scale deformable damage features through series expansion deformable convolution combined with appropriate expansion coefficient, and can capture more sufficient multi-scale feature information with a small amount of calculation;

[0052] 3) The application proposes a deep and shallow layer attention feature fusion module, in order to strengthen the model's discriminability to the crack area, a deep and shallow layer attention feature fusion module is proposed, which obtains channel information through channel attention, captures spatial information of deep and shallow layer fusion through the proposed block decoupling attention, and then refines the features through the proposed feature refinement module. It can strengthen the network's feature extraction of the damage area, thereby relieving the interference of the background area. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The flowchart of the application of a multi-scale pavement crack detection method based on deep and shallow layer attention feature fusion;

[0054] Figure 2 The overall framework diagram of the application;

[0055] Figure 3 The schematic diagram of the scale adaptive (SA) module;

[0056] Figure 4 The schematic diagram of the deep and shallow layer attention feature fusion (DSAFF) module;

[0057] Figure 5 The schematic diagram of the channel attention mechanism (CA);

[0058] Figure 6 The schematic diagram of the block decoupling attention mechanism (BDAM);

[0059] Figure 7 The schematic diagram of the feature refinement (FR) module. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. The specific examples described here are only used to explain the technical scheme of the application, and are not limited to the application.

[0061] The road crack detection based on the vehicle-mounted camera has a large scale variation because the crack itself is long and short, thick and thin. Meanwhile, according to the distance of the crack region from the vehicle-mounted camera, the scale variation of the crack target is further caused. In addition, the crack has the characteristics of less semantic information and irregular shape. The existing method does not process the characteristics of the crack well, and often ignores this problem when obtaining multi-scale information, resulting in poor detection performance of the crack under the condition of large scale variation. In order to solve this problem, the present application provides a scale adaptive module. Through the series expansion deformable convolution of the appropriate expansion coefficient combination, the adaptive scale deformable damage feature is obtained. This method can enhance the model's ability to capture features of cracks of various scales.

[0062] Some existing road damage detection methods ignore the influence of the background region on accurate detection of the road crack, and they lack the ability to identify the crack region and the ability to alleviate the background interference. Some methods that pay attention to this problem mostly use conventional attention mechanisms and are designed for the characteristics of the crack being long and thin and having weak semantics, ignoring the long-distance dependence relationship of the crack. In order to solve this problem, the present application provides a detection scheme based on a deep and shallow layer attention feature fusion module and a feature refinement module. After the image is input into the backbone network to obtain multi-scale features, the P2, P3 and P4 feature maps obtained in the backbone network are input into the Neck to refine the features. The Neck is mainly composed of the deep and shallow layer attention feature fusion (DSAFF) module and the feature refinement (FR) module proposed by the present application. The DSAFF uses channel attention to obtain channel information, uses the proposed block decoupled attention (BDAM) module to obtain spatial information of deep and shallow layer fusion, and the FR is used to refine edge information. Finally, the detection head is input to obtain the final detection result. At the same time, in order to make the positive and negative samples more reasonable during training, the SimOTA dynamic label assignment scheme is introduced.

[0063] As Figure 1 shown, the technical scheme of the present application includes the following steps:

[0064] S1: collecting road crack images and pre-processing the images;

[0065] S2: constructing a multi-scale road crack detection model based on deep and shallow layer attention feature fusion;

[0066] S3: optimizing the label assignment scheme during training;

[0067] S4: training the multi-scale road crack detection model based on deep and shallow layer attention feature fusion constructed;

[0068] S5: verifying the performance of the model.

[0069] S1: For the images in the RDD2020 dataset and the CNRDD dataset, the respective resolutions are adjusted to input sizes of 640x640 and 1600x1600, respectively, according to their respective image sizes. In the training phase, data augmentation methods such as image rotation, color space transformation, and scale transformation are used to enrich the data quantity of the samples and improve the generalization performance of the model.

[0070] S2.1 adopts CSPDarkNet53 as the backbone network to extract features;

[0071] S2.2 The overall framework proposed by the present application is shown in Figure 2 It aims to solve how to alleviate the poor performance of the model in the case of large crack size variation and how to improve the ability of the model to strengthen crack features and alleviate background information interference. The proposed multi-scale road crack detection model based on deep and shallow layer attention feature fusion, which is referred to as SDFNet detection model by the present application, includes:

[0072] First, the pre-trained CSPDarkNet53 is used as a backbone network to extract hierarchical features (P2, P3, P4, P5). Second, P5 is input into the scale adaptation (SA) module proposed by the present application to obtain multi-scale information. Then, P2, P3, and P4 are input into the Neck to refine the features. The Neck is mainly composed of the deep and shallow layer attention feature fusion (DSAFF) module and the feature refinement (FR) module proposed by the present application. DSAFF uses channel attention to obtain channel information and the proposed block decoupled attention (BDAM) module to obtain spatial information of deep and shallow layer fusion. FR is used to refine edge information. Finally, the detection head is input to obtain the final detection result.

[0073] The size of the receptive field roughly determines how much semantic information of the original image is obtained. A too large receptive field will cause interference of redundant information, and a too small receptive field will result in insufficient target information acquisition, therefore, multi-scale information acquisition is necessary. In the YOLOv5s detector, for multi-scale feature acquisition, the SPP module is used, which uses the maximum pooling operation to obtain a larger receptive field. However, the maximum pooling operation discards a large amount of picture information, which is fatal for pixel-scarce cracks. At the same time, road crack detection has the characteristic of irregular shape, and conventional convolution and pooling operations cannot effectively obtain damage area information.

[0074] To solve the above problems, in order to reduce information loss and adapt to the irregular characteristics of the crack shape, the scale adaptive module SA is proposed. The SA module proposed in the application is different from the parallel max-pooling operation used in SPP, and a serial dilated deformable convolution is selected to obtain larger receptive field information. The serial operation can use fewer convolution operations to the same receptive field size, reducing the amount of calculation. The dilation coefficient combination used in the application is (3, 6, 12), and for three dilated convolutions of parallel operation, the combination receptive field is only 12, while for serial operation, the theoretical receptive field reaches (3+(3-1)×(3-1))+(3+(3-1)×(6-1))+(3+(3-1)×(12-1))-2=43, which is much higher than the receptive field value obtained by parallel operation. The deformable convolution DCN can adaptively fit the damage area, so as to capture more damage features. However, a problem caused by too large dilation coefficient is that the sampling points are too sparse, which will cause information loss. In order to make up for the information loss caused by too large dilation coefficient, the application adopts a jump connection operation.

[0075] As shown in Figure 3 , the specific structure of the scale adaptive module (SA) is as follows: first, a 1x1 convolution is added to reduce the input feature map dimension to C out / 2, to obtain the feature map f1, which is then input into the 3x3 dilated deformable convolution with a dilation coefficient of 3, and the obtained result is added to f1 to obtain the feature map f2. The dilation coefficients of the 3x3 dilated deformable convolution are changed to 6 and 12 respectively, and the above process is repeated to obtain the feature maps f3 and f4. Finally, f i ,i∈(1,2,3,4) are spliced and input into a 1x1 convolution to obtain an output feature map f5 rich in rich context information. The calculation is as follows.

[0076]

[0077] x out =f 1×1 (C(x0,x1,x2,x3))

[0078] Where x in , x out represent input and output feature maps respectively, C() and f 1×1 () represent splicing and convolution operations respectively, represents a 3x3 dilated deformable convolution, and d represents the dilation coefficient.

[0079] Most network feature fusion methods for multi-layer feature fusion use simple aggregation such as splicing and addition, without considering the different roles of low-level and high-level features in crack detection, and cannot effectively obtain valuable information related to cracks, resulting in waste of computing resources and image information. At the same time, due to the structure of the crack, a method is needed to capture its long-distance dependence relationship, strengthen the crack feature, and weaken the background information.

[0080] In view of the above problems, the present application proposes a deep and shallow layer attention feature fusion module DSAFF. DSAFF includes two branches, one is an attention branch, and the other is a shallow layer spatial information acquisition branch. The attention branch uses channel attention CA and BDAM module to obtain channel and spatial information. The CA module can obtain the importance of the feature channel, while the BDAM module can capture the long-distance dependence relationship of the crack by decoupling through horizontal and vertical pooling, and use the block and foreground-background decoupling operation to alleviate the influence of the background on the acquisition of the long-distance dependence relationship, so as to achieve the purpose of strengthening the crack feature and weakening the background information. The shallow layer spatial information acquisition branch is used to acquire shallow layer spatial information and input into the BDAM. The shallow layer spatial information has more spatial detail information, which can alleviate the information loss of the semantic weak crack in the down-sampling, so as to realize the attention to the area where the spatial information is lost in the down-sampling process.

[0081] As shown in Figure 4 , the deep and shallow layer attention feature first uses high-level features and low-level features to do simple splicing, and then uses channel attention CA (as shown in Figure 5 ) to obtain channel information. The shallow layer spatial information acquisition branch and the CA output feature are fused after processing, and input into the block decoupling attention mechanism BDAM (as shown in Figure 6 ) to obtain the spatial information of deep and shallow layer fusion. The shallow layer spatial information acquisition branch is a 3x3 maximum pooling with a step of 2, and then uses global maximum pooling to change the channel number to 1 to obtain shallow layer spatial information. The maximum pooling operation can retain the significant features of the shallow layer feature and supplement the deep layer feature. CA is compressed to Cx1x1 dimension by global average pooling, scaled by 1x1 convolution, and then uses Sigmoid activation to obtain the importance of the channel.

[0082] The process of the block decoupling attention mechanism is as follows:

[0083] First, the deep and shallow layer spatial information fusion is carried out, and the process is as follows.

[0084]

[0085] For the fused feature, For the cth channel of the spliced feature, C is the number of channels, F q For shallow features.

[0086] Since the feature map integrates the importance coefficient of the channel, and then a global average pooling at the channel level is performed, if a transverse or longitudinal pooling is performed again, the feature value is very small, all near 0. After Sigmoid activation, they are all near 0.5, the difference is very small, and the difference between the damage area and the background area cannot be reflected. In view of this problem, the present application proposes MN (Max Normalizing), which is processed by MN, and its formula is as follows.

[0087] F R =y*F r / (max(abs(F r )))

[0088] Where max() represents the maximum value in F r , abs() represents the absolute value of all data in F r , and y is a self-defined value. MN can map all values in F r to the value range of [-y, y], while preserving the original size relationship, and the value used in the present application is 1.

[0089] Considering the elongated feature of the crack, the present application decouples the longitudinal and transverse pooling to capture the long-distance dependence of the crack. At the same time, since the transverse and longitudinal pooling of the feature map that is too large will lower the overall average of some background areas, the attention based on the transverse and longitudinal directions will be affected. In view of this problem, the present application divides F R into several regions, and the present application takes 4x4, and then performs the above operation on each region to slow down the influence. The process is shown in the following formula.

[0090]

[0091]

[0092] Where F′ R,P , F″ R,P , F R,P represent the output features of the transverse and longitudinal pooling of the Pth patch and F R , respectively, and c, i, j represent the first dimension, row and column, respectively.

[0093] After transverse and longitudinal activation, the attention map will activate some irrelevant regions. For example, in a certain patch, there is a transverse crack or a longitudinal crack. Then the activation value of the row or column will be larger. However, the row or column contains not only the crack area, but also the normal road surface area or other background area, which should not be highly focused. In order to alleviate this situation, the present application decouples the high-frequency area and the low-frequency area of the feature, as shown in the following formula.

[0094] f low,H (i,j)=min(f patch (i,j),F′ R,P (i,j))

[0095] f low,V (i,j)=min(f patch (i,j),F″ R,P (i,j))

[0096] f high,H (i,j)=max(f patch (i,j),F′ R,P (i,j))

[0097] f high,V (i,j)=max(f patch (i,j),F″ R,P (i,j))

[0098] where f high,H , f high,V , f low,H , f low,V respectively represent decoupled high and low frequency transverse and longitudinal pooling feature maps, f patch represents the region feature map, F′ R,P , F″ R,P are the same as before. min represents taking the small value of the corresponding position, and max is taking the large value.

[0099] Finally, the feature is activated by Sigmoid, as shown in the following formula.

[0100]

[0101] where f i,j p represents the value of the i-th row and the j-th column of the p-th patch, represents the value of the i-th row and the j-th column of f high,H after Sigmoid, Similarly, the above formula is f i,j p greater than and greater than , F′ R,P the value of the first row and the jth column. If the former is less than the latter, the corresponding becomes The latter is the same.

[0102] As Figure 7 shown, in order to further fit the morphological characteristics of the crack and further refine the characteristics, the present application proposes a feature refinement (FR) module to further enhance the acquisition of features.

[0103] The specific process is as follows. First, two 1x1 convolutions are used to adjust the dimensions, so that the number of channels is adjusted to C out / 2, reducing the amount of calculation, and forming two branches. Considering the factor that cracks are mostly linear structures, the present application replaces the common 3x3 convolution with a cascaded 3x1 and 1x3 convolution, the output dimension of the first 3x1 convolution is C out , and the output dimension of the following 1x3 convolution is C out / 2. On the one hand, the extracted features are more suitable for linear structures, and on the other hand, the amount of calculation is further reduced, improving the detection speed of the model. One of the branches is subjected to two cascaded 3x1 and 1x3 convolutions, the features obtained in this process are spliced with the features of the initial input of the other branch, and then a 1x1 convolution is used to fuse the features to form the final refined features.

[0104] S3: In order to make the positive and negative sample distribution more reasonable during training, the present application introduces a SimOTA label distribution scheme to achieve better performance.

[0105] YOLOv5s adopts a static label distribution scheme, which sets a fixed positive and negative sample distribution scheme for all targets. This will cause some targets to be assigned many low-quality labels, and for some other targets, there are too few positive samples, so it is not reasonable. In order to make the positive and negative sample distribution more reasonable during training, the present application introduces a SimOTA label distribution scheme to achieve better performance.

[0106] The step-by-step process of SimOTA is as follows. First, determine the positive sample candidate region. Specifically, the present application defines the sample points within the 5*5 range of the center point of the real box as the positive sample candidate region. Second, since the anchor box mechanism of YOLOv5 assigns 3 anchor boxes to each sample point. Therefore, for each real box, calculate the sum of the classloss, iouloss and non-candidate region loss ~ IsInBoxCenter of each anchor box as the cost, and the specific cost formula is as follows:

[0107] Cost=ClsLoss CE +α*RegLossIOU + β * ( ~ IsInBoxCenter )

[0108]

[0109] RegLoss IOU = -log((p∩t) / (p∪t))

[0110] where α, β are weight coefficients, M represents the number of categories, y ic represents the sign function (0 or 1, 1 when the real category of sample i is c, otherwise 0), p ic represents the probability that sample i belongs to category c, p represents the predicted box, t represents the real box, ClsLoss CE represents the cross-entropy classification loss, RegLoss IOU represents the IOU regression loss.

[0111] Then the cost of each real box is selected for the 10 anchor boxes, and the sum of the IOU of the 10 anchor boxes is calculated to get the dynamic k value of each real box. Finally, the first k anchor boxes with the minimum cost are taken as positive samples for each real box, and the rest are negative samples, and the loss is calculated for training. The loss function is shown in equation (13). Respectively by ClsLoss CE , RegLoss IOU and ObjLoss BCE three loss components, λ, α, β are weight coefficients.

[0112] Cost = λ * ClsLoss CE + α * RegLoss IOU + β * ObjLoss BCE

[0113] where ClsLoss CE is the same as above, the regression loss RegLoss IOU is as follows.

[0114] RegLoss IOU = 1-(p∩t) / (p∪t)

[0115] confidence loss ObjLoss BCE is as follows.

[0116] ObjLoss BCE = L1(box_p,box_t)

[0117]

[0118] Wherein box_p represents the position prediction output by the model, and box_t represents the position label in the data set.

[0119] S4.1: Set the training parameters, and the model adopts the SGD optimizer. The batch size is set to 16, the momentum parameter momentum is 0.937, and the initial learning rate is 0.01.

[0120] S4.2: Put the road crack images in the training set in the data set into the multi-scale road crack detection model based on deep and shallow layer attention feature fusion for training.

[0121] S5: According to the multi-scale road crack detection model based on deep and shallow layer attention feature fusion trained, the performance of the model is evaluated from the detection F1-Score, mAP, test time and FLOPs indicators. F1-Score is a comprehensive evaluation indicator of Precision and Recall, mAP is the average correct rate (AP) of all class detection, test time indicates the speed of the model during testing, and FLOPs indicates the number of floating point operations of the model, which is used to measure the complexity of the model. In the target detection task, the evaluation indicators are defined as follows:

[0122]

[0123]

[0124] Wherein P represents the precision, and R represents the recall. mAP is the average value of the integral of all class P-R curves, wherein N represents the number of classes, P n is the average precision of the sample class.

[0125] The experimental platform of the application is set as follows: the CPU is Inter Core i9-10900K, the memory is 64GB, the GPU is NVIDIA GeForce RTX P4000, the display memory is 8GB, the operating system is Ubuntu 18.04, and the deep learning framework is Pytorch.

[0126] The above only expresses the preferred embodiments of the application, which are described in more detail and in more detail, but cannot be understood as limiting the scope of the patent of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications, improvements and substitutions can be made, which all belong to the protection scope of the application. Therefore, the protection scope of the patent of the application should be subject to the appended claims.

Claims

1. A multi-scale pavement crack detection method based on the fusion of deep and shallow attention features, characterized in that: Includes the following steps: S1, Acquire images of road surface cracks and preprocess the images; S2, input the image into the backbone network to extract hierarchical features P2, P3, P4, P5; S3, input P5 into the scale-adaptive SA module to obtain multi-scale information; S4, input P2, P3, P4 into the Neck to refine the features; the Neck includes a deep and shallow attention feature fusion DSAFF module and a feature refinement FR module; the DSAFF module uses channel attention to obtain channel information, and uses a block decoupled attention BDAM module to obtain spatial information of deep and shallow fusion, and FR is used to refine edge information; S5, input the results into the detection head to obtain the final detection results; S6 introduces SimOTA to optimize the label allocation scheme during training; In step S4, the deep and shallow attention feature fusion DSAFF module includes: The DSAFF module includes two branches: an attention branch and a shallow spatial information acquisition branch. The attention branch uses the channel attention CA module and the BDAM module to acquire channel and spatial information. The DSAFF module first uses high-level features and low-level features to perform a simple concatenation, and then uses channel attention (CA) to obtain channel information. The shallow spatial information acquisition branch and the features output from the CA module are processed and fused, then input into the BDAM module to obtain the spatial information of the deep-shallow fusion. The shallow spatial information acquisition branch performs downsampling with a 3×3 max pooling with a stride of 2, and then uses global max pooling to reduce the number of channels to 1 to obtain shallow spatial information. The CA module compresses the feature map to a C×1×1 dimension through global average pooling, scales it using 1×1 convolution, and then uses Sigmoid activation to obtain the importance of the channels. The BDAM module first performs deep and shallow spatial information fusion, as shown in the following equation; ； As a feature of fusion, This represents the c-th channel of the splicing feature, where C is the number of channels. These are shallow features; Perform MN processing, as shown in the formula below; ； Where max() represents taking The maximum value in the range, abs() represents Take the absolute value of all data in the table, where y is a user-defined numerical value; Will Divide the data into several regions, and then perform horizontal and vertical pooling operations on each region. The process is shown in the formula below. ； ； in , , Represent the output features of the P-th patch after horizontal and vertical pooling, respectively. c, i, and j represent the nth dimension, row, and column, respectively. , ; Decouple the high-frequency and low-frequency regions as shown in the following equation; ； ； ； ； in , , , These represent the high- and low-frequency horizontal and vertical pooling feature maps of decoupled systems, respectively. This represents the feature map of the region, where min indicates taking the minimum value at the corresponding location, and max indicates taking the maximum value. The feature is activated by Sigmoid, as shown in the following equation; ； in This represents the value in the i-th row and j-th column of the p-th patch. express The value in the i-th row and j-th column after the Sigmoid function. Similarly, the above formula is Greater than And greater than In this situation, express The value in the first row and j-th column; if the former is less than, then the corresponding... Become The latter is similar; In step S4, the feature refinement FR module includes: First, the dimensions are adjusted using two 1×1 convolutions, so that the number of channels is adjusted to... This forms two branches; instead of the common 3×3 convolution, a cascaded 3×1 and 1×3 convolution is used, with the output dimension of the first 3×1 convolution being... The output dimension of the subsequent 1×3 convolution is One branch undergoes two cascaded 3×1 and 1×3 convolutions, and the features obtained in this process are concatenated with the initial input features of the other branch. Then, a 1×1 convolution is used to fuse the features to form the final refined features.

2. The multi-scale pavement crack detection method based on deep and shallow layer attention feature fusion according to claim 1, characterized in that: Step S3 includes: The SA module differs from the parallel max pooling operation used in SPP by choosing to use cascaded dilated deformable convolutions to obtain larger receptive field information, and the dilation coefficient combination used is (3,6,12); at the same time, it adopts skip connection operation. First, add a 1×1 convolution to reduce the dimension of the input feature map to... , to obtain feature map Then, it is input into a 3×3 dilated deformable convolution with an expansion coefficient of 3, and the result is the same as... Adding them together yields the feature map. The dilation coefficients of the 3×3 dilated deformable convolution were changed to 6 and 12 respectively to obtain the feature maps. and Finally, After concatenation, the data is fed into a 1×1 convolution to obtain the output feature map. The calculation is shown in the following formula; ； ； in , These represent the input and output feature maps, respectively. and These represent the splicing and convolution operations, respectively. This represents a dilated deformable convolution with a kernel size of 3×3, where d represents the dilation coefficient.

3. The multi-scale pavement crack detection method based on deep and shallow layer attention feature fusion according to claim 1, characterized in that: In step S6, the SimOTA process is as follows: First, positive sample candidate regions are determined by defining sample points within a 5x5 area of ​​the center of the ground truth bounding box. Second, for each ground truth bounding box, the class loss, IoU loss, and non-candidate region loss for each anchor box are calculated. The sum is taken as the cost, and the specific cost formula is as follows: ； in , These are weighting coefficients; then, for each ground truth box, the 10 anchor boxes with the lowest cost are selected, and the sum of the IOU with these 10 anchor boxes is calculated to obtain the dynamic k value for each ground truth box; finally, for each ground truth box, the top k anchor boxes with the lowest cost ranking are used as positive samples, and the rest are negative samples, and the loss is calculated for training. The loss function during training is shown in the following formula; respectively by , and It consists of three losses. , , These are weighting coefficients; 。

Images