Road crack detection method and device based on cross-modal feature fusion and medium

Abstract

The present application relates to a kind of road crack detection method, device and medium based on cross-modal feature fusion, method includes the following steps: obtaining the color feature and depth feature of road crack data, color depth feature is obtained by fusing color feature and depth feature, color feature, depth feature and color depth feature are input into the detection network based on cross-modal three-flow feature fusion network for detection;Wherein, cross-modal three-flow feature fusion network includes cross-modal interactive fusion module and global interactive module, color feature extraction module and depth feature extraction module extract the color feature and depth feature of road crack data respectively, color feature, depth feature and color depth feature are fused by improved cross-modal interactive fusion module, form comprehensive feature, again by global interactive module for splicing, obtain splicing result, and detection network obtains detection result based on splicing result.Compared with prior art, the present application has the advantages of high precision, high definition and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road detection, in particular to a road crack detection method and device based on cross-modal feature fusion and a medium. BACKGROUND

[0002] Traditional road detection methods are mainly manual, which has problems such as dangerous working environment, low detection efficiency and excessive dependence on subjective experience of people, and it is difficult to guarantee the comprehensiveness and accuracy of the results. The intelligentization of road detection is an inevitable trend of its development.

[0003] Road crack detection can be regarded as an image segmentation problem in the field of computer vision. In the prior art, the image segmentation method based on deep learning has achieved more results. The semantic segmentation method can obtain the information that the image itself needs to express according to the texture, scene and other high-level semantic features of the image, and can segment the shape of the crack at the pixel level. The asphalt pavement crack segmentation algorithm based on the improved K-means algorithm can effectively solve the problems of low accuracy, mismatch between segmentation results and original images and other problems of existing asphalt pavement crack segmentation algorithms. The method of generating a synthetic image to enhance the structure of the crack segmentation can cope with the irregularity and diversity of the crack, as well as the changes of lighting conditions and background. Adding a residual module to the U-Net convolutional layer can improve the network and solve the optimization problem of unbalanced positive and negative samples in the crack segmentation process.

[0004] The above crack detection research has achieved good results, but most of the methods extract single feature information, which cannot effectively extract the detail information and spatial information of the crack, thereby affecting its actual application. Therefore, a detection method capable of extracting multiple features to reflect the detail information and spatial information of the crack is necessary. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a road crack detection method, device and medium based on cross-modal feature fusion.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A road crack detection method based on cross-modal feature fusion, comprising the following steps:

[0008] Obtaining color features and depth features of road crack data, fusing the color features and the depth features to obtain color depth features, inputting the color features, the depth features and the color depth features into a detection network based on a cross-modal three-flow feature fusion network for detection to obtain a detection result;

[0009] The cross-modal three-flow feature fusion network comprises a cross-modal interaction fusion module and a global interaction module, a color feature extraction module and a depth feature extraction module extract color features and depth features of the road crack data respectively, the color features, the depth features and the color-depth features are fused through the improved cross-modal interaction fusion module to form comprehensive features, and the global interaction module is used for splicing to obtain a splicing result, and a detection network obtains a detection result based on the splicing result.

[0010] Further, the cross-modal interaction fusion module comprises: performing convolution and convolution block attention mechanism processing on the color features and the depth features respectively, performing element-wise multiplication on the processed color features and the processed depth features, and then splicing the multiplication result, the color features and the color-depth features to form comprehensive features.

[0011] Further, the global interaction module comprises an average pooling path, a first adaptive pooling path, a second adaptive pooling path and an original feature path, each path adopts parallel input, is respectively passed through global attention and convolution to obtain features of different scales, and the features of different scales are spliced and fused to obtain an output of the global interaction module.

[0012] Further, the loss function used by the detection network comprises a structural similarity loss function, a local depth smoothness loss function, an edge sharpening loss function and a comprehensive loss function, the structural similarity loss function, the local depth smoothness loss function and the edge sharpening loss function are used for supervising the color-depth features, and the comprehensive loss function is used for supervising the color features and the depth features.

[0013] Further, the calculation expression of the structural similarity loss function is as follows:

[0014]

[0015]

[0016]

[0017] In the formula, l MM-SSIM-L1 is a structural similarity loss function value, M indicates different scales, μ p and μ g indicate the mean of the predicted image and the real image respectively, σ p and σ g indicate the standard deviation between the predicted image and the real image, β m and γ m indicate the relative importance between the two terms, c1 and c2 are constants, f(x i ) is a predicted value, and y iwherein, l is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant.

[0018] Further, the calculation expression of the local depth smoothness loss function is as follows:

[0019]

[0020]

[0021]

[0022] wherein, l is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. loc-smoothing is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. po is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. represents a Sobel vector in the horizontal direction, represents a Sobel vector in the vertical direction, is a pixel adjacent to p, and cosine is a cosine similarity.

[0023] Further, the calculation expression of the edge sharpening loss function is as follows:

[0024]

[0025]

[0026] wherein, l is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. wtv is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. p0 is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. represents a Sobel vector in the horizontal direction, represents a Sobel vector in the vertical direction, is a pixel adjacent to p.

[0027] Further, the expression of the comprehensive loss function is as follows:

[0028]

[0029]

[0030]

[0031] l RCS2 is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. Wiou is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. Wbce is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. Mae

[0032] wherein, l is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. Wiou is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. Wbce is the local depth smoothness loss value, D is the depth feature, G is the ground truth, n is the predicted total amount, and a is a constant. MaeFor the average absolute error, n The number of object boxes is represented by b, the coordinates of the i-th object box are represented by b, the coordinates of the real labeled box of the i-th object are represented by g, IOU(b i ,g i ) represents the intersection over union between the i-th object box and the real labeled box, w represents the weight value, y is the real label vector, z is the predicted label vector, c is the total number of training sets, f(x i ) is the predicted value, y i is the real value.

[0033] In a second aspect of the present application, a road crack detection device based on cross-modal feature fusion includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements any one of the above road crack detection methods based on cross-modal feature fusion.

[0034] In a third aspect of the present application, a storage medium has a program stored thereon, which, when executed, implements any one of the above road crack detection methods based on cross-modal feature fusion.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] 1) The present application is based on color features, depth features and color-depth features obtained by fusion, and a cross-modal three-stream feature fusion network is designed. The features are fused through a cross-modal interactive fusion module, which enhances the model's perception of feature channels and spatial positions. Then, through a global interaction module, different scale features are integrated to reduce information dispersion. Through the combination of multiple loss functions, the structural integrity and boundary quality of the predicted cracks are measured, thereby obtaining clearer segmentation results with high detection accuracy.

[0037] 2) The present application uses multiple loss functions to comprehensively evaluate the image detection results from the aspects of image brightness, color, continuity, spatial consistency, edge sharpening, noise, boundary details and classification accuracy, etc., to obtain crack detection results with high definition and clear boundaries, improving the accuracy of detection. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The network structure diagram of the present application is shown in the figure;

[0039] Figure 2 The structure diagram of the cross-modal interactive fusion module is shown in the figure;

[0040] Figure 3 The overall structure diagram of CBAM is shown in the figure;

[0041] Figure 4 The structure diagram of the global interaction module is shown in the figure;

[0042] Figure 5 Figure 1 is a structural diagram of a global attention mechanism;

[0043] Figure 6 Figure 2 is a diagram of various types of crack examples for an encoder;

[0044] Figure 7 Figure 3 is a comparison diagram of detection results. DETAILED DESCRIPTION

[0045] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present application, and gives a detailed implementation and specific operation process, but the protection scope of the present application is not limited to the following embodiments.

[0046] Embodiment 1

[0047] The present application is a road crack detection method, device and medium based on cross-modal feature fusion, comprising the following steps:

[0048] Obtain color features and depth features of road crack data, fuse the color features and the depth features to obtain color depth features, input the color features, the depth features and the color depth features into a detection network based on a cross-modal three-flow feature fusion network for detection to obtain a detection result;

[0049] The cross-modal three-flow feature fusion network comprises a cross-modal interactive fusion module and a global interactive module, a color feature extraction module and a depth feature extraction module extract color features and depth features of road crack data respectively, the color features, the depth features and the color depth features are fused through the improved cross-modal interactive fusion module to form comprehensive features, and then the global interactive module is used for splicing to obtain a splicing result, and the detection network obtains the detection result based on the splicing result.

[0050] 1. Cross-modal three-flow feature fusion network architecture

[0051] 1.1 Overall architecture

[0052] Different modalities of input data have their own characteristics. RGB images present the color and texture details of the scene through color features, while depth images focus on capturing the details of the spatial structure through depth features. RGB-D combines these two characteristics and provides a four-dimensional color-depth feature that integrates the spatial features and texture information of the data. In addition, the quality difference of depth images can affect model training, and low-quality depth images can lead to reduced training effectiveness. To solve this problem, the present application designs a cross-modal three-flow feature fusion network architecture to promote the interaction and fusion between different modal feature, thereby reducing the adverse effects of low-quality depth images on model training. Although the model can accurately identify the target, it often ignores details such as spatial position and texture, so it is necessary to combine high-level features with low-level features to improve recognition accuracy.

[0053] As shown in Figure 1 , the cross-modal three-flow feature fusion network is designed based on the RC-SegNet model using an encoder-decoder structure. The backbone network uses ResNet101, which retains different scale information through five times of skip connection, and the feature dimension of each skip connection is 64, 256, 512, 1024, and 2048, respectively. During the encoding process, the RGB, RGB-D, and Depth features are fused through the cross-modal interaction fusion module CIM for each path, enhancing the model's perception of feature channels and spatial positions, and better capturing the structure and shape of the scene. In the decoding stage, the features with the same channel and corresponding modalities after the global interaction module GIM are spliced, allowing the model to better retain the details and context information of the crack. To supervise the training effect of the RC-SegNet model, the RCS loss function is used to measure the boundary quality and structural difference between the model's crack segmentation and the actual crack, guiding the model's segmentation results to be more complete and clear.

[0054] 1.2 Cross-modal interaction fusion module CIM

[0055] To address the problem of feature redundancy and insufficient representation information in the cross-modal feature fusion process, the present application combines channel attention mechanism and spatial attention mechanism to design the cross-modal interaction fusion module CIM. As shown in Figure 2 , the depth features and RGB features are respectively processed through the convolution module and the convolution block attention mechanism (CBAM) (as shown in Figure 3The RGB features and the depth features are fused after the output of the Sigmoid function, specifically, the RGB features and the depth features after the Sigmoid processing are multiplied element by element, and then the product, the RGB features and the RGBD features are spliced to form a comprehensive feature representation.

[0056] The channel attention module in CBAM extracts different dimensional features extracted by the backbone network, respectively through MaxPool and AvgPool to obtain two attention feature maps, and the features are sent into a shared multi-layer perception to generate a channel feature map, and finally through a Sigmoid layer and the input to obtain a channel attention feature map M c (F), as shown in Figure 3 The calculation expression of the channel attention feature map M c (F) is:

[0057] M c (F) = Sigmoid(MLP(AvgPool(F)) + MLP(MaxPool(F)))

[0058] In the formula, F is the input feature map, MLP is the full connection layer, AvgPool and MaxPool are the global average value pooling and the global maximum value pooling respectively, and the activation function is Sigmoid.

[0059] In the channel dimension of M c (F), average pooling and maximum pooling are performed, respectively representing the spatial distribution of different statistical information. Then the feature maps of different spatial information are spliced, and a 7x7 convolution operation is used on the spliced feature map to capture the spatial context information to generate the final spatial attention feature map, and finally through a Sigmoid layer, the spatial attention feature map M s (F) is obtained.

[0060] M s (M c (F)) = Sigmoid(f 7×7 ([AvgPool(M c (F)); MaxPool(M c (F))])

[0061] In the formula, M c (F) is the channel attention feature map, f 7x7 represents a convolution operation with a convolution kernel size of 7, AvgPool and MaxPool are the global maximum value pooling and the global average value respectively, and the activation function uses Sigmoid.

[0062] M c (F) and M s(F) is multiplied by the original input feature map to produce the final feature map F".

[0063]

[0064]

[0065] where F is the original input feature, M c (F) is the channel attention feature map, M s (F') is the spatial attention feature map, Table element level multiplication.

[0066] 1.3 Global Interaction Module GIM

[0067] The global interaction module is used to capture the global information of the features, reduce the degree of information diffusion in different scale features when extracting the features, and enhance the understanding of the overall structure of the image.

[0068] The global interaction module includes an average pooling path, a first adaptive pooling path, a second adaptive pooling path and an original feature path, each path adopts parallel input, respectively passes through global attention and convolution to obtain features of different scales, and the features of different scales are spliced and fused to obtain the output of the global interaction module.

[0069] The global interaction module adds a global attention mechanism in different scales of the global context module, so as to reduce the information diffusion in the feature extraction process of the model, and further improve the global interaction ability of the model in the up-sampling process. Specifically, the present application adopts 3*3 average pooling, 5*5 adaptive pooling, 7*7 adaptive pooling, and four paths of original features respectively pass through the global attention module and convolution to splice and fuse features of different scales, so as to realize the improvement of the global interaction ability of the model, as shown in Figure 4 .

[0070] The GAM introduces 3D-permutation and the channel attention and convolution spatial attention submodules of the multilayer perceptron. The Channel Attention submodule uses 3D-permutation to preserve information in three dimensions. Then a two-layer multilayer perceptron is used to amplify the cross-dimensional channel-space dependency. Spatial Attention uses two convolution layers for spatial information fusion. While removing the max-pooling operation, ChannelShuffle with Group convolution is introduced to reduce the computational amount, as shown in Figure 5 .

[0071] 1.4 Loss function

[0072] The loss function of the present invention is improved on the basis of PopNet loss function, and the combination of l RCS1 and l RCS2 The overall structure and contour features of the RC-SegNet network are measured.

[0073] For the network with RGB-D splicing as input, the MS-SSIM-L1 structural similarity loss function is used to supervise the structure, texture, brightness and other features in feature extraction. This loss function combines the L1 loss function on the basis of the MM-SSIM loss function. The MM-SSIM loss function uses a structural similarity metric at multiple scales when calculating image similarity. By comparing the structure, texture, and brightness of images at different scales, MM-SSIM can capture the similarity of images more comprehensively. MS-SSIM is prone to brightness changes and color deviations, but it can retain high-frequency information (edges and details of the image), while l L1 The loss function can better keep the brightness and color unchanged. The final MS-SSIM-L1 structural similarity loss function is:

[0074]

[0075]

[0076]

[0077] Where, l MM-SSIM-L1 is the structural similarity loss function value, M Indicates different scales, μ p 、μ g Represents the mean of the predicted image and the real image, the standard deviation between the predicted image and the real image, σ p , σ g represents the covariance between the predicted image and the real image, β m , γ m Indicates the relative importance between two items, c1 and c2 are constants, f(x i ) is the predicted value, y i is the true value, n is the predicted total amount, α is a constant, which is set to 0.84 in this embodiment, and G is the Gaussian distribution parameter.

[0078] In the feature extraction task of RGB-D, the depth image is the image corresponding to the RGB image, which contains the depth information of each pixel. However, due to the interference of factors such as sensor noise, occlusion or depth inconsistency, the depth image may contain discontinuous, noisy or inaccurate depth values. The goal of the local depth smoothing loss function is to make the depth image more smooth and continuous by penalizing the discontinuity and noise in the depth image. By optimizing this loss function, the depth image generated by the network can have better spatial consistency and accuracy. Thus, l loc-smoothing The target area is kept smooth, and the boundary pixels are clear. The calculation expression of the local depth smoothing loss function is:

[0079]

[0080]

[0081]

[0082] In the formula, D po is the depth feature extracted in the RC-SegNet model, G is the true value, p is a pixel representing the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is the pixel adjacent to p, cosine is the cosine similarity, l loc-smoothing is the value of the local depth smoothing loss function.

[0083] In addition to the MS-SSIM-L1 loss function and the local depth smoothing loss function, an edge sharpening loss function based on Sobel is also used to calculate the edge intensity difference between the generated image and the real image. By minimizing the edge intensity difference, the model can generate clearer and sharper edges. The calculation expression of the edge sharpening loss function is:

[0084]

[0085]

[0086] where D p0 is the depth feature extracted in the RC-SegNet model, p is a pixel representing the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is the pixel adjacent to p, and γ is equal to 0.5.

[0087] The three loss functions are finally combined with weights to supervise the RC-SegNet network taking RGB-D as input (stitching). The calculation expression of the weight is:

[0088] l RCS1 =l MS-SSIM-L1 +λ1·l loc-smoothing +λ2·l wtv

[0089] wherein l RCS1 is the weighted loss function value, λ1 is the weight coefficient of the local depth smoothing loss function, and λ2 is the weight coefficient of the edge sharpening loss function.

[0090] For the model taking RGB and depth map as input, the Wiou Loss, Mae Loss and Wbce loss are combined to obtain a comprehensive loss function to measure the RC-SegNet. Specifically, the comprehensive loss function is:

[0091] The Wiou Loss value l Wiou is introduced to measure the overlap between the predicted target and the real target, and then the minimum weight of each part in the predicted target and the real target is used for weighting to accurately describe the contribution of each crack and measure the similarity between the predicted two. The calculation expression is:

[0092]

[0093] wherein n represents the number of object boxes, b represents the coordinates of the i-th object box, g represents the coordinates of the real annotation box of the i-th object, IOU(b i ,g i ) represents the intersection over union between the i-th object box and the real annotation box, and w represents the weight value.

[0094] The binary cross-entropy loss function Wbce loss measures the difference between the probability distribution predicted by the model and the real label. The calculation expression is:

[0095]

[0096] wherein l Wbce is the binary cross-entropy loss function value, y is the real label vector, z is the predicted label vector, w is the weight value, and c is the total number of the training set.

[0097] The mean absolute error is the average of the absolute difference between all predicted values and real values. The mean absolute error loss function Mae Loss can preserve the boundary details of the image in the semantic segmentation task. The calculation expression of the mean absolute error loss function is:

[0098]

[0099] In the formula, l Mae is the average absolute error loss function value, n is the total number of samples, f(x i ) is the predicted value, y i is the true value.

[0100] 2. Experimental process of the present application

[0101] 2.1 Experimental data set

[0102] The road crack data set used in the present application is derived from real data independently shot, collected by a road inspection vehicle with a camera and a laser emitter. Data collection was carried out on different highways, and a total of 4683 crack RGB images and depth maps were collected. The data was processed according to the data format and process of NJU2K using Labelme. The number of training set and test set in the pavement crack image data set is 3752 and 931 respectively.

[0103] The statistics of each crack category are shown in Table 1.

[0104] Table 1 Statistics of each crack category

[0105] Type Picture number Label number Horizontal crack 1914 8611 Vertical crack 1843 8142 Mesh crack 926 4307 Total 4683 21059

[0106] As can be seen from the table, the covered crack types are relatively comprehensive, and the collected crack examples are shown in Figure 6 The number of vertical cracks is 1843, the number of horizontal cracks is 1914, the number of net cracks is 926, and a total of 4683 crack pictures. The number of horizontal cracks is 8611, the number of vertical cracks is 8142, the number of net cracks is 4307, and a total of 21059 crack labels.

[0107] 2.2 Experimental equipment and evaluation index

[0108] During training, the input image size is set to 256x256, the training period is 100, the batch size is 2, the initial learning rate is 0.0001, the learning rate decay is 0.1, and Adm is used as the optimization function of the model.

[0109] In the present application, MIoU, Mae, Smeasure, Max_F and E_measure are mainly used as evaluation indexes. MIoU is the ratio of the intersection and union of the true value and the predicted value, used to calculate the degree of coincidence between the predicted region and the true region. The definition is as follows:

[0110]

[0111] where k+1 is the number of classes, TP is the number of pixels that are both true and predicted as cracks, FN is the number of pixels that are true as cracks but predicted as background, and FP is the number of pixels that are predicted as cracks but true as background.

[0112] Mae is a statistical measure of the error between the predicted value and the true value, which is the average of the absolute error between all predicted values and corresponding true values.

[0113]

[0114] where, is the predicted value, y i is the true value, and n is the total number of predictions.

[0115] Structure-measure (Smeasure) is extended from the Structural Similarity Index (SSIM), which is an index for measuring the structural similarity between two images. Smeasure takes into account the classification information at the pixel level, not only comparing the similarity of pixel values, but also considering the structural relationship between pixels. It calculates the structural similarity matrix between the predicted segmentation map and the true segmentation map, and then takes the average to get the Smeasure value. The value of Smeasure ranges from 0 to 1, with 1 indicating a perfect match and 0 indicating no match. It is defined as follows:

[0116]

[0117]

[0118] S o = μ * O FG + (1- μ) * O BG

[0119] S m = α * S o + (1- α) * S r

[0120] where S r indicates the barycenter position, then cut into four regions, and calculate the weighted average of the SSIM of the four regions according to the area of the region as the weight of the pixels. α is a constant of 0.5, O FG is the similarity of the foreground, O BG is the similarity of the background, μ is the proportion of the foreground in the total pixels, x FG indicates the probability value of the predicted foreground region, indicates the average value of x FG , x BG indicates the probability value of the predicted foreground region, indicates the average value of x BGaverage value of S m is the Smeasure value.

[0121] Max_F is the maximum value of adaptive F-measure over the whole image, which represents the best overall performance of the segmentation result. It considers both precision and recall of the model. Precision represents the proportion of positive samples correctly classified by the model to the total number of samples predicted as positive by the model. It is defined as follows:

[0122]

[0123] Recall represents the proportion of positive samples correctly classified by the model to the total number of real positive samples. It is defined as follows:

[0124]

[0125] where TP represents the model correctly predicting positive examples as positive examples, FP represents the model incorrectly predicting negative examples as positive examples, and FN represents the model incorrectly predicting positive examples as negative examples.

[0126] Max_F can be calculated for each pair of P-R, and the maximum F β is usually taken as the evaluation index. β

[0127]

[0128] where β 2 is usually taken as 0.3.

[0129] E-measure is an evaluation index based on the enhanced alignment mapping between the saliency map and the real saliency map. It is defined as follows:

[0130]

[0131] where h and w are the height and width of the mapping respectively, and φ FM is used to evaluate the foreground map and noise.

[0132] 2.3 Comparison of different algorithms

[0133] ​The comparison results are shown in Table 2. To further verify the performance of the improved algorithm, the application studies compare the improved model RC-SegNet with other mainstream RGB-D segmentation algorithms PopNet, XMSNe, BBSNet, DMRA, SPNet, HiDANet, AFNet, RFNet, TBINet, HINet, etc. using the same data set. Smeasure, Mae, Max_F, E-measure and other evaluation indexes are used to compare the advantages and disadvantages of each model. As can be seen from Table 2, the indicators of RC-SegNet are in the first and second places. Compared with the PopNet network model, the improved model RC-SegNet effectively improves the model performance, and Miou, Smeasure, Max_F, E-measure are increased by 2.06%, 1.88%, 4.20%, 2.12% respectively, and Mae is reduced by 1.53%. At the same time, in order to verify the effectiveness of the model improvement, the cracks are compared on different models. The segmentation effect of the model of the application is better than that of the current mainstream segmentation network in terms of details and overall structure.

[0134] Table 2 comparison results of model indicators

[0135]

[0136] As Figure 7 shown, the application respectively segments mixed cracks (first row), vertical cracks (second row), horizontal cracks (third row) and network cracks (fourth row), and the segmentation results are better than those of the existing model in terms of details (texture, thickness, etc.) and overall structure.

[0137] 2.4 Ablation experiment

[0138] The application compares different loss functions. The experimental results are shown in Table 3, and Table 3 is a comparison of different loss functions. The Orgin Loss of PopNet is the original loss, and the SSIM in PopNet is changed to MS-SSIM-L1, which compares the structure, texture and brightness of the image at different scales. Miou, Smeasure, Max_F, E-measure are increased by 0.52%, 0.69%, 2.92%, 0.16% respectively, and Mae is reduced by 1.27%. RSC is to add the Mae loss function based on MS-SSIM-L1, which better preserves the boundary details of the image in the semantic segmentation task. Compared with the original loss, Miou, Smeasure, Max_F, E-measure are increased by 0.85%, 1.13%, 3.93%, 0.20% respectively, and Mae is reduced by 1.49%.

[0139] Table 3 comparison results of different loss functions

[0140] Loss Miou Smeasure Mae Max_F E-measure Orgin loss(PopNet) 0.7469 0.7733 0.0256 0.6711 0.9234 MS-SSIM-L1(PopNet) 0.7521 0.7802 0.0129 0.7003 0.9250 RSC(Invention) 0.7554 0.7846 0.0107 0.7104 0.9254

[0141] The present application compares different backbone networks, and the experimental results are shown in Table 4. Table 4 is a comparison of different backbone networks. PopNet+VGG11 uses VGG11 as the backbone network, and the Miou and Smeasure are 0.7124 and 0.7421, respectively. Compared with other backbone networks, ResNet101 achieves the best in five indicators, and the Miou and Smeasure can reach 0.7485 and 0.7765.

[0142] Table 4 Comparison results of different backbone networks

[0143]

[0144] To verify the effectiveness of each improved module, the present application trains on the road crack dataset and calculates the evaluation indicators. Compared with PopNet, the model PopNet+ResNet101 with ResNet101 as the model backbone network, the E-measure increases by 1.68%, and the Mae decreases by 1.17%, which shows that the structure can obtain deeper features. PopNet+GIM increases Max_F by 1.77% and reduces Mae by 1.42%, which shows that the module can retain more rich structure information in the upsampling process. PopNet+CIM with CIM as the feature fusion module, Max_F increases by 1.77% and Mae decreases by 1.42%, which shows that the features of different modalities are effectively refined in the feature fusion process. Compared with PopNet+CIM, PopNet+CIM+Loss increases Max_F by 1.52% and Miou by 0.44%, which shows that the loss function of the present application can better measure the pros and cons of segmentation. Based on this, the present application fuses each module, and the Miou, Smeasure, Max_F and E-measure of the original algorithm increase by 2.06%, 1.88%, 4.20% and 2.12%, respectively, and the Mae decreases by 1.53%. It can be seen that the RC-SegNet model can effectively segment the details and structure of the crack.

[0145] Table 5 Ablation experiment results

[0146] Module Miou Smeasure Mae Max_F E-measure PopNet 0.7469 0.7733 0.0256 0.6711 0.9234 PopNet+ResNet101 0.7485 0.7765 0.0139 0.6828 0.9402 PopNet+GIM 0.7496 0.7800 0.0114 0.6888 0.9270 PopNet+CIM 0.7534 0.7871 0.0141 0.6886 0.9366 PopNet+Loss 0.7554 0.7846 0.0107 0.7104 0.9254 PopNet+CIM+Loss 0.7578 0.7877 0.0115 0.7038 0.9390 RC-SegNet(Ours) 0.7675 0.7921 0.0103 0.7131 0.9446

[0147] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A road crack detection method based on cross-modal feature fusion, characterized in that, The method comprises the following steps: Color features and depth features of road crack data are acquired, the color features and the depth features are fused to obtain color depth features, the color features, the depth features and the color depth features are input into a detection network based on a cross-modal three-flow feature fusion network for detection to obtain a detection result; The cross-modal three-flow feature fusion network comprises a cross-modal interactive fusion module and a global interactive module, the color feature extraction module and the depth feature extraction module extract color features and depth features of road crack data respectively, the color features, the depth features and the color depth features are fused by the improved cross-modal interactive fusion module to form comprehensive features, and the global interactive module is used for splicing to obtain a splicing result, and the detection network obtains the detection result based on the splicing result; The loss function used by the detection network comprises a structural similarity loss function, a local depth smoothing loss function, an edge sharpening loss function and a comprehensive loss function, the structural similarity loss function, the local depth smoothing loss function and the edge sharpening loss function are used for supervising the color depth features, and the comprehensive loss function is used for supervising the color features and the depth features; The calculation expression of the structural similarity loss function is as follows: In the formula, is a structural similarity loss function value, represents different scales, , respectively represent the mean of the predicted image and the real image, the standard deviation between the predicted image and the real image, represents the covariance between the predicted image and the real image, , represents the relative importance between the two items, , is a constant, is a predicted value, is a real value, is a predicted total amount, is a constant, is a Gaussian distribution parameter; The calculation expression of the local depth smoothing loss function is as follows: wherein, is a local depth smoothness loss function value, is a depth feature, is a ground truth value, is a pixel within a crack region, represents a Sobel vector in a horizontal direction, represents a Sobel vector in a vertical direction, is a neighboring pixel, is a cosine similarity; The calculation expression of the edge sharpening loss function is as follows: wherein is an edge sharpening loss function value, is a deep feature, is a pixel within a crack region, represents a Sobel vector in a horizontal direction, represents a Sobel vector in a vertical direction, is a neighboring pixel; The expression of the comprehensive loss function is as follows: In the formula, is the overlapping component, is the binary cross-entropy loss, is the mean absolute error, represents the number of object boxes, represents the coordinates of the th object box, represents the coordinates of the th object's real label box, represents the intersection over union between the th object box and the real label box, represents the weight value, is the real label vector, is the predicted label vector, is the total number of training sets, is the predicted value, is the real value.

2. The road crack detection method based on cross-modal feature fusion according to claim 1, characterized in that, The cross-modal interactive fusion module comprises the following steps: the color features and the depth features are respectively subjected to convolution and convolution block attention mechanism processing, the processed color features and the depth features are subjected to element-by-element multiplication, and then the product result, the color features and the color depth features are spliced to form comprehensive features.

3. The road crack detection method based on cross-modal feature fusion according to claim 1, characterized in that, The global interactive module comprises an average pooling path, a first adaptive pooling path, a second adaptive pooling path and an original feature path, each path adopts parallel input, is respectively subjected to global attention and convolution to obtain features of different scales, and the features of different scales are spliced and fused to obtain the output of the global interactive module.

4. A road crack detection device based on cross-modal feature fusion, comprising a memory, a processor, and a program stored in the memory, characterized in that, The processor implements the road crack detection method based on cross-modal feature fusion when the program is executed.

5. A storage medium having stored thereon a program, characterized by The program is executed to implement the road crack detection method based on cross-modal feature fusion.

Images