Road crack detection method and device based on cross-modal feature fusion and medium

Abstract

The invention relates to a road crack detection method and device based on cross-modal feature fusion and a medium, and the method comprises the following steps: obtaining color features and depth features of road crack data, and fusing the color features and the depth features to obtain color and depth features; inputting the color features, the depth features and the color depth features into a detection network based on a cross-modal three-flow feature fusion network for detection; wherein the cross-modal three-flow feature fusion network comprises a cross-modal interaction fusion module and a global interaction module, and a color feature extraction module and a depth feature extraction module extract color features and depth features of road crack data respectively; the color features, the depth features and the color depth features are fused through the improved cross-modal interaction fusion module to form comprehensive features, then the comprehensive features are spliced through the global interaction module to obtain a splicing result, and the detection network obtains a detection result based on the splicing result. Compared with the prior art, the method has the advantages of high precision, high definition and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of road detection, and in particular to a road crack detection method, device and medium based on cross-modal feature fusion. Background Art

[0002] Traditional road surface inspection methods are mainly manual, with problems such as dangerous working environment, low inspection efficiency and over-reliance on human subjective experience. It is difficult to ensure the comprehensiveness and accuracy of the results. Intelligent road surface inspection is an inevitable trend in its development.

[0003] Road crack detection can be regarded as an image segmentation problem in the field of computer vision. In the prior art, image segmentation methods based on deep learning have achieved many results. Among them, the semantic segmentation method can obtain the information that the image itself needs to express based on the texture, scene and other high-level semantic features of the image, and can segment the morphology of the crack itself at the pixel level. The asphalt pavement crack segmentation algorithm based on the improved K-means algorithm can effectively address the problems of low accuracy and mismatch between segmentation results and original images in the current asphalt pavement crack segmentation algorithm. The method of generating synthetic image enhanced structural crack segmentation can cope with the irregularity and diversity of cracks, as well as changes in lighting conditions and background. Adding a residual module to the U-Net convolutional layer to improve the network can solve the optimization problem of imbalanced positive and negative samples in the crack segmentation process.

[0004] The above crack detection research has achieved good results, but most methods extract relatively simple feature information and cannot effectively extract the detail information and spatial information of the crack, thus affecting its practical application. Therefore, a detection method that can extract multiple features to reflect the detail information and spatial information of the crack is very necessary. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a road crack detection method, device and medium based on cross-modal feature fusion.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A road crack detection method based on cross-modal feature fusion includes the following steps:

[0008] Obtain color features and depth features of road crack data, fuse the color features and depth features to obtain color depth features, input the color features, depth features and color depth features into a detection network based on a cross-modal three-stream feature fusion network for detection, and obtain detection results;

[0009] The cross-modal three-stream feature fusion network includes a cross-modal interactive fusion module and a global interactive module. The color feature extraction module and the depth feature extraction module extract the color features and depth features of the road crack data respectively. The color features, depth features and color depth features are fused through the improved cross-modal interactive fusion module to form a comprehensive feature, which is then spliced ​​through the global interactive module to obtain the splicing result. The detection network obtains the detection result based on the splicing result.

[0010] Furthermore, the cross-modal interactive fusion module includes: performing convolution and convolution block attention mechanism on the color features and depth features respectively, multiplying the processed color features and depth features element by element, and then concatenating the product results, color features and color depth features to form comprehensive features.

[0011] Furthermore, the global interaction module includes an average pooling path, a first adaptive pooling path, a second adaptive pooling path and an original feature path. Each path adopts parallel input, and obtains features of different scales through global attention and convolution respectively. The features of different scales are spliced ​​and fused to obtain the output of the global interaction module.

[0012] Furthermore, the loss functions used by the detection network include structural similarity loss function, local depth smoothing loss function, edge sharpening loss function and comprehensive loss function. The structural similarity loss function, local depth smoothing loss function and edge sharpening loss function are used to supervise color depth features, and the comprehensive loss function is used to supervise color features and depth features.

[0013] Furthermore, the calculation expression of the structural similarity loss function is:

[0014]

[0015]

[0016]

[0017] In the formula, l MM-SSIM-L1 is the structural similarity loss function value, M Indicates different scales, μ p , μ g Represents the mean of the predicted image and the real image, the standard deviation between the predicted image and the real image, σ p , σ g represents the covariance between the predicted image and the real image, β m , γ m Indicates the relative importance between two items, c1 and c2 are constants, f(x i ) is the predicted value, y iis the true value, n is the total predicted value, α is a constant, and G is the Gaussian distribution parameter.

[0018] Furthermore, the calculation expression of the local depth smoothing loss function is:

[0019]

[0020]

[0021]

[0022] In the formula, l loc-smoothing is the local depth smoothing loss function value, D po is the depth feature, G is the true value, p is the pixel in the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is p adjacent pixels, and cosine is the cosine similarity.

[0023] Furthermore, the calculation expression of the edge sharpening loss function is:

[0024]

[0025]

[0026] In the formula, l wtv is the edge sharpening loss function value, D p0 is the depth feature, p is the pixel in the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is the number of adjacent pixels.

[0027] Furthermore, the expression of the comprehensive loss function is:

[0028]

[0029]

[0030]

[0031] l RCS2 = l Wiou +l Wbce +l Mae

[0032] In the formula, l Wiou is the overlap component, l Wbce is the binary cross entropy loss, l Maeis the mean absolute error, n represents the number of object boxes, b represents the coordinates of the i-th object box, g represents the coordinates of the real annotation box of the i-th object, and IOU (b i ,g i ) represents the intersection-over-combination ratio between the i-th object box and the true annotation box, w represents the weight value, y is the true label vector, z is the predicted label vector, c is the total number of training sets, f(x i ) is the predicted value, y i is the true value.

[0033] In a second aspect of the present invention, a road crack detection device based on cross-modal feature fusion includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements any of the above road crack detection methods based on cross-modal feature fusion.

[0034] A third aspect of the present invention is a storage medium having a program stored thereon, which, when executed, implements any of the above road crack detection methods based on cross-modal feature fusion.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) Based on color features, depth features and fused color depth features, the present invention designs a cross-modal three-stream feature fusion network, fuses features through a cross-modal interactive fusion module, and enhances the model's perception of feature channels and spatial positions. Then, through a global interactive module, features of different scales are integrated to reduce information diffusion. Through a combination of multiple loss functions, the structural integrity and boundary quality of the predicted cracks are measured, thereby obtaining clearer segmentation results with higher detection accuracy.

[0037] 2) The present invention adopts a variety of loss functions to comprehensively evaluate the image detection results from aspects such as image brightness, color, continuity, spatial consistency, edge sharpening, noise, boundary details and classification accuracy, so as to obtain crack detection results with high definition and clear boundaries, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a network structure diagram of the present invention;

[0039] Figure 2 This is the structural diagram of the cross-modal interaction fusion module;

[0040] Figure 3 This is the overall structure diagram of CBAM;

[0041] Figure 4 It is the structure diagram of the global interaction module;

[0042] Figure 5 This is the structure diagram of the global attention mechanism;

[0043] Figure 6 This is an example diagram of various types of cracks in the encoder;

[0044] Figure 7 The following is a comparison chart of the test results. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] Example 1

[0047] The present invention is a road crack detection method, device and medium based on cross-modal feature fusion, comprising the following steps:

[0048] Obtain color features and depth features of road crack data, fuse the color features and depth features to obtain color depth features, input the color features, depth features and color depth features into a detection network based on a cross-modal three-stream feature fusion network for detection, and obtain detection results;

[0049] The cross-modal three-stream feature fusion network includes a cross-modal interactive fusion module and a global interactive module. The color feature extraction module and the depth feature extraction module extract the color features and depth features of the road crack data respectively. The color features, depth features and color depth features are fused through the improved cross-modal interactive fusion module to form a comprehensive feature, which is then spliced ​​through the global interactive module to obtain the splicing result. The detection network obtains the detection result based on the splicing result.

[0050] 1. Cross-modal three-stream feature fusion network architecture

[0051] 1.1 Overall Architecture

[0052] Input data of different modalities have their own characteristics. RGB images present the color and texture details of the scene through color features, while depth images Depth focus on capturing the detailed information of the spatial structure through depth features. RGB-D combines these two characteristics to provide a four-dimensional color depth feature that integrates the spatial features and texture information of the data. In addition, the quality difference of depth images will affect model training, and low-quality depth images may lead to reduced training results. In order to solve this problem, the present invention designs a cross-modal three-stream feature fusion network architecture, which aims to promote the interactive fusion of features of different modalities, thereby reducing the adverse effects of low-quality depth images on model training. Although the model can accurately identify the target, it often ignores details such as spatial position and texture. Therefore, it is necessary to combine high-level features with low-level features to improve recognition accuracy.

[0053] like Figure 1 As shown in the figure, the cross-modal three-stream feature fusion network is designed based on the RC-SegNet model using the encoder-decoder structure. Its backbone network uses ResNet101, which retains information of different scales in five jump connections. The feature dimensions of each jump connection are 64, 256, 512, 1024, and 2048 respectively. In the encoding process, the cross-modal interactive fusion module CIM is used to fuse the three features of RGB, RGB-D, and Depth for each path, enhancing the model's perception of feature channels and spatial positions, thereby better capturing the structure and shape of the scene. In the decoding stage, the features with the same channels and corresponding modalities in the downsampling process are spliced ​​after passing through the global interactive module GIM, so that the model can better retain the details and context information of the cracks. In order to supervise the training effect of the RC-SegNet model, the RCS loss function is used to measure the boundary quality and structural differences between the model segmented cracks and the actual cracks, thereby guiding the model's segmentation results to be more complete and clearer.

[0054] 1.2 Cross-modal interaction fusion module CIM

[0055] In order to solve the problem of feature redundancy and insufficient representation information in the cross-modal feature fusion process, the present invention combines the channel attention mechanism and the spatial attention mechanism to design a cross-modal interactive fusion module CIM. Figure 2 As shown, the depth features and RGB features are respectively processed by the convolution module and the convolutional block attention mechanism (CBAM) (as shown in Figure 3After the output of the Sigmoid function, the RGB features and the depth features are fused. Specifically, the RGB features after Sigmoid processing are multiplied element by element with the depth features, and then the product results, RGB features and RGBD features are concatenated to form a comprehensive feature representation.

[0056] The channel attention module in CBAM obtains two attention feature maps through MaxPool and AvgPool for the different dimensional features extracted by the backbone network, and sends the features to a shared multi-layer perceptron to generate a channel feature map. Finally, the channel attention feature map M is obtained through the Sigmoid layer and the input. c (F) Figure 3 As shown, the channel attention feature map M c The calculation expression of (F) is:

[0057] M c (F)=Sigmoid(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0058] In the formula, F is the input feature map, MLP is the fully connected layer, AvgPool and MaxPool are global average pooling and global maximum pooling respectively, and the activation function is Sigmoid.

[0059] In M c (F) Average pooling and maximum pooling are performed on the channel dimension, representing the spatial distribution of different statistical information. Then the feature maps of different spatial information are concatenated, and a 7×7 convolution operation is used on the concatenated feature map to capture the spatial context information to generate the final spatial attention feature map. Finally, the sigmoid layer is used to obtain the spatial attention feature map M. s (F).

[0060] M s (M c (F)) = Sigmoid(f 7×7 ([AvgPool(M c (F)); MaxPool(M c (F))]))

[0061] Where M c (F) is the channel attention feature map, f 7x7 Represents a convolution operation with a convolution kernel size of 7. AvgPool and MaxPool are global maximum pooling and global averaging respectively, and the activation function uses Sigmoid.

[0062] M c (F) and M s(F′) is then multiplied with the original input feature map for adaptive feature correction to generate the final feature map F″.

[0063]

[0064]

[0065] Among them, F is the original input feature, M c (F) is the channel attention feature map, M s (F′) is the spatial attention feature map, Multiply table element-wise.

[0066] 1.3 Global Interaction Module GIM

[0067] The global interaction module is used to capture the global information of features, reduce the degree of information dispersion in features of different scales when extracting features, and enhance the understanding of the overall structure of the image.

[0068] The global interaction module includes an average pooling path, a first adaptive pooling path, a second adaptive pooling path and an original feature path. Each path uses parallel input and obtains features of different scales through global attention and convolution respectively. The features of different scales are spliced ​​and fused to obtain the output of the global interaction module.

[0069] The global interaction module adds a global attention mechanism to different scales of the global context module to reduce information diffusion in the model feature extraction process, thereby improving the global interaction ability of the model during upsampling. Specifically, the present invention uses 3×3 average pooling, 5×5 adaptive pooling, 7×7 adaptive pooling, and original features to splice and fuse features of different scales through the global attention module and convolution, thereby improving the global interaction ability of the model. Figure 4 shown.

[0070] GAM introduces 3D-permutation and the channel attention and convolutional spatial attention submodules of multi-layer perceptrons. The Channel Attention submodule uses 3D-permutation to retain information in three dimensions. Then a two-layer multi-layer perceptron is used to amplify the cross-dimensional channel-spatial dependency. Spatial Attention uses two convolutional layers to fuse spatial information. While removing the maximum pooling operation, ChannelShuffle with Group convolution is introduced to reduce the amount of computation, such as Figure 5 shown.

[0071] 1.4 Loss Function

[0072] The loss function of the present invention is improved on the basis of PopNet loss function. RCS1 and l RCS2 The overall structure and profile features of the RC-SegNet network are measured.

[0073] For the network with RGB-D splicing as input, the MS-SSIM-L1 structural similarity loss function is used to supervise the structure, texture, brightness and other features in feature extraction. This loss function combines the L1 loss function on the basis of the MM-SSIM loss function. The MM-SSIM loss function uses structural similarity metrics at multiple scales when calculating image similarity. By comparing image features such as structure, texture, and brightness at different scales, MM-SSIM can capture the similarity of images more comprehensively. MS-SSIM is prone to brightness changes and color deviations, but it can retain high-frequency information (edges and details of the image), while l L1 The loss function can better keep the brightness and color unchanged. The final MS-SSIM-L1 structural similarity loss function is:

[0074]

[0075]

[0076]

[0077] In the formula, l MM-SSIM-L1 is the structural similarity loss function value, M Indicates different scales, μ p , μ g Respectively represent the mean of the predicted image and the real image, the standard deviation between the predicted image and the real image, σ p , σ g represents the covariance between the predicted image and the real image, β m , γ m Indicates the relative importance between two items, c1 and c2 are constants, f(x i ) is the predicted value, y i is the true value, n is the predicted total amount, α is a constant, which is set to 0.84 in this embodiment, and G is the Gaussian distribution parameter.

[0078] In the RGB-D feature extraction task, the depth image is the image corresponding to the RGB image, which contains the depth information of each pixel. However, due to interference from factors such as sensor noise, occlusion, or depth inconsistency, the depth image may contain discontinuous, noisy, or inaccurate depth values. The goal of the local depth smoothing loss function is to make the depth image smoother and more continuous by penalizing discontinuities and noise in the depth image. By optimizing this loss function, the depth image generated by the network can have better spatial consistency and accuracy. Thus, we get l loc-smoothing The target area is kept smooth and the boundary pixels are clear. The calculation expression of the local depth smoothing loss function is:

[0079]

[0080]

[0081]

[0082] Where D po is the deep feature extracted from the RC-SegNet model, G is the true value, p is the pixel in the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is the pixel adjacent to p, cosine is the cosine similarity, l loc-smoothing is the local depth smoothing loss function value.

[0083] In addition to the MS-SSIM-L1 loss function and the local depth smoothing loss function, an edge sharpening loss function based on Sobel is also used to calculate the edge intensity difference between the generated image and the real image. By minimizing the edge intensity difference, the model can generate clearer and sharper edges. The calculation expression of the edge sharpening loss function is:

[0084]

[0085]

[0086] Among them, D p0 is the deep feature extracted from the RC-SegNet model, p represents the pixels in the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is p adjacent pixels, and γ is equal to 0.5.

[0087] Finally, the three loss functions are weighted and combined to supervise the RC-SegNet network with RGB-D as input (splicing). The weighted calculation expression is:

[0088] l RCS1 = l MS-SSIM-L1 +λ1·l loc-smoothing +λ2·l wtv

[0089] In the formula, l RCS1 is the weighted loss function value, λ1 is the weight coefficient of the local depth smoothing loss function, and λ2 is the weight coefficient of the edge sharpening loss function.

[0090] For the model with RGB and depth images as input, we combine Wiou Loss, Mae Loss, and Wbce loss to get a comprehensive loss function to measure RC-SegNet. Specifically:

[0091] Introducing the Wiou Loss value l Wiou The overlap between the predicted target and the true target is measured, and then the predicted target and the true target are weighted using the minimum weight of each part to accurately describe the contribution of each crack and measure the similarity between the two predictions. The calculation expression is:

[0092]

[0093] In the formula, n represents the number of object boxes, b represents the coordinates of the i-th object box, g represents the coordinates of the real annotation box of the i-th object, and IOU(b i ,g i ) represents the intersection-over-union ratio between the i-th object box and the true annotation box, and w represents the weight value.

[0094] The binary cross entropy loss function Wbce loss measures the difference between the probability distribution predicted by the model and the true label. The calculation expression is:

[0095]

[0096] In the formula, l Wbce is the binary cross entropy loss function value, y is the true label vector, z is the predicted label vector, w is the weight value, and c is the total number of training sets.

[0097] The mean absolute error is the average of the absolute differences between all predicted values ​​and the true values. The mean absolute error loss function Mae Loss can retain the boundary details of the image in the semantic segmentation task. The calculation expression of the mean absolute error loss function is:

[0098]

[0099] In the formula, l Mae is the mean absolute error loss function value, n is the total number of samples, f(x i ) is the predicted value, y i is the true value.

[0100] 2. Experimental process of the present invention

[0101] 2.1 Experimental Dataset

[0102] The road crack dataset used in this paper is derived from real data taken independently, collected by a road inspection vehicle equipped with a camera and a laser transmitter. Data collection was carried out on different highways, and a total of 4683 crack RGB images and depth images were collected. Labelme was used to process the data according to the data format and process of NJU2K. The number of training sets and test sets in the road crack image dataset is 3752 and 931, respectively.

[0103] The statistics of each crack type are shown in Table 1.

[0104] Table 1 Statistics of various crack categories

[0105] type Number of pictures Number of tags Horizontal cracks 1914 8611 Vertical cracks 1843 8142 Web-like cracks 926 4307 total 4683 21059

[0106] It can be seen from the table that the types of cracks covered are relatively comprehensive. Examples of cracks collected are as follows: Figure 6 As shown in the figure, there are 1843 vertical cracks, 1914 horizontal cracks, 926 mesh cracks, and a total of 4683 crack images. There are 8611 horizontal cracks, 8142 vertical cracks, 4307 mesh cracks, and a total of 21059 crack labels.

[0107] 2.2 Experimental equipment and evaluation indicators

[0108] During training, the input image size is set to 256×256, the training cycle is 100, the batch size is 2, the initial learning rate is 0.0001, the learning rate decay is 0.1, and Adm is used as the optimization function of the model.

[0109] In this paper, MIoU, Mae, Smeasure, Max_F, and E_measure are mainly used as evaluation indicators. MIoU is the ratio of the intersection and union of the true value and the predicted value, which is used to calculate the overlap between the predicted area and the true area. The definition is as follows:

[0110]

[0111] Where k+1 is the number of categories, TP is the pixel whose true value and predicted value are both cracks, FN is the pixel whose true value is crack but predicted value is background, and FP is the pixel whose true value is background but predicted value is crack.

[0112] Mae is a statistic that measures the error between the predicted value and the true value. It calculates the average of the absolute errors between all predicted values ​​and the corresponding true values.

[0113]

[0114] In the formula, is the predicted value, y i is the true value, and n is the total number of predictions.

[0115] Structure-measure (Smeasure) is an extension of the structural similarity index (SSIM), which is a measure of the structural similarity between two images. Smeasure takes into account the classification information at the pixel level, not only comparing the similarity of pixel values, but also considering the structural relationship between pixels. It calculates the structural similarity matrix between the predicted segmentation map and the true segmentation map, and then takes the average value to get the Smeasure value. The value range of Smeasure is between 0 and 1, 1 means a perfect match, and 0 means a mismatch. The definition is as follows:

[0116]

[0117]

[0118] S o =μ*O FG +(1-μ)*O BG

[0119] S m =α*S o +(1-α)*S r

[0120] In the formula, S r It means that the image is divided into four regions according to the center of gravity. The weight of each region is calculated by the area of ​​the pixels in the four regions. The weighted average of the SSIM of the four regions is calculated. α is a constant of 0.5, O FG is the similarity of the foreground, O BG is the similarity of the background, μ is the ratio of the foreground to the total pixels, x FG represents the probability value of predicting the foreground area, Represents x FG The average value, x BG represents the probability value of predicting the foreground area, Represents x BGThe average value, S m is the Smeasure value.

[0121] The maximum F-measure (Max_F) is the maximum value of the adaptive F-measure on the entire image, which represents the best overall performance of the segmentation result. It takes into account the precision and recall of the model. Precision represents the ratio of the number of positive samples correctly classified by the model to the number of samples predicted as positive by the model. It is defined as follows:

[0122]

[0123] Recall represents the ratio of the number of positive samples correctly classified by the model to the number of all true positive samples. It is defined as follows:

[0124]

[0125] In the formula, TP means that the model correctly predicts the positive example as a positive example, FP means that the model incorrectly predicts the negative example as a positive example, and FN means that the model incorrectly predicts the positive example as a negative example.

[0126] Max_F can calculate an F based on each pair of PR. β . Usually take the largest F β As an evaluation indicator.

[0127]

[0128] In the formula, β 2 The general value is 0.3.

[0129] E-measure is an evaluation metric based on the enhanced alignment mapping between the saliency map and the true saliency map. It is defined as follows:

[0130]

[0131] Where h and w are the height and width of the mapping, respectively, and φ FM Used to evaluate foreground image and noise.

[0132] 2.3 Comparison of different algorithms

[0133] The comparison results are shown in Table 2. In order to further verify the performance of the improved algorithm, the present invention uses the same data set to compare the improved model RC-SegNet with other mainstream RGB-D segmentation algorithms PopNet, XMSNe, BBSNet, DMRA, SPNet, HiDANet, AFNet, RFNet, TBINet, HINet, etc. The evaluation indicators such as Smeasure, Mae, Max_F, and E-measure are used to compare the advantages and disadvantages of each model. As can be seen from Table 2, the various indicators of RC-SegNet are in the first and second place. Compared with the PopNet network model, the improved model RC-SegNet effectively improves the model performance, Miou, Smeasure, Max_F, and E-measure increase by 2.06%, 1.88%, 4.20%, and 2.12% respectively, and Mae decreases by 1.53%. At the same time, in order to verify the effectiveness of the model improvement, the cracks are compared on different models. The segmentation effect of the model of the present invention is better than the current mainstream segmentation network in terms of details and overall structure.

[0134] Table 2 Comparison results of model indicators

[0135]

[0136] like Figure 7 As shown, the present invention segments mixed cracks (first row), vertical cracks (second row), transverse cracks (third row), and mesh cracks (fourth row) respectively, and the segmentation results are superior to the existing models in terms of details (texture, thickness, etc.) and overall structure.

[0137] 2.4 Ablation Experiment

[0138] The present invention experimentally compares different loss functions, and the experimental results are shown in Table 3, which is a comparison of different loss functions. The Orgin Loss of PopNet is the original loss. The SSIM in PopNet is changed to MS-SSIM-L1, and the structure, texture, brightness and other features of the image are compared at different scales. Miou, Smeasure, Max_F, and E-measure increase by 0.52%, 0.69%, 2.92%, and 0.16%, respectively, and Mae decreases by 1.27%. RSC adds the Mae loss function on the basis of MS-SSIM-L1, which better retains the boundary details of the image in the semantic segmentation task. Compared with the original loss, Miou, Smeasure, Max_F, and E-measure increase by 0.85%, 1.13%, 3.93%, and 0.20%, respectively, and Mae decreases by 1.49%.

[0139] Table 3 Comparison results of different loss functions

[0140] Loss Miou Smeasure Mae Max_F E-measure Origin loss (PopNet) 0.7469 0.7733 0.0256 0.6711 0.9234 MS-SSIM-L1(PopNet) 0.7521 0.7802 0.0129 0.7003 0.9250 RSC (the present invention) 0.7554 0.7846 0.0107 0.7104 0.9254

[0141] The present invention compares different backbone networks, and the experimental results are shown in Table 4. Table 4 is a comparison of different backbone networks. PopNet+VGG11 uses VGG11 as the backbone network, and its Miou and Smeasure are 0.7124 and 0.7421 respectively. Compared with other backbone networks, ResNet101 achieves the best in five indicators, and Miou and Smeasure can reach 0.7485 and 0.7765.

[0142] Table 4 Comparison results of different backbone networks

[0143]

[0144] To verify the effectiveness of each improved module, the present invention is trained on the road crack dataset and calculates its evaluation index. Compared with PopNet, the model PopNet+ResNet101 with ResNet101 as the model backbone network, E-measure increased by 1.68%, and Mae decreased by 1.17%, indicating that the structure can obtain deeper features; PopNet+GIM increased by 1.77% in Max_F and decreased by 1.42% in Mae, indicating that the module can retain richer structural information in the process of upsampling; PopNet+CIM with CIM as the feature fusion module, Max_F increased by 1.77%, and Mae decreased by 1.42%, indicating that the features of different modes are effectively refined during the feature fusion process. Compared with PopNet+CIM, PopNet+CIM+Loss increased by 1.52% in Max_F and 0.44% in Miou, indicating that the loss function of the present invention can better measure the pros and cons of segmentation. Based on this, the present invention integrates various modules, and compared with the original algorithm, Miou, Smeasure, Max_F, and E-measure are increased by 2.06%, 1.88%, 4.20%, and 2.12%, respectively, and Mae is reduced by 1.53%. It can be seen that the RC-SegNet model can effectively segment the details and structures of cracks.

[0145] Table 5 Ablation experiment results

[0146] Modules Miou Smeasure Mae Max_F E-measure PopNet 0.7469 0.7733 0.0256 0.6711 0.9234 PopNet+ResNet101 0.7485 0.7765 0.0139 0.6828 0.9402 PopNet+GIM 0.7496 0.7800 0.0114 0.6888 0.9270 PopNet+CIM 0.7534 0.7871 0.0141 0.6886 0.9366 PopNet+Loss 0.7554 0.7846 0.0107 0.7104 0.9254 PopNet+CIM+Loss 0.7578 0.7877 0.0115 0.7038 0.9390 RC-SegNet(Ours) 0.7675 0.7921 0.0103 0.7131 0.9446

[0147] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A road crack detection method based on cross-modal feature fusion, characterized in that: The following steps are involved: Obtain color features and depth features of road crack data, fuse the color features and depth features to obtain color depth features, input the color features, depth features and color depth features into a detection network based on a cross-modal three-stream feature fusion network for detection, and obtain detection results; The cross-modal three-stream feature fusion network includes a cross-modal interactive fusion module and a global interactive module. The color feature extraction module and the depth feature extraction module respectively extract the color features and depth features of the road crack data. The color features, depth features and color depth features are fused through the improved cross-modal interactive fusion module to form a comprehensive feature, which is then spliced ​​through the global interactive module to obtain a splicing result. The detection network obtains a detection result based on the splicing result.

2. The road crack detection method based on cross-modal feature fusion according to claim 1 is characterized in that: The cross-modal interactive fusion module includes: performing convolution and convolution block attention mechanism on the color features and depth features respectively, multiplying the processed color features and depth features element by element, and then concatenating the product results, color features and color depth features to form comprehensive features.

3. The road crack detection method based on cross-modal feature fusion according to claim 1 is characterized in that: The global interaction module includes an average pooling path, a first adaptive pooling path, a second adaptive pooling path and an original feature path. Each path adopts parallel input, and obtains features of different scales through global attention and convolution respectively. The features of different scales are spliced ​​and fused to obtain the output of the global interaction module.

4. The road crack detection method based on cross-modal feature fusion according to claim 1 is characterized in that: The loss functions used by the detection network include a structural similarity loss function, a local depth smoothing loss function, an edge sharpening loss function and a comprehensive loss function. The structural similarity loss function, the local depth smoothing loss function and the edge sharpening loss function are used to supervise color depth features, and the comprehensive loss function is used to supervise color features and depth features.

5. The road crack detection method based on cross-modal feature fusion according to claim 4 is characterized in that: The calculation expression of the structural similarity loss function is: In the formula, l MM-SSIM-L1 is the structural similarity loss function value, M Indicates different scales, μ p , μ g Represents the mean of the predicted image and the real image, the standard deviation between the predicted image and the real image, σ p , σ g represents the covariance between the predicted image and the real image, β m , γ m Indicates the relative importance between two items, c1 and c2 are constants, f(x i ) is the predicted value, y i is the true value, n is the total predicted value, α is a constant, and G is the Gaussian distribution parameter.

6. The road crack detection method based on cross-modal feature fusion according to claim 4 is characterized in that: The calculation expression of the local depth smoothing loss function is: In the formula, l loc-smoothing is the local depth smoothing loss function value, D po is the depth feature, G is the true value, p is the pixel in the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is p adjacent pixels, and cosine is the cosine similarity.

7. The road crack detection method based on cross-modal feature fusion according to claim 4 is characterized in that: The calculation expression of the edge sharpening loss function is: In the formula, l wtv is the edge sharpening loss function value, D p0 is the depth feature, p is the pixel in the crack area, represents the Sobel vector in the horizontal direction, represents the Sobel vector in the vertical direction, is the number of adjacent pixels.

8. The road crack detection method based on cross-modal feature fusion according to claim 4 is characterized in that: The expression of the comprehensive loss function is: l RCS2 ＝l Wiou +l Wbce +l Mae In the formula, l Wiou is the overlap component, l Wbce is the binary cross entropy loss, l Mae is the mean absolute error, n represents the number of object boxes, b represents the coordinates of the i-th object box, g represents the coordinates of the real annotation box of the i-th object, and IOU (b i ,g i ) represents the intersection-over-combination ratio between the i-th object box and the true annotation box, w represents the weight value, y is the true label vector, z is the predicted label vector, c is the total number of training sets, f(x i ) is the predicted value, y i is the true value.

9. A road crack detection device based on cross-modal feature fusion, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the road crack detection method based on cross-modal feature fusion as described in any one of claims 1-8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, a road crack detection method based on cross-modal feature fusion as described in any one of claims 1 to 8 is implemented.

Images