Pavement crack detection method and device and electronic equipment

Abstract

The invention provides a pavement crack detection method and apparatus, and an electronic device. The method comprises the steps of obtaining a crack image and a crack prediction model obtained by pre-training; wherein the crack prediction model comprises a visual modeling model, an inference model and a dynamic multi-modal fusion network; performing hidden space global visual modeling on the crack image based on a visual modeling model to obtain hidden space global visual features; generating a semantic mask matrix guided by global scene semantics for the crack image based on an inference model; and inputting the hidden space global visual features and the semantic mask matrix into a dynamic multi-modal fusion network, and predicting to obtain crack information. According to the invention, pixel-level prediction of the crack can be carried out, accurate detection of the pavement crack is realized, and technical support is provided for pavement maintenance.

Description

Technical Field

[0001] The present invention relates to the technical field of road maintenance, and in particular to a road surface crack detection method, device and electronic equipment. Background Art

[0002] Pavement cracks are one of the key diseases that affect the service life of roads and driving safety. Their timely and accurate detection and repair are crucial to pavement management departments. Traditional pavement crack detection methods usually rely on manual inspections or simple image processing techniques, which are not only time-consuming and labor-intensive, but also have limited application effects in large-scale road networks. With the development of computer vision technology, pavement crack detection based on deep learning has gradually become mainstream, showing significant advantages.

[0003] The main challenge of pavement crack detection is the interference of complex backgrounds, such as shadows, road markings, and debris, which can easily lead to misjudgment of traditional detection methods. Deep learning-based crack detection technology uses loss function constraints to intelligently train a large number of road crack images, enabling the model to adaptively learn and mine complex crack features, with recognition accuracy far exceeding traditional methods. This method can not only accurately segment crack areas under complex backgrounds and lighting conditions, but also effectively reduce errors and interference factors in manual inspections. It is highly automated and robust, and can quickly handle crack detection tasks in large-scale road networks, providing solid scientific and technological support for road maintenance decisions. Summary of the invention

[0004] In order to achieve high-precision pixel-level prediction of cracks, the present invention provides a pavement crack detection method, device and electronic device. The technical solution proposed by the present invention is as follows: In a first aspect, the present invention provides a pavement crack detection method, comprising: Acquire a crack image and a pre-trained crack prediction model; wherein the crack prediction model includes a visual modeling model, an inference model, and a dynamic multimodal fusion network; Performing latent space global visual modeling on the crack image based on the visual modeling model to obtain latent space global visual features; generating a global scene semantics-guided semantic mask matrix for the crack image based on the inference model; The latent space global visual features and semantic mask matrix are input into the dynamic multimodal fusion network to predict the crack information.

[0005] Optionally, the visual modeling model includes a first convolutional neural network, a multi-scale convolutional neural network, and a graph convolutional aggregation network; The performing latent space global visual modeling on the crack image based on the visual modeling model to obtain latent space global visual features includes: Extracting features from the crack image based on a first convolutional neural network to obtain a feature map; A sparse self-attention mechanism is introduced to perform local hidden layer feature representation on the feature map to obtain sparse self-attention features; Extracting features from the feature map based on a multi-scale convolutional neural network to obtain multi-scale features; The multi-scale features and the sparse self-attention features are fused based on a graph convolutional aggregation network to obtain the latent space global visual features.

[0006] Optionally, the inference model includes a segmentation model, a second convolutional neural network and an adaptive graph convolutional network; generating a semantic mask matrix guided by global scene semantics for the crack image based on the inference model includes: Segmenting the crack image by segmenting the large model to obtain an initial multi-scale mask matrix; Extracting context features of the crack image using a second convolutional neural network, and generating context-aware features in combination with the initial multi-scale mask matrix; The context-aware features are globally modeled using an adaptive graph convolutional network, and a contrastive learning mechanism is introduced to obtain a semantic mask matrix.

[0007] Optionally, the dynamic multimodal fusion network includes an adaptive weight generation module, an interaction network and a multi-task model; The method of inputting the latent space global visual features and the semantic mask matrix into the dynamic multimodal fusion network to predict crack information includes: Based on the adaptive weight generation module, the global statistical information of the latent space global visual features and the semantic mask matrix is ​​processed to generate dynamic fusion weights; The interactive network performs feature fusion on the latent space global visual features and the semantic mask matrix based on the dynamic fusion weights to obtain a fused feature map; The multi-task model performs crack detection and feature estimation based on the fused feature map to predict crack information.

[0008] Optionally, before the global statistical information of the latent space global visual features and the semantic mask matrix is ​​processed based on the adaptive weight generation module to generate the dynamic fusion weight, the method further includes: The latent space global visual features and the semantic mask matrix are mapped to the same feature dimension.

[0009] Optionally, the dynamic multimodal fusion network is trained in the following manner: Input the data set into the dynamic multimodal fusion network to obtain the output of each task; Calculate the loss of each task based on the output and the true label, and calculate the total loss based on the constructed total loss function; Model parameters of the dynamic multimodal fusion network are updated based on the total loss.

[0010] In a second aspect, the present invention provides a road surface crack detection device, comprising: An acquisition module, used to acquire a crack image and a pre-trained crack prediction model; wherein the crack prediction model includes a visual modeling model, an inference model and a dynamic multimodal fusion network; A modeling module, used for performing latent space global visual modeling on the crack image based on a visual modeling model to obtain latent space global visual features; An inference module, configured to generate a semantic mask matrix guided by global scene semantics for the crack image based on an inference model; The fusion module is used to input the latent space global visual features and semantic mask matrix into the dynamic multimodal fusion network to predict the crack information.

[0011] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the pavement crack detection method as described in the first aspect when executing the computer program.

[0012] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the pavement crack detection method as described in the first aspect.

[0013] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the pavement crack detection method as described in the first aspect.

[0014] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The pavement crack detection method provided by the present invention aims to overcome the shortcomings of the existing crack detection methods, namely, being susceptible to background interference and insufficient detection accuracy in complex scenes. Firstly, the YOLO framework is combined with a sparse self-attention mechanism to perform global visual modeling, and a large segmentation model is introduced to generate a semantically guided mask matrix. Finally, dynamic multimodal fusion and an adaptive attention mechanism are used to achieve high-precision pixel-level prediction of cracks.

[0015] For a given crack image, latent space global visual modeling is performed to obtain latent space global visual features. At the same time, an inference model is introduced to construct semantic mask features guided by contextual semantics, which further improves the pixel-level representation of cracks and provides effective support for visual modeling. Then, based on the dynamic multimodal fusion network, feature aggregation is performed on the latent space global visual features and the semantic mask matrix to perform pixel-level prediction of cracks, thereby achieving accurate detection of pavement cracks and providing technical support for pavement maintenance.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic flow chart of the pavement crack detection method provided by the present invention.

[0020] Figure 2 This is the overall framework diagram of crack prediction provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the pavement crack detection device provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] At present, the application of deep learning methods in pavement crack detection technology mainly covers self-supervised learning, contrastive learning, spatiotemporal convolutional networks (ST-CNN) and graph neural networks (GNN). Self-supervised learning extracts complex features of cracks through adaptive training of unlabeled data; contrastive learning optimizes the ability to distinguish crack features by enhancing the similarity of similar samples; spatiotemporal convolutional networks capture the dynamic evolution of cracks by integrating time and space information and improve the ability to predict crack development trends; graph neural networks construct crack graph structures to deeply analyze the geometric morphology and spatiotemporal relationship of cracks. Through the integration of these technologies, deep learning performs well in pavement crack detection, with higher accuracy and adaptability, providing strong support for long-term monitoring and intelligent maintenance of roads.

[0025] However, the inventors found that the existing models have obvious deficiencies in analyzing the pixel-level relationship between crack objects and scenes. This limitation makes it impossible for the model to fully capture the morphology and structural characteristics of cracks and the contextual semantics in complex environments, which directly affects the evaluation of the pavement maintenance cycle, increases the cost of road maintenance and potential safety risks. After further research and development, the inventors made the present invention.

[0026] Combine the following Figure 1-Figure 4 The present invention describes a pavement crack detection method, device and electronic device.

[0027] Figure 1 It is a flow chart of the pavement crack detection method provided by the present invention. The method performs latent space global visual modeling of a given crack image based on the YOLO framework, and at the same time introduces a large segmentation model to construct mask features guided by contextual semantics, providing effective support for visual modeling, and ultimately achieving accurate detection of pavement cracks, providing technical support for airport pavement maintenance decisions.

[0028] like Figure 1 As shown, the method includes the following: Step 101: Acquire a crack image and a pre-trained crack prediction model.

[0029] Images containing cracks are acquired through cameras, drones or other image acquisition devices. These images come from a variety of scenes such as road surfaces, building surfaces, bridge structures, etc., ensuring that the image quality meets the analysis requirements, including clarity, lighting conditions, etc. The above crack prediction model is a pre-trained deep learning model, including a visual modeling model, an inference model, and a dynamic multimodal fusion network. The model is trained based on a large amount of crack image data and has the ability to identify cracks of different types and shapes.

[0030] Step 102: perform latent space global visual modeling on the crack image based on the visual modeling model to obtain latent space global visual features.

[0031] The visual modeling model maps the input crack image into a high-dimensional latent space to capture the global visual features in the image. These features include the shape, texture, direction, etc. of the crack, which are helpful for subsequent analysis and recognition.

[0032] Step 103: Generate a semantic mask matrix guided by global scene semantics for the crack image based on the inference model

[0033] The inference model uses global scene information to guide the crack recognition process. This helps to accurately distinguish cracks from other objects in complex backgrounds. By analyzing the contextual information in the image (such as road surface texture, building structure, etc.), the inference model is able to generate a semantic mask matrix . Semantic mask matrix The areas in the image that may contain cracks are marked, providing important clues for subsequent crack information extraction.

[0034] Step 104: transform the latent space global visual features and the semantic mask matrix Input the dynamic multimodal fusion network to obtain crack information.

[0035] Dynamic multimodal fusion network combines global visual features in latent space and the semantic mask matrix , and further extract crack information through multimodal fusion technology. The dynamic multimodal fusion network can make full use of information from different sources, namely visual features and semantic information, to improve the accuracy and robustness of crack identification. Finally, the network outputs crack information, including detailed parameters such as the location, shape, and size of the crack.

[0036] The pavement crack detection method provided by the present invention combines a visual modeling model, an inference model and a dynamic multimodal fusion network, and the process can more accurately identify crack information in crack images. Hidden space global visual modeling and global scene semantic guidance help to distinguish cracks from other objects in complex backgrounds and improve recognition accuracy. The dynamic multimodal fusion network can make full use of information from different sources to improve the robustness of the crack recognition system. Even under poor lighting conditions and poor image quality, the process can maintain high recognition performance. The process is not only suitable for pavement crack detection, but can also be extended to crack recognition in various scenes such as building surfaces and bridge structures.

[0037] In an alternative embodiment, referring to Figure 2 As shown, the visual modeling model includes a first convolutional neural network, a multi-scale convolutional neural network and a graph convolutional aggregation network; The step S102 described above performs latent space global visual modeling on the crack image based on the visual modeling model to obtain latent space global visual features, specifically including the following steps: S1021. Perform feature extraction on the crack image based on a first convolutional neural network to obtain a feature map.

[0038] First, let the input crack image The feature map obtained after the first convolutional neural network Conv is expressed as ,in, H , W , C Represent the height, width and number of channels of the feature map respectively.

[0039] S1022, introduce a sparse self-attention mechanism to perform local hidden layer feature representation on the feature map to obtain a sparse self-attention feature .

[0040] Define the sparse attention weights as ,in i and j Represents two positions in the feature map, and its calculation expression is:

[0041] in, and are the linear transformation matrices of query and key respectively, is the dimension of the key vector, and Representation feature map Middle position i and j The eigenvector of the above linear transformation matrix , The specific generation method is as follows: When the neural network is initialized, and Initialization is performed according to the initialization strategy (such as Xavier initialization or Kaiming initialization) to ensure that the neural network has good learning performance in the initial stage. and It will be continuously adjusted through the back propagation algorithm. Back propagation will adjust the and The update of these matrices is completed through optimization algorithms (such as SGD, Adam, etc.), and the specific formula is:

[0042] in, is the learning rate, is the loss function. After multiple rounds of iterative training, the model will gradually converge. and The values ​​of are determined so that they can effectively map the input features into suitable query and key vector spaces, thereby achieving effective information extraction and feature aggregation in the self-attention mechanism.

[0043] Then, to achieve sparse selection, a gating mechanism is applied and a threshold τ is set, retaining only the part with a median attention weight greater than τ:

[0044] For the selection of the above threshold τ, the sparse attention weight can be analyzed based on statistical characteristics. distribution, select an appropriate threshold. For example, you can choose sparse attention weights The mean or median is used as the threshold to retain the most important part.

[0045] To choose sparse attention weights Taking the mean of as the threshold as an example, the model obtains sparse attention weights of different dimensions When flattening, multiple one-dimensional arrays are obtained. These sparse attention weights You can take the sparse attention weight values ​​of multiple batches of samples on the entire dataset or on a small dataset. Then, calculate the mean of all sparse attention weight values ​​and use the calculated mean Assign to τ:

[0046] Finally, the sparse self-attention feature It is expressed as:

[0047] in, Represents the sparse neighborhood associated with position i. Through this step, the model is able to aggregate global features from the crack area. For the selection of the above sparse neighborhood, eight points around the lower left, upper left, upper right, lower left, and lower right of position i can be used. If these points are not filtered out by τ, they are calculated, and if they are filtered out, they are not calculated.

[0048] S1023. Perform feature extraction on the feature map based on a multi-scale convolutional neural network to obtain multi-scale features.

[0049] Extracting multi-scale features from different layers of the YOLO framework :

[0050] in, Represents a convolutional neural network of scale n, n=3, 5, 7, 9, these features represent different levels of feature representation captured for a given image.

[0051] S1024, based on the graph convolutional aggregation network, the multi-scale features and the sparse self-attention features Fusion is performed to obtain the global visual features of the latent space .

[0052] In order to further improve the expressiveness of features, the present invention combines the sparse self-attention features extracted by the sparse self-attention mechanism with convolutional features of different scales, and adjusts the features through an optimization algorithm. With sparse self-attention features Based on graph convolution aggregation network The fusion strategy of and sparse self-attention features As nodes in the graph, through the node mapping function The feature matrix is ​​mapped into a graph node form. Finally, the graph convolution is used for information interaction and feature aggregation, which can effectively integrate multi-scale information and global context. Finally, the optimized latent space global visual feature is obtained. :

[0053] The above node mapping function This function takes the input sparse self-attention features , through some linear transformation Mapped to the graph node space. That is, the features in the feature space are projected into the space of the node vector, producing The specific definitions are as follows:

[0054] in, Represents the L-2 normalization function, which normalizes the channel dimension of each node vector. Reshaping function The resolution of the input feature is converted to c×h×w, where K = h×w, and a spatial feature of scale h×w is compressed into a point (value) representation, and then combined with the sparse self-attention feature Do multiplication to further enhance the node representation.

[0055] The above step S1011 extracts features from the crack image through the first convolutional neural network (CNN), which can capture low-level features in the image, such as edges, textures, etc. These features provide a basis for subsequent high-level feature extraction and modeling. The comprehensiveness of feature extraction is ensured, providing a rich source of information for the subsequent modeling process.

[0056] Step S1022 introduces a sparse self-attention mechanism to process the feature map, which can focus on important areas in the image and generate sparse self-attention features. This mechanism helps the model identify crack areas in complex backgrounds and improves the robustness of the model. It enhances the model's ability to represent local features of the image and improves the recognition accuracy of the model in complex scenes.

[0057] Step S1023 further processes the feature map through a multi-scale convolutional neural network to obtain multi-scale features. Multi-scale features can capture information of different scales in the image, improve the model's ability to detect cracks of different shapes and sizes, and enhance the generalization of the model.

[0058] Step S1024 fuses the multi-scale features and sparse self-attention features based on the graph convolutional aggregation network to obtain the latent space global visual features. The graph convolutional aggregation network can capture the complex relationship between features, thereby generating more representative global features. By fusing multiple features, more comprehensive and accurate global visual features are generated, providing strong support for subsequent crack identification.

[0059] The present invention designs an efficient visual modeling model that can make full use of various information in crack images to generate high-quality latent space global visual features. These features not only contain local details of the image, but also integrate global contextual information, providing a solid foundation for subsequent crack identification tasks. In practical applications, the model can more accurately identify information such as the location, shape and size of the cracks.

[0060] In summary, the latent space global visual modeling process based on the above steps shows significant beneficial effects in terms of the comprehensiveness of feature extraction, the representation ability of local features, the fusion of multi-scale features, and the representation of global features, thereby improving the performance and accuracy of the entire crack identification system.

[0061] In an alternative embodiment, referring to Figure 2 As shown, the reasoning model in the above S103 includes a large segmentation model, a second convolutional neural network and an adaptive graph convolutional network; the generation of a semantic mask matrix guided by global scene semantics for the crack image based on the reasoning model includes: S1031 , segmenting the crack image using a segmentation model (Segment Anything Model, SAM) to obtain an initial multi-scale mask matrix.

[0062] First, the input crack image is segmented using the large model Perform preprocessing to generate image pyramids of different resolutions ,in, S Indicates the number of scales. The image of each scale Input to the segmentation model SAM to generate multiple preliminary semantic mask sets , each mask Corresponding to a pixel-level segmentation area. Using the multi-scale fusion strategy, the segmentation masks of different scales are The fusion is performed, and the redundancy and conflict are eliminated through the mask consistency optimization algorithm, and finally a unified initial multi-scale mask matrix is ​​obtained. ,The key of this step is to refine the segmentation results through ,multiscale processing and adaptive mask optimization.

[0063]

[0064] in, Represents a multi-scale fusion operation.

[0065] S1032, extracting context features of the crack image using a second convolutional neural network , and combined with the initial multi-scale mask matrix to generate context-aware features .

[0066] In generating the multi-scale semantic segmentation mask, that is, the initial multi-scale mask matrix After that, the global semantic features are constructed through the context enhancement mechanism. The second convolutional neural network is used to extract the input crack image Contextual features Then, combined with the initial multi-scale mask matrix , generating preliminary context-aware features :

[0067] in, is a learnable weight factor used to emphasize the feature contribution of different semantic regions.

[0068] S1033. Use Adaptive Graph Convolutional Networks (A-GCNs) to globally model the context-aware features, and introduce a contrastive learning mechanism to obtain a semantic mask matrix.

[0069] Context-aware features After extraction, an adaptive graph convolutional network is used to extract context-aware features. Perform global modeling. The network can capture the dependencies between different regions by performing convolution operations on the graph structure and generate a more semantically consistent global feature representation, namely the initial mask matrix

[0070] in, is an adaptively constructed adjacency matrix representing the graph nodes The relationship between.

[0071] Then, the contrastive learning mechanism is introduced to refine the initial mask matrix , and get the final mask matrix Specifically, the contrastive learning mechanism is introduced to the initial mask matrix By comparing positive and negative sample pairs, the model can better distinguish cracks from background areas, further improving the accuracy of semantic masks. The loss function of contrastive learning is defined as:

[0072] in, is the positive sample mask, is the negative sample mask, is the temperature coefficient. Positive sample mask It represents the accurately marked target area, i.e. the crack area, obtained through the SAM model . Negative sample mask Represents the non-target area, that is, the background or other irrelevant area, through Get. The semantic mask matrix optimized by contrastive learning It will have higher semantic segmentation accuracy and robustness and can be used for subsequent crack detection tasks.

[0073] The present invention generates semantic information of the global scene by segmenting the large model and constructing an adaptive, semantically guided semantic mask matrix The matrix can not only identify and segment the important areas of the crack and its surrounding environment, but also be optimized through multi-level context information to improve the accuracy and robustness of crack detection.

[0074] The pavement crack detection method provided by the present invention performs latent space global visual modeling of a given crack image based on the YOLO framework, and at the same time introduces a large segmentation model to construct mask features guided by contextual semantics, further improving the pixel-level representation of cracks, providing effective support for visual modeling, and ultimately achieving accurate detection of pavement cracks, providing necessary technical support for airport pavement maintenance decisions.

[0075] In an optional embodiment, the latent space global visual features and the semantic mask matrix The dynamic multimodal fusion network (DMFN) is used to jointly learn the feature inputs of different modalities. The DMFN adaptively adjusts the weights of each modal feature to cope with the changes in different crack morphologies and backgrounds. The dynamic multimodal fusion network includes an adaptive weight generation module, an interactive network, and a multi-task model.

[0076] Reference Figure 2 As shown, the above S104 describes the global visual features of the latent space and the semantic mask matrix Input the dynamic multimodal fusion network to obtain crack information, including: S1041. Global visual features of latent space and the semantic mask matrix Normalize the latent space global visual features using linear transformation and the semantic mask matrix Mapped to the same feature dimension d , we get the normalized latent space global visual features And the normalized semantic mask matrix : =

[0077] in, and is the linear transformation matrix, d It is a unified feature dimension.

[0078] S1042, introducing an adaptive weight generation module, and based on the adaptive weight generation module, performing normalization on the latent space global visual features. And the normalized semantic mask matrix The global statistical information (such as mean, standard deviation, etc.) is processed to generate dynamic fusion weights:

[0079] in, represents the mean operation, Represents the standard deviation operation. and are the trainable parameters of the neural network, is the Sigmoid activation function, which is used to limit the weights between [0,1]. and is the global visual feature of latent space and the semantic mask matrix Dynamic fusion weights.

[0080] S1043, the interactive network processes the normalized latent space global visual features based on the dynamic fusion weights And the normalized semantic mask matrix Perform feature fusion to obtain the fused feature map : =

[0081] S1044. The multi-task model performs crack detection and feature estimation based on the fused feature map to predict crack information.

[0082] In order to achieve pixel-level prediction of cracks, the present invention designs a multi-task learning architecture, namely the above-mentioned multi-task model, which not only needs to predict the pixel-level position of the cracks, but also needs to estimate the width, depth and other features of the cracks at the same time. Specifically, after obtaining the fused feature map F, the multi-task model uses different branches to perform pixel-level segmentation, width estimation and depth estimation of the cracks respectively. Each branch will output corresponding prediction results, such as segmentation maps, width maps and depth maps. Specifically, the above-mentioned multi-task model includes a segmentation network, which is mainly responsible for the pixel-level segmentation of cracks, that is, predicting whether each pixel in the image belongs to the crack area. This branch uses a convolutional neural network as the basis, combined with upsampling layers such as transposed convolution layers or bilinear interpolation to restore the original resolution of the image, and outputs a pixel-level segmentation map of the same size as the input image.

[0083] The above multi-task model also includes two branches responsible for crack width and depth estimation tasks. These two branches also use convolutional neural networks as the basis to extract task-specific features and use fully connected layers to predict the width and depth of cracks. Finally, the model outputs pixel-level prediction results of cracks (i.e., segmentation maps) and estimates of crack width and depth. These results can be directly used for pavement crack detection, evaluation, and maintenance decisions.

[0084] The present invention uses an adaptive weight generation module, and the method can dynamically generate fusion weights according to the global statistical information of the latent space global visual features and the semantic mask matrix. This dynamic adjustment capability enables the model to better cope with data changes in different scenarios and improve the accuracy and robustness of fusion. The interactive network performs feature fusion on the latent space global visual features and the semantic mask matrix based on the dynamically generated fusion weights. This process effectively combines the visual features and semantic information, so that the fused feature map contains richer and more comprehensive information. This optimized feature map helps subsequent crack detection and feature estimation tasks to obtain more accurate results. The multi-task model is used to perform crack detection and feature estimation on the fused feature map, which makes full use of the correlation between tasks and improves the generalization ability and overall performance of the model. Multi-task learning not only helps to improve the accuracy of crack detection, but also can simultaneously estimate multiple features of cracks, such as length, width, depth, etc., providing important reference for subsequent maintenance and repair work. Before feature fusion, the latent space global visual features and the semantic mask matrix are mapped to the same feature dimension by mapping, ensuring the compatibility and comparability between features. This step is the basis for successful feature fusion and helps to reduce information loss and noise interference caused by inconsistent dimensions.

[0085] In summary, the present invention successfully achieves pixel-level prediction of cracks by innovatively combining global features and mask matrices and introducing multi-task learning. The final prediction result not only has sub-pixel accuracy, but also can accurately describe the width and depth of the cracks, providing strong technical support for the detection and maintenance decisions of pavement cracks. Moreover, it can overcome the limitations of the prior art in analyzing the pixel-level relationship between crack objects and scenes. The model can fully capture the morphology, structural characteristics and contextual semantics of cracks in complex environments, thereby facilitating the evaluation of factors affecting the pavement maintenance cycle and reducing the cost of road maintenance and potential safety risks.

[0086] In an optional embodiment, the dynamic multimodal fusion network is trained in the following manner: S201, introduce multi-task learning, input the data set into the dynamic multimodal fusion network, and obtain the output of each task; For pixel-level segmentation tasks, the loss function can use binary cross entropy loss:

[0087] in, N is the total number of pixels, It is i The true label of each pixel (0 or 1), The model predicts i The probability that a pixel belongs to a crack.

[0088] For the crack width estimation task, the loss function can use the mean square error (MSE) as the loss function:

[0089] in, M is the number of crack samples with width annotations in the training data, It is j The true width of samples, is the width predicted by the model.

[0090] For the crack depth estimation task, similar to the crack width estimation, crack depth estimation is also a regression task, and MSE can also be used as the loss function:

[0091] in, K is the number of crack samples with depth annotations in the training data, It is k The true depth of samples, is the depth predicted by the model.

[0092] Combining the above three loss functions, we get the total loss function. In summary, the loss function of multi-task learning can be defined as,

[0093] in, is the pixel-level segmentation loss, is the estimated crack width loss, is the crack depth estimated loss, and is the weight coefficient, is the loss of the above contrastive learning. Among them, and Used to balance the loss contribution between different tasks and can be optimized through model training.

[0094] S202. Calculate the loss of each task based on the output and the true label, and calculate the total loss based on the constructed total loss function.

[0095] S203, based on the total loss, the model parameters of the dynamic multimodal fusion network are updated. Specifically, the gradient of the total loss function relative to the model parameters is calculated using the back propagation algorithm. The model parameters are updated according to the gradient using the optimizer. The performance of the model is evaluated using the validation set. This helps monitor whether the model is overfitting and can be used as a basis for adjusting hyperparameters. After the training is completed, the final performance of the model is evaluated using the test set. This is the performance indicator of the model in practical applications. Hyperparameters such as the learning rate, batch size, and regularization parameters are adjusted according to the performance of the validation set. The trained model is deployed in practical applications to perform pixel-level prediction, width estimation, and depth estimation of cracks.

[0096] The present invention introduces multi-task learning and optimizes the losses of multiple tasks at the same time. The dynamic multimodal fusion network not only learns the specific features for a single task, but also can share and transfer knowledge between different tasks. This cross-task learning helps to improve the generalization ability of the model, enabling it to better adapt to unseen data and tasks. The method of updating model parameters based on total loss ensures that the network can comprehensively consider the performance of all tasks, thereby avoiding overfitting on a single task and ignoring the importance of other tasks. This method helps to find a more balanced and optimized set of parameters so that the network can achieve better performance on multiple tasks.

[0097] The pavement crack detection device provided by the present invention is described below. The pavement crack detection device described below and the pavement crack detection method described above can be referenced to each other.

[0098] Reference Figure 3 As shown, the device comprises: An acquisition module 301 is used to acquire a crack image and a pre-trained crack prediction model; wherein the crack prediction model includes a visual modeling model, an inference model and a dynamic multimodal fusion network; A modeling module 302 is used to perform latent space global visual modeling on the crack image based on a visual modeling model to obtain latent space global visual features; An inference module 303, configured to generate a semantic mask matrix guided by global scene semantics for the crack image based on an inference model; The fusion module 304 is used to input the latent space global visual features and the semantic mask matrix into the dynamic multimodal fusion network to predict the crack information.

[0099] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the pavement crack detection method.

[0100] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pavement crack detection method provided by the above-mentioned methods.

[0102] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the pavement crack detection method provided by the above-mentioned methods.

[0103] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A road crack detection method, characterized in that: include: Acquire a crack image and a pre-trained crack prediction model; wherein the crack prediction model includes a visual modeling model, an inference model, and a dynamic multimodal fusion network; Performing latent space global visual modeling on the crack image based on the visual modeling model to obtain latent space global visual features; generating a global scene semantics-guided semantic mask matrix for the crack image based on the inference model; The latent space global visual features and semantic mask matrix are input into the dynamic multimodal fusion network to predict the crack information.

2. The pavement crack detection method according to claim 1, characterized in that: The visual modeling model includes a first convolutional neural network, a multi-scale convolutional neural network and a graph convolutional aggregation network; The performing latent space global visual modeling on the crack image based on the visual modeling model to obtain latent space global visual features includes: Extracting features from the crack image based on a first convolutional neural network to obtain a feature map; A sparse self-attention mechanism is introduced to perform local hidden layer feature representation on the feature map to obtain sparse self-attention features; Extracting features from the feature map based on a multi-scale convolutional neural network to obtain multi-scale features; The multi-scale features and the sparse self-attention features are fused based on a graph convolutional aggregation network to obtain the latent space global visual features.

3. The pavement crack detection method according to claim 1, characterized in that: The inference model includes a segmentation model, a second convolutional neural network and an adaptive graph convolutional network; the generation of a semantic mask matrix guided by global scene semantics for the crack image based on the inference model includes: Segmenting the crack image by segmenting the large model to obtain an initial multi-scale mask matrix; Extracting context features of the crack image using a second convolutional neural network, and generating context-aware features in combination with the initial multi-scale mask matrix; The context-aware features are globally modeled using an adaptive graph convolutional network, and a contrastive learning mechanism is introduced to obtain a semantic mask matrix.

4. The pavement crack detection method according to claim 1, characterized in that: The dynamic multimodal fusion network includes an adaptive weight generation module, an interaction network and a multi-task model; The method of inputting the latent space global visual features and the semantic mask matrix into the dynamic multimodal fusion network to predict crack information includes: Based on the adaptive weight generation module, the global statistical information of the latent space global visual features and the semantic mask matrix is ​​processed to generate dynamic fusion weights; The interactive network performs feature fusion on the latent space global visual features and the semantic mask matrix based on the dynamic fusion weights to obtain a fused feature map; The multi-task model performs crack detection and feature estimation based on the fused feature map to predict crack information.

5. The pavement crack detection method according to claim 4, characterized in that: Before processing the global statistical information of the latent space global visual features and the semantic mask matrix based on the adaptive weight generation module to generate the dynamic fusion weight, the method further includes: The latent space global visual features and the semantic mask matrix are mapped to the same feature dimension.

6. The pavement crack detection method according to claim 1, characterized in that: The dynamic multimodal fusion network is trained in the following manner: Input the data set into the dynamic multimodal fusion network to obtain the output of each task; Calculate the loss of each task based on the output and the true label, and calculate the total loss based on the constructed total loss function; Model parameters of the dynamic multimodal fusion network are updated based on the total loss.

7. A road crack detection device, characterized in that: include: An acquisition module, used to acquire a crack image and a pre-trained crack prediction model; wherein the crack prediction model includes a visual modeling model, an inference model and a dynamic multimodal fusion network; A modeling module, used for performing latent space global visual modeling on the crack image based on a visual modeling model to obtain latent space global visual features; An inference module, configured to generate a semantic mask matrix guided by global scene semantics for the crack image based on an inference model; The fusion module is used to input the latent space global visual features and semantic mask matrix into the dynamic multimodal fusion network to predict the crack information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the pavement crack detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the pavement crack detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the pavement crack detection method according to any one of claims 1 to 6 is implemented.

Images