Bridge concrete crack intelligent detection method based on deep neural network

Abstract

The invention relates to the technical field of crack detection, in particular to a bridge concrete crack intelligent detection method based on a deep neural network. The method comprises the following steps: firstly, acquiring images of a bridge crack high-incidence area through an unmanned aerial vehicle, constructing an inspection data set, and performing preliminary classification on the images by using a mutual information clustering algorithm to distinguish crack images from non-crack images; acquiring position information of a crack area based on the crack image, and acquiring ultrasonic data of the crack area by using an ultrasonic probe; establishing a multi-modal crack data set by fusing ultrasonic data and crack image data, detecting the multi-modal crack data set by using a multi-modal fusion network, and distinguishing true cracks from false cracks; the method comprises the following steps: carrying out 3D modeling on true cracks to generate a visual crack model, grading the cracks based on a damage evaluation method, and formulating corresponding repair measures for mild, moderate and serious damages to ensure the safety and stability of a bridge structure.

Description

Technical Field

[0001] The present invention relates to the technical field of crack detection, and in particular to an intelligent detection method for bridge concrete cracks based on a deep neural network. Background Art

[0002] Bridges are key infrastructure in the transportation network of modern society, connecting cities and villages, economy and life. With the increase of traffic load and the erosion of the natural environment, bridge structures are facing various forms of aging and damage, especially concrete cracks. Although these cracks may seem small in the early stage, if they are not discovered and treated in time, they may cause steel corrosion, structural failure, and even bridge collapse, causing huge economic losses and casualties. Therefore, timely and accurate detection of cracks in bridge concrete and assessment of their severity are crucial to ensure bridge safety.

[0003] Traditional bridge crack detection methods mainly rely on simple tool assistance, such as crack width rulers and conventional sensors. These methods can detect surface cracks to a certain extent, but there are many limitations. Conventional tools are difficult to conduct comprehensive inspections at high altitudes and in complex environments, have limited detection capabilities for fine cracks, and cannot accurately assess the depth and severity of crack extension. Defect detection technology based on deep neural networks can quickly and accurately identify cracks on the surface of bridges through automated image processing, which not only improves detection efficiency but also eliminates human errors. With the help of equipment such as drones, it is possible to cover difficult-to-reach areas of bridges, greatly making up for the shortcomings of traditional tool detection.

[0004] However, the current defect detection technology based on deep neural networks still has certain shortcomings. First, the detection accuracy based solely on image data is limited, especially under complex lighting and material surface conditions, the edge and width of the crack may not be accurately identified. Secondly, the image can only capture surface information and cannot deeply obtain the specific morphology and depth of the crack, which limits the comprehensive assessment of the severity of the crack and makes it difficult to propose targeted repair plans based on the depth, direction and other characteristics of the crack.

[0005] Therefore, an intelligent detection method for bridge concrete cracks based on deep neural network is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide an intelligent detection method for bridge concrete cracks based on deep neural networks, which can achieve efficient and accurate detection of bridge concrete cracks by fusing data of different modes. In order to solve the problems existing in the prior art, the present invention first proposes a mutual information clustering algorithm for preliminary classification of concrete crack images to improve the accuracy of crack identification; secondly, the present invention proposes a multimodal fusion network, which combines drone images and ultrasonic data to comprehensively extract the spatial and temporal characteristics of cracks and achieve multi-angle accurate detection; finally, the present invention proposes a damage assessment method, which quantitatively evaluates the degree of crack damage based on the geometric characteristics of the cracks and the structural stress distribution, and formulates targeted repair plans.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A bridge concrete crack intelligent detection method based on deep neural network, comprising:

[0009] Using drones to inspect images of high-incidence areas of bridge cracks to obtain a bridge inspection dataset, wherein the high-incidence areas of bridge cracks include the connection between bearings and beams, expansion joints, expansion joints, beam ends, cantilever structures, fatigue load concentration areas, and concrete pouring joints;

[0010] Clustering the bridge inspection data set using a mutual information clustering algorithm to obtain a clustering result, wherein the clustering result includes a non-crack image and a crack image;

[0011] Acquire location information of the crack area according to the crack image, and use an ultrasonic probe to collect data on the crack area to acquire ultrasonic data;

[0012] Establishing a multimodal crack data set based on the ultrasonic data and the crack image data, and detecting the multimodal crack data set using a multimodal fusion network to obtain a detection result, wherein the detection result includes true cracks and false cracks;

[0013] The real crack is 3D modeled according to the ultrasonic data and the crack image data to obtain a 3D visualized concrete crack model.

[0014] Furthermore, clustering the bridge inspection data set using a mutual information clustering algorithm includes:

[0015] The images in the bridge inspection data set are converted into grayscale images, and the noise of the grayscale images is removed by using a median filter to obtain a denoised image; the edge, texture and shape features of the denoised image are extracted by using a SIFT algorithm to obtain a feature vector; the mutual information between the feature vectors is calculated to obtain the mutual information value and construct a similarity matrix; the similarity matrix is ​​clustered using a clustering algorithm to identify non-crack images and crack images.

[0016] Furthermore, the clustering algorithm is a hierarchical clustering algorithm, the hierarchical clustering algorithm adopts an average linkage method, and the similarity calculation of the hierarchical clustering algorithm is a Euclidean distance calculation.

[0017] Furthermore, establishing the multimodal crack data set includes: filtering the noise in the ultrasonic data to extract the echo signal; using time-frequency transformation to convert the echo signal from the time domain to the time-frequency domain to obtain a time-frequency two-dimensional matrix; mapping the time-frequency two-dimensional matrix values ​​to grayscale values ​​between 0 and 255 to generate a two-dimensional grayscale image; spatially aligning the two-dimensional grayscale image and its corresponding crack image so that they are in the same coordinate system; matching the two-dimensional grayscale image and its corresponding crack image to establish a multimodal crack data set.

[0018] Furthermore, the time-frequency transform is a short-time Fourier transform, and the short-time Fourier transform formula is:

[0019]

[0020] Where t represents time, f represents frequency, S(t,f) represents the signal components of the time-frequency image in time and frequency, τ represents the time variable, represents the integral of the time variable in the interval, x(τ) represents the signal in the time domain, h(τ_t) represents the short-time window function, e_ j2πfτ Represents the kernel function of the Fourier transform.

[0021] Furthermore, the multimodal fusion network includes: an input layer, a feature extraction layer, a multimodal fusion layer, a decision layer, and an output layer;

[0022] The input layer includes a grayscale image input module and a crack image input module. The grayscale image input module receives a two-dimensional grayscale image and obtains a low-dimensional grayscale feature map through convolution. The crack image input module receives a crack image corresponding to the two-dimensional grayscale image and obtains a low-dimensional crack feature map through convolution.

[0023] The feature extraction layer includes a time-frequency feature extraction module and a space feature extraction module. The time-frequency feature extraction module receives the low-dimensional grayscale feature map and extracts its time-frequency features to obtain a time-frequency feature map. The space feature extraction module receives the low-dimensional crack feature map and extracts its space features to obtain a space feature map.

[0024] The multimodal fusion layer receives the time-frequency feature map and the spatial feature map, and obtains a multimodal fusion feature map after weighted fusion;

[0025] The decision layer receives the multimodal fusion feature map, processes the multimodal fusion feature map through a Sigmoid function, and outputs a classification result;

[0026] The output layer is used to output a final result, which includes true cracks and false cracks.

[0027] Furthermore, performing 3D modeling on the true cracks includes:

[0028] The ultrasonic data and the crack image data are denoised and enhanced; the crack image data is converted into a three-dimensional point cloud through a multi-view stereo vision method, and depth information is extracted from the ultrasonic data to generate an ultrasonic point cloud; the three-dimensional point cloud is registered with the ultrasonic point cloud using a point cloud matching algorithm to obtain registered point cloud data; the registered point cloud data is merged in combination with surface and internal features to generate a 3D point cloud model; the 3D point cloud model is denoised and simplified, and the three-dimensional model is displayed using a visualization tool.

[0029] Further, registering the three-dimensional point cloud with the ultrasonic point cloud using a point cloud matching algorithm includes:

[0030] A preliminary alignment is performed through the sensor position information so that the three-dimensional point cloud and the ultrasonic point cloud are located in the same coordinate system; in the ultrasonic point cloud, each point matches the nearest point in the three-dimensional point cloud to generate a corresponding point pair; by minimizing the error between the two point clouds, a translation matrix and a rotation matrix are solved; the translation matrix and the rotation matrix are applied to the ultrasonic point cloud to update its position and direction; the process of point pair matching and transformation solution is repeated to gradually reduce the error between the three-dimensional point cloud and the ultrasonic point cloud until convergence, and the final transformation matrix is ​​output.

[0031] Furthermore, the method further comprises:

[0032] Using a damage assessment method to conduct damage assessment on the concrete crack model and formulate repair measures, the damage assessment includes mild damage, moderate damage and severe damage;

[0033] The damage assessment method is formulated as follows:

[0034]

[0035] Among them, D total Indicates the degree of structural damage, V d represents the volume of the crack area, V t represents the total volume of the bridge structure, σ max represents the maximum stress at the crack, σ nomrepresents the theoretical stress in the undamaged state, w1 represents the damage ratio weight coefficient, and w2 represents the stress damage weight coefficient;

[0036] The injury assessment includes minor injury, moderate injury and severe injury. total <0.1, it is assessed as slight injury; when 0.1≤D total <0.3, it is assessed as moderate damage; total When ≥0.3, it is assessed as severe injury.

[0037] Further, the remediation measures include:

[0038] Minor damage: Check crack development regularly and use visual inspection and ultrasonic inspection to monitor crack changes;

[0039] Moderate damage: Use polymer-modified mortar to fill cracks, ensure crack surfaces are smooth, and apply a protective coating to prevent moisture penetration;

[0040] Severe damage: Clean severe cracks, remove loose concrete, fill cracks using grouting technology and re-pour concrete.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention proposes a clustering algorithm based on mutual information, which measures similarity and performs clustering by calculating the mutual information between image features. The algorithm can effectively identify crack images and non-crack images, especially showing higher accuracy and robustness when processing complex image data. Compared with traditional methods, the present invention can not only classify crack images more accurately, but also better handle high-dimensional data and noise interference, significantly improving the efficiency and reliability of crack detection.

[0043] 2. The present invention proposes a multimodal fusion network, which can accurately detect cracks in bridge concrete by fusing drone image data and ultrasonic data. The network can fully utilize the advantages of each modal data and enhance the perception and understanding of crack characteristics by extracting and weighted fusion of time-frequency features and spatial features. The multimodal fusion network can effectively improve the detection accuracy, especially when dealing with cracks with complex structures, and can provide more comprehensive and accurate judgments. Compared with the detection method of a single modality, the network effectively improves the accuracy of crack detection.

[0044] 3. The present invention proposes a damage assessment method, which combines the crack image and ultrasonic detection data of the bridge structure and uses numerical calculation to quantitatively assess the severity of the cracks. This method can accurately identify the size and shape of the cracks and their impact on the bearing capacity of the structure, and classify the damage according to the assessment results, and provide corresponding repair suggestions. Compared with traditional empirical assessment methods, the damage assessment method proposed in the present invention is more accurate and objective, and can effectively improve the scientific nature and decision-making efficiency of bridge maintenance, providing reliable protection for the long-term safety of bridge structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a first embodiment of an intelligent detection method for bridge concrete cracks based on a deep neural network provided by the present invention;

[0046] Figure 2 A multi-modal fusion network structure diagram provided by an embodiment of the present invention;

[0047] Figure 3 This is a flow chart of Example 2 of an intelligent detection method for bridge concrete cracks based on a deep neural network provided by the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment 1

[0050] An intelligent detection method for bridge concrete cracks based on deep neural network, such as Figure 1 As shown, including:

[0051] S110: Use drones to inspect images of bridge crack-prone areas and obtain bridge inspection datasets;

[0052] S120: clustering the bridge inspection data set using a mutual information clustering algorithm to obtain a clustering result, where the clustering result includes a non-crack image and a crack image;

[0053] S130: acquiring position information of the crack area according to the crack image, and collecting data of the crack area using an ultrasonic probe to acquire ultrasonic data;

[0054] S140: establishing a multimodal crack data set based on the ultrasonic data and the crack image data, and detecting the multimodal crack data set using a multimodal fusion network to obtain a detection result, where the detection result includes true cracks and false cracks;

[0055] S150: Perform 3D modeling on the real cracks according to the ultrasonic data and the crack image data to obtain a 3D visualized concrete crack model.

[0056] See also Figure 1 S110 in the example uses drones to inspect images of areas with high incidence of bridge cracks and obtains a bridge inspection dataset;

[0057] Specifically, a high-resolution UAV system is selected, equipped with an optical camera to ensure that clear images can be captured; the UAV flight altitude, speed, camera shooting frequency and overlap are set to ensure that high-resolution images are obtained for subsequent analysis. In a feasible implementation plan, the flight altitude is controlled at 30-50 meters, and the horizontal and vertical overlap of the image is maintained at more than 60%; the main inspection targets are determined based on the structural design, age and use of the bridge, with a focus on areas with high crack incidence, including the connection between the support and the beam, expansion joints, expansion joints, beam ends, cantilever structures, fatigue load concentration areas and concrete pouring joints;

[0058] See also Figure 1 In S120, a mutual information clustering algorithm is used to cluster the bridge inspection data set to obtain a clustering result, where the clustering result includes a non-crack image and a crack image;

[0059] Furthermore, clustering the bridge inspection data set using a mutual information clustering algorithm includes: converting the images in the bridge inspection data set into grayscale images, using a median filter to remove noise from the grayscale images, and obtaining a denoised image; using a SIFT algorithm to extract edge, texture, and shape features of the denoised image to obtain a feature vector; calculating the mutual information between the feature vectors, obtaining a mutual information value and constructing a similarity matrix; and using a clustering algorithm to perform cluster analysis on the similarity matrix to identify non-crack images and crack images.

[0060] Specifically, the RGB color image is converted into a grayscale image using a weighted method, and the median filtering technique is used to replace the value of the central pixel with the median value in the local window. The Canny edge detection algorithm is used to extract the edge information of the crack, the gray-level co-occurrence matrix is ​​used to extract the texture features of the image, and the shape features of the crack are extracted based on morphological processing. All the extracted features are combined to form a feature vector for similarity calculation. A clustering algorithm is used to perform image clustering based on the similarity matrix. The number of clusters is determined by calculating the similarity between images, maximizing the intra-class similarity or minimizing the inter-class similarity, and the images are divided into two categories: crack images and non-crack images to obtain the final classification results.

[0061] The mutual information clustering algorithm is used to quantify the degree of shared information between two data sets, which is particularly suitable for situations where the feature space is complex and nonlinearly related. Compared with traditional distance measurement methods, mutual information can better capture hidden features and patterns in images or data and improve the accuracy of clustering. In addition, it does not rely on specific feature distributions, is suitable for the fusion analysis of multimodal data, and helps to process high-dimensional or heterogeneous data to achieve more accurate clustering results.

[0062] Furthermore, the clustering algorithm is a hierarchical clustering algorithm, the hierarchical clustering algorithm adopts an average linkage method, and the similarity calculation of the hierarchical clustering algorithm is a Euclidean distance calculation.

[0063] Specifically, the hierarchical clustering algorithm initializes each image sample as a separate cluster, finds the two clusters with the largest mutual information similarity in the current cluster set, and merges them into one cluster, where the similarity is calculated using Euclidean distance. At the same time, the corresponding similarity matrix also needs to be updated, and the average link method is used to update the similarity between different clusters after merging. Repeat this merging process until all samples are finally merged into a large cluster. The merging process can generate a cluster tree that shows the hierarchical relationship between each sample. By analyzing the cluster tree, it can be decided where to truncate the cluster tree according to actual needs, and then determine the final number of clusters.

[0064] The hierarchical clustering algorithm and the average linkage method for similarity calculation can effectively capture the hierarchical structure of different clusters in the data, and the similarity between samples is measured by Euclidean distance, which ensures the intuitiveness and accuracy of clustering. The average linkage method avoids the chain effect of the single linkage method and the overly tight clustering problem of the complete linkage method by calculating the average distance between sample pairs between clusters, making the clustering results more stable and reliable.

[0065] See also Figure 1 In S130, position information of the crack area is obtained according to the crack image, and data of the crack area is collected using an ultrasonic probe to obtain ultrasonic data;

[0066] Specifically, the image data obtained by the drone is combined with the physical model of the bridge structure, and GPS is used to locate the specific position of the crack on site, marking the starting and end points of the crack area. According to the size and depth of the crack, an ultrasonic probe with a suitable frequency and mode is selected. The probe frequency usually used to detect cracks is between 2.5MHz and 10MHz. The ultrasonic probe is placed on the surface of the marked crack area to ensure that the probe is in full contact with the concrete surface, and a coupling agent is used to reduce the loss of sound wave reflection. The ultrasonic detection equipment is started to emit ultrasonic pulses and receive ultrasonic data.

[0067] See also Figure 1 In S140, a multimodal crack data set is established based on the ultrasonic data and the crack image data, and a multimodal fusion network is used to detect the multimodal crack data set to obtain a detection result, where the detection result includes a true crack and a false crack;

[0068] Furthermore, establishing the multimodal crack data set includes: filtering the noise in the ultrasonic data to extract the echo signal; using time-frequency transformation to convert the echo signal from the time domain to the time-frequency domain to obtain a time-frequency two-dimensional matrix; mapping the time-frequency two-dimensional matrix values ​​to grayscale values ​​between 0 and 255 to generate a two-dimensional grayscale image; spatially aligning the two-dimensional grayscale image and its corresponding crack image so that they are in the same coordinate system; matching the two-dimensional grayscale image and its corresponding crack image to establish a multimodal crack data set.

[0069] Specifically, high-pass filtering is used to process ultrasonic data to remove noise components and obtain echo signals; time-frequency transformation is performed on the echo signals, and a time-frequency two-dimensional matrix is ​​generated by performing spectral analysis on the echo signals in multiple time periods; the values ​​of the time-frequency two-dimensional matrix are normalized, and the maximum value is set to 255 and the minimum value is set to 0, and the numerical range of the matrix is ​​mapped to the grayscale value range; affine transformation registration technology is used to spatially match the ultrasonic grayscale image and the crack image to ensure that the geometric shape, position and size of the images match and that the crack area corresponds accurately in the two images; the aligned and matched grayscale image and the crack image are jointly stored as a multimodal crack dataset, and each data item includes the corresponding external image features and internal ultrasonic features of the crack.

[0070] The multimodal fusion method based on ultrasonic data and crack images can simultaneously capture the external morphology and internal structural information of the cracks, and achieves accurate processing of ultrasonic signals through noise filtering, time-frequency transformation, and grayscale image generation. By aligning and matching the grayscale image with the crack image space, a multimodal crack dataset containing visual and structural information is established, thereby improving the accuracy and reliability of crack detection and providing more comprehensive data support for subsequent damage assessment and repair strategies.

[0071] Furthermore, the time-frequency transform is a short-time Fourier transform, and the short-time Fourier transform formula is:

[0072]

[0073] Where t represents time, f represents frequency, S(t,f) represents the signal components of the time-frequency image in time and frequency, τ represents the time variable, represents the integral of the time variable in the interval, x(τ) represents the signal in the time domain, h(τ_t) represents the short-time window function, e_ j2πfτ Represents the kernel function of the Fourier transform.

[0074] Converting ultrasonic echo signals into the time-frequency domain through short-time Fourier transform can effectively capture the frequency changes of the signal in different time periods, thereby providing the time-frequency characteristics of the signal. This process can simultaneously display the time and frequency information of the ultrasonic signal, which helps to better analyze the local characteristics and changes in the crack area. Compared with direct analysis in the time domain or frequency domain, short-time Fourier transform provides richer features for the dynamic structural changes of the crack, which helps to more accurately identify and evaluate the depth and morphology of the crack.

[0075] Furthermore, the multimodal fusion network is Figure 2 As shown, it includes: input layer, feature extraction layer, multimodal fusion layer, decision layer and output layer;

[0076] The input layer includes a grayscale image input module and a crack image input module. The grayscale image input module receives a two-dimensional grayscale image and obtains a low-dimensional grayscale feature map through convolution. The crack image input module receives a crack image corresponding to the two-dimensional grayscale image and obtains a low-dimensional crack feature map through convolution.

[0077] The feature extraction layer includes a time-frequency feature extraction module and a space feature extraction module. The time-frequency feature extraction module receives the low-dimensional grayscale feature map and extracts its time-frequency features to obtain a time-frequency feature map. The space feature extraction module receives the low-dimensional crack feature map and extracts its space features to obtain a space feature map.

[0078] The multimodal fusion layer receives the time-frequency feature map and the spatial feature map, and obtains a multimodal fusion feature map after weighted fusion;

[0079] The decision layer receives the multimodal fusion feature map, processes the multimodal fusion feature map through a Sigmoid function, and outputs a classification result;

[0080] The output layer is used to output a final result, which includes true cracks and false cracks.

[0081] Specifically, the grayscale image input module receives the two-dimensional grayscale image generated by time-frequency transformation, performs convolution operation through a 3×3 convolution kernel, extracts a low-dimensional grayscale feature map, and captures the time-frequency characteristics of the grayscale image. The crack image input module receives the crack image matching the grayscale image, performs convolution operation through a 3×3 convolution kernel, extracts a low-dimensional crack feature map, and captures the spatial characteristics of the crack image; the time-frequency feature extraction module receives the low-dimensional grayscale feature map, uses the LSTM module to further extract its time-frequency characteristics, generates a time-frequency feature map, and displays the characteristics of the crack signal changing with time and frequency; the spatial feature extraction module receives the low-dimensional crack feature map, uses the CBAM module to further extract the spatial characteristics of the crack, generates a spatial feature map, and captures the morphology, texture and edge information of the crack; The state fusion layer integrates the time-frequency feature map and the spatial feature map through weighted fusion, combines the information of different modes, and obtains a more comprehensive multimodal fusion feature map; the multimodal fusion feature map is passed to the decision layer, and the output is normalized into a probability value through the Sigmoid activation function, indicating the probability that each area is a true crack or a false crack; the cross entropy loss function is used to calculate the gap between the model classification result and the true label to determine the error size of the model; the Adam optimizer is used to adjust the weights of each layer of the network through the back propagation algorithm, minimize the value of the loss function, and gradually improve the detection accuracy of the model; the output layer classifies the cracks according to their time-frequency and spatial characteristics, and outputs the classification results calculated by the model, including the judgment of whether the cracks are true cracks or false cracks.

[0082] The multimodal fusion network combines ultrasonic data and crack image data. Through the weighted fusion of time-frequency features and spatial features, it can comprehensively capture the surface morphology and internal structure information of the cracks and provide a more comprehensive crack feature expression. Compared with the single-modality detection method, multimodal fusion effectively improves the accuracy and robustness of crack identification, especially in complex environments. It can more accurately distinguish between true cracks and false cracks, providing a reliable basis for bridge structure health monitoring and subsequent repair decisions.

[0083] See also Figure 1 S150 in the system performs 3D modeling of real cracks based on ultrasonic data and crack image data to obtain a 3D visualized concrete crack model.

[0084] Furthermore, 3D modeling of the real crack includes: denoising and enhancing the ultrasonic data and the crack image data; converting the crack image data into a three-dimensional point cloud through a multi-view stereo vision method, and extracting depth information from the ultrasonic data to generate an ultrasonic point cloud; using a point cloud matching algorithm to align the three-dimensional point cloud with the ultrasonic point cloud to obtain aligned point cloud data; combining the aligned point cloud data with surface and internal features to generate a 3D point cloud model; denoising and simplifying the 3D point cloud model, and using visualization tools to display the three-dimensional model.

[0085] Specifically, the ultrasonic data and crack image data are denoised, and image enhancement technology is used to optimize image quality to improve the effect of subsequent feature extraction; multiple crack image data are converted into three-dimensional point clouds to reflect the spatial geometric information of the structure surface, and depth information is extracted from the ultrasonic data to generate a three-dimensional ultrasonic point cloud reflecting the interior of the structure; the three-dimensional point cloud and the ultrasonic point cloud are accurately registered using a point cloud matching algorithm to ensure that the two are aligned in the same coordinate system; the registered three-dimensional point cloud is fused with the ultrasonic point cloud, and the surface and internal structural information are combined to generate a comprehensive 3D point cloud model containing surface geometric features and internal depth data; the fused 3D point cloud model is denoised, and a simplified algorithm is used to reduce point cloud redundancy to improve the efficiency and operability of the model.

[0086] 3D modeling of real cracks can fully and accurately capture the geometric characteristics of cracks, which not only provides a more intuitive visualization effect, but also facilitates quantitative analysis of the severity of cracks. Through 3D modeling, we can better understand the development of cracks in space, support the accurate assessment of structural damage and the formulation of repair strategies, and improve the efficiency and accuracy of health monitoring of bridges and other structures.

[0087] Furthermore, the three-dimensional point cloud and the ultrasonic point cloud are registered using a point cloud matching algorithm, including: performing preliminary alignment through sensor position information so that the three-dimensional point cloud and the ultrasonic point cloud are located in the same coordinate system; in the ultrasonic point cloud, each point matches the nearest point in the three-dimensional point cloud to generate a corresponding point pair; by minimizing the error between the two point clouds, a translation matrix and a rotation matrix are solved; the translation matrix and the rotation matrix are applied to the ultrasonic point cloud to update its position and direction; and the process of point pair matching and transformation solving is repeated to gradually reduce the error between the three-dimensional point cloud and the ultrasonic point cloud until convergence, and the final transformation matrix is ​​output.

[0088] The point cloud matching algorithm can efficiently align point cloud data from different sources and ensure that they are fused in the same coordinate system. Through the point cloud matching algorithm, the error between point clouds can be minimized and a high-precision registration effect can be achieved. This not only improves the accuracy of data fusion, but also enhances the integrity of 3D modeling, especially in the comprehensive expression of surface and internal structure information, which helps to more comprehensively analyze the morphology and location of cracks.

[0089] Images of areas prone to bridge cracks are taken by drones, and the mutual information clustering algorithm is used to classify the images to distinguish crack images from non-crack images. The specific location of the cracks is determined based on the crack images, and ultrasonic probes are used to collect data in the crack area. A multimodal crack dataset is established by combining ultrasonic data and image data, and a multimodal fusion network is used to detect true cracks and false cracks. Finally, 3D modeling is performed on true cracks to generate a visual crack model to achieve accurate detection and comprehensive evaluation of bridge cracks.

[0090] Embodiment 2

[0091] As another optional embodiment of the present application, refer to Figure 3 , is a flow chart of a second embodiment of an intelligent detection method for bridge concrete cracks based on a deep neural network provided by the present application. This embodiment is mainly an extension of the intelligent detection method described in the first embodiment above, such as Figure 3 As shown, the method includes:

[0092] S210: Use drones to inspect images of bridge crack-prone areas and obtain bridge inspection datasets;

[0093] S220: clustering the bridge inspection data set using a mutual information clustering algorithm to obtain a clustering result, wherein the clustering result includes a non-crack image and a crack image;

[0094] S230: acquiring position information of the crack area according to the crack image, and collecting data of the crack area using an ultrasonic probe to acquire ultrasonic data;

[0095] S240: establishing a multimodal crack data set based on the ultrasonic data and the crack image data, and detecting the multimodal crack data set using a multimodal fusion network to obtain a detection result, where the detection result includes true cracks and false cracks;

[0096] S250: Perform 3D modeling of the real cracks according to the ultrasonic data and the crack image data to obtain a 3D visualized concrete crack model;

[0097] S260: Use the damage assessment function to conduct damage assessment on the concrete crack model and formulate repair measures. The damage assessment includes mild damage, moderate damage and severe damage.

[0098] The detailed process of steps S210-S250 can refer to the relevant introduction of steps S110-S150 in the first embodiment, which will not be repeated here.

[0099] See also Figure 3 S260, using a damage assessment method to perform damage assessment on the concrete crack model and formulate repair measures, wherein the damage assessment includes light damage, moderate damage and severe damage;

[0100] The damage assessment method is formulated as follows:

[0101]

[0102] Among them, D total Indicates the degree of structural damage, V d represents the volume of the crack area, V t represents the total volume of the bridge structure, σ max represents the maximum stress at the crack, σ nom represents the theoretical stress in the undamaged state, w1 represents the damage ratio weight coefficient, and w2 represents the stress damage weight coefficient;

[0103] The injury assessment includes minor injury, moderate injury and severe injury. total <0.1, it is assessed as slight injury; when 0.1≤D total <0.3, it is assessed as moderate damage; total When ≥0.3, it is assessed as severe injury.

[0104] The damage assessment function can be used to quantitatively analyze structural defects such as bridge cracks, and accurately assess their severity by calculating the impact of cracks on the overall strength and stability of the structure. This method helps determine the risk level of the damaged area, supports prioritizing the treatment of severely damaged areas, formulates reasonable repair plans, extends the service life of the structure, and reduces maintenance costs and safety hazards. This precise assessment method is more scientific and reliable than traditional subjective judgment.

[0105] Further, the remediation measures include:

[0106] Minor damage: Check crack development regularly and use visual inspection and ultrasonic inspection to monitor crack changes;

[0107] Moderate damage: Use polymer-modified mortar to fill cracks, ensure crack surfaces are smooth, and apply a protective coating to prevent moisture penetration;

[0108] Severe damage: Clean severe cracks, remove loose concrete, fill cracks using grouting technology and re-pour concrete.

[0109] In this embodiment, a set of crack data and stress data at the crack of a bridge structure is shown in Table 1, where the damage ratio weight coefficient w1=0.4, the stress damage weight coefficient w2=0.6, and based on the damage assessment method, the structural damage degree at the connection between the support and the beam is calculated as:

[0110]

[0111] According to the damage assessment, it is determined to be moderate damage. Therefore, polymer modified mortar can be used to fill the cracks during repair, ensure that the crack surface is smooth, and apply a protective coating to prevent moisture penetration; based on the damage assessment method, it can be calculated that the structural damage degree of the beam end is 0.08, and the damage assessment is determined to be minor damage. The crack development can be checked regularly, and visual inspection and ultrasonic inspection can be used to monitor changes in the cracks; based on the damage assessment method, it can be calculated that the damage degree of the cantilever structure is 0.40, and the damage assessment is determined to be severe damage. During repair, the severe cracks should be cleaned, the loose concrete should be removed, the cracks should be filled using grouting technology, and the concrete should be re-poured.

[0112] Table 1. Crack data and stress data at cracks of a group of bridge structures

[0113]

[0114] Repair measures are formulated to carry out graded treatment according to the degree of damage to ensure targeted repair and long-term structural stability. Minor damage is monitored through regular inspections and multiple tests to prevent crack expansion; moderate damage is effectively sealed by polymer-modified mortar filling and protective coating treatment to prevent water infiltration and further damage; severe damage is completely repaired by grouting technology and re-poured concrete to restore its bearing capacity and service life. This graded repair plan can maximize the safety of bridge structures, extend service life, and reduce maintenance costs.

[0115] Embodiment 3

[0116] A bridge inspection team in a certain city used a deep neural network-based intelligent detection method for bridge concrete cracks proposed in the present invention when conducting regular inspections of bridges. Before applying the intelligent detection method proposed in the present invention, the team used a detection method based on U-Net.

[0117] Table 2. Comparison data before and after application of the intelligent detection method proposed by the present invention

[0118]

[0119] As shown in Table 2, before applying the intelligent detection method proposed in the present invention, the recognition rate of crack images by the detection method based on U-Net is 93%, which is slightly higher than the recognition rate after the application of the intelligent detection method. The main reason is that the clustering algorithm based on mutual information can only cluster the drone images according to basic features, while the U-Net based on deep neural network can learn more features and thus show a higher recognition rate. In the data comparison of identifying true cracks, the recognition rate of the intelligent detection method proposed in the present invention is significantly higher than that of the detection method based on U-Net. The reason for this phenomenon is that the multimodal fusion network that fuses ultrasonic data and drone data can comprehensively capture the surface shape of the cracks. state and internal structure information, thereby obtaining a more accurate recognition effect; in terms of image processing speed, the intelligent detection method proposed in the present invention is significantly higher than the detection method based on U-Net. The main reason is that the clustering algorithm based on mutual information can pre-cluster the bridge inspection data set into crack images and non-crack images, thereby improving the recognition efficiency of the multimodal fusion network for true cracks; finally, the intelligent detection method proposed in the present invention also has a great advantage in the degree of defect repair, which mainly depends on the method's ability to propose a damage assessment method based on the 3D concrete crack model and perform damage assessment, while providing different repair measures according to different degrees of damage to maximize the safety of the bridge structure.

[0120] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent detection method for bridge concrete cracks based on deep neural network, characterized in that: include: Using drones to inspect images of high-incidence areas of bridge cracks to obtain a bridge inspection dataset, wherein the high-incidence areas of bridge cracks include the connection between bearings and beams, expansion joints, expansion joints, beam ends, cantilever structures, fatigue load concentration areas, and concrete pouring joints; Clustering the bridge inspection data set using a mutual information clustering algorithm to obtain a clustering result, wherein the clustering result includes a non-crack image and a crack image; Acquire location information of the crack area according to the crack image, and use an ultrasonic probe to collect data on the crack area to acquire ultrasonic data; Establishing a multimodal crack data set based on the ultrasonic data and the crack image data, and detecting the multimodal crack data set using a multimodal fusion network to obtain a detection result, wherein the detection result includes true cracks and false cracks; The real crack is 3D modeled according to the ultrasonic data and the crack image data to obtain a 3D visualized concrete crack model.

2. According to claim 1, a method for intelligent detection of bridge concrete cracks based on deep neural network is characterized in that: Clustering the bridge inspection data set using the mutual information clustering algorithm includes: Converting the image in the bridge inspection data set into a grayscale image, and removing the noise of the grayscale image using a median filter to obtain a denoised image; Using SIFT algorithm to extract edge, texture and shape features of the denoised image to obtain a feature vector; Calculate the mutual information between feature vectors, obtain the mutual information value and construct a similarity matrix; A clustering algorithm is used to perform cluster analysis on the similarity matrix to identify non-crack images and crack images.

3. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 2 is characterized in that: The clustering algorithm is a hierarchical clustering algorithm, the hierarchical clustering algorithm adopts an average link method, and the similarity calculation of the hierarchical clustering algorithm is Euclidean distance calculation.

4. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 1 is characterized in that: Establishing the multimodal crack data set includes: Filtering the noise in the ultrasonic data to extract the echo signal; Convert the echo signal from the time domain to the time-frequency domain using a time-frequency transform to obtain a two-dimensional time-frequency matrix; Mapping the time-frequency two-dimensional matrix values ​​to grayscale values ​​between 0 and 255 to generate a two-dimensional grayscale image; Spatially aligning the two-dimensional grayscale image and its corresponding crack image so that they are in the same coordinate system; The two-dimensional grayscale image and its corresponding crack image are matched to establish a multimodal crack data set.

5. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 4 is characterized in that: The time-frequency transform is short-time Fourier transform, and the short-time Fourier transform formula is: Where t represents time, f represents frequency, S(t,f) represents the signal components of the time-frequency image in time and frequency, τ represents the time variable, represents the integral of the time variable in the interval, x(τ) represents the signal in the time domain, h(τ_t) represents the short-time window function, e_ j2πfτ Represents the kernel function of the Fourier transform.

6. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 1 is characterized in that: The multimodal fusion network includes: input layer, feature extraction layer, multimodal fusion layer, decision layer and output layer; The input layer includes a grayscale image input module and a crack image input module. The grayscale image input module receives a two-dimensional grayscale image and obtains a low-dimensional grayscale feature map through convolution. The crack image input module receives a crack image corresponding to the two-dimensional grayscale image and obtains a low-dimensional crack feature map through convolution. The feature extraction layer includes a time-frequency feature extraction module and a space feature extraction module. The time-frequency feature extraction module receives the low-dimensional grayscale feature map and extracts its time-frequency features to obtain a time-frequency feature map. The space feature extraction module receives the low-dimensional crack feature map and extracts its space features to obtain a space feature map. The multimodal fusion layer receives the time-frequency feature map and the spatial feature map, and obtains a multimodal fusion feature map after weighted fusion; The decision layer receives the multimodal fusion feature map, processes the multimodal fusion feature map through a Sigmoid function, and outputs a classification result; The output layer is used to output a final result, which includes true cracks and false cracks.

7. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 1 is characterized in that: The 3D modeling of the true cracks includes: Denoising and enhancing the ultrasonic data and the crack image data; The crack image data is converted into a three-dimensional point cloud by a multi-view stereo vision method, and depth information is extracted from the ultrasonic data to generate an ultrasonic point cloud; Using a point cloud matching algorithm to register the three-dimensional point cloud with the ultrasonic point cloud to obtain registered point cloud data; Merging the registered point cloud data in combination with surface and internal features to generate a 3D point cloud model; The 3D point cloud model is subjected to noise reduction and simplification processing, and a visualization tool is used to display the three-dimensional model.

8. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 7 is characterized in that: Using a point cloud matching algorithm to register the three-dimensional point cloud with the ultrasonic point cloud comprises: Performing preliminary alignment through sensor position information so that the three-dimensional point cloud and the ultrasonic point cloud are located in the same coordinate system; In the ultrasonic point cloud, each point is matched with the nearest point in the three-dimensional point cloud to generate a corresponding point pair; By minimizing the error between the two point clouds, the translation matrix and rotation matrix are solved; Applying the translation matrix and the rotation matrix to the ultrasonic point cloud to update its position and orientation; The process of point pair matching and transformation solving is repeated to gradually reduce the error between the three-dimensional point cloud and the ultrasonic point cloud until convergence, and the final transformation matrix is ​​output.

9. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 1 is characterized in that: The method further comprises: Using a damage assessment method to conduct damage assessment on the concrete crack model and formulate repair measures, the damage assessment includes mild damage, moderate damage and severe damage; The damage assessment method is formulated as follows: Among them, D total Indicates the degree of structural damage, V d represents the volume of the crack area, V t represents the total volume of the bridge structure, σ max represents the maximum stress at the crack, σ nom represents the theoretical stress in the undamaged state, w1 represents the damage ratio weight coefficient, and w2 represents the stress damage weight coefficient; The injury assessment includes minor injury, moderate injury and severe injury. total <0.1, it is assessed as slight injury; when 0.1≤D total <0.3, it is assessed as moderate damage; total When ≥0.3, it is assessed as severe injury.

10. The intelligent detection method for bridge concrete cracks based on deep neural network according to claim 9 is characterized in that: The corrective measures include: Minor damage: Check crack development regularly and use visual inspection and ultrasonic inspection to monitor crack changes; Moderate damage: Use polymer-modified mortar to fill cracks, ensure crack surfaces are smooth, and apply a protective coating to prevent moisture penetration; Severe damage: Clean severe cracks, remove loose concrete, fill cracks using grouting technology and re-pour concrete.

Images