A crack depth prediction method based on an infrared thermal imager and deep learning

Abstract

The application provides a crack depth prediction method based on an infrared thermal imager and deep learning, relates to the technical field of crack prediction, and through synchronous acquisition of a cooling infrared thermal image sequence of a crack region under thermal excitation and a visible light image, multi-modal features are extracted after registration and fusion; further, based on consistency verification of thermal conduction inversion, width-depth empirical relationship and traditional detection means, high-confidence crack depth labels are automatically generated to construct a labeled data set; finally, a deep learning model containing attention enhancement and multi-scale fusion is designed to learn the mapping relationship between input images and depth, and precise crack depth prediction is realized. The application introduces a multi-source information consistency verification mechanism, mutually verifies physical model inversion, empirical statistics and measured data, effectively solves the bottleneck problem of difficulty in obtaining high-quality depth labels in engineering practice, and provides reliable training data basis for the deep learning model.

Description

Technical Field

[0001] This invention relates to the field of crack prediction technology, specifically a crack depth prediction method based on infrared thermal imager and deep learning. Background Technology

[0002] Cracks are the most common and damaging form of structural damage in civil engineering infrastructure such as bridges, tunnels, historical buildings, and high-grade pavements during their long-term service life. Cracks not only compromise the integrity and aesthetics of the structure but also act as conduits for moisture and corrosive media, accelerating the corrosion of internal steel reinforcement and material deterioration. Failure to detect and assess cracks in a timely manner can lead to a decrease in structural load-bearing capacity and even catastrophic accidents. Therefore, accurate and rapid detection and assessment of cracks, especially their depth, is of crucial engineering value and practical significance for evaluating structural health, predicting remaining life, and developing preventative maintenance strategies.

[0003] Currently, mainstream crack depth detection methods in the engineering field mainly fall into two categories. The first is traditional manual inspection and visible light-based machine vision inspection. These methods are inefficient, highly subjective, and can only acquire the surface two-dimensional geometry of the crack (such as length and width), failing to address the core indicator determining structural safety: depth information. The second is non-destructive or minimal-destructive testing techniques based on ultrasound, ground-penetrating radar, or core drilling. For example, while ultrasonic pulse-echo methods can assess depth, they are easily affected by interference from internal steel reinforcement, require coupling agents, and have low detection efficiency. Core drilling methods can cause localized damage to the structure, making large-scale application difficult. These methods all struggle to achieve non-contact, efficient, and highly accurate quantitative assessment of crack depth, especially when dealing with complex structures that are large-scale, high-altitude, or inaccessible to personnel.

[0004] In recent years, infrared thermal imaging technology has emerged as a promising field for structural defect identification due to its advantages of non-contact, large-area, and rapid detection. This technology identifies internal defects by sensing changes in the temperature field of a structural surface under thermal excitation. Studies have shown that defects such as cracks can block heat conduction, thereby creating an abnormal temperature field on the surface. However, most existing infrared detection methods remain at the level of qualitative identification of cracks or delineation of surface morphology.

[0005] How to isolate environmental interference from transient, multi-dimensional thermal image sequences, accurately extract weak thermal feature signals strongly correlated with crack depth, and effectively fuse them with surface geometric features to establish a quantitative inversion model that can measure depth from what is visible, is a key technical bottleneck that urgently needs to be overcome in this field. Furthermore, relying solely on physical models for inversion results in multiple solutions and limitations, while deep learning methods, although possessing powerful nonlinear mapping capabilities, are highly dependent on high-quality training datasets with accurate depth labels, which are extremely difficult to obtain in practical engineering.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a crack depth prediction method based on infrared thermal imager and deep learning, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A crack depth prediction method based on infrared thermal imager and deep learning, comprising the following steps:

[0010] Step 1: At the same detection location, simultaneously acquire infrared thermal image sequence and visible light image of the crack area. The infrared thermal image sequence is a thermal image of the crack area cooling process acquired under thermal excitation.

[0011] Step 2: Determine the keyframes in the infrared thermal image sequence and crop out the region of interest of the keyframes. Spatially register and stitch the region of interest with the visible light image to obtain the multimodal fusion features of the dual-channel input image and the crack region. The multimodal fusion features include infrared thermal features and visible light geometric features.

[0012] Step 3: Based on multimodal fusion features, obtain multiple estimates of crack depth using at least two different depth estimation methods, perform consistency checks on the multiple estimates, and when the consistency meets the preset requirements, determine the depth label of the crack region and construct a labeled dataset;

[0013] Step 4: Build a deep learning model, train it using a labeled dataset, and learn the mapping relationship between the dual-channel input image and the depth label;

[0014] Step 5: Acquire infrared thermal image sequence and visible light image of the structure to be tested, and obtain dual-channel input image after preprocessing. Input the dual-channel input image into the trained deep learning model to obtain the crack depth prediction value.

[0015] Furthermore, the infrared thermal imager and the visible light camera are fixed on the same detection platform, and the optical axes of the lenses of the two devices are arranged in parallel or jointly calibrated before acquisition to ensure that the two images have overlapping fields of view.

[0016] Furthermore, before data acquisition, a thermal excitation device is used to heat the crack area. During the heating process, the surface temperature of the crack area is monitored in real time using an infrared thermal imager. When the temperature at the center point of the crack area reaches a preset threshold or the rate of temperature rise first shows a decreasing inflection point, the heating stage is determined to be over and thermal excitation is stopped immediately. The thermal excitation device is a low-power hot air gun, and the heating temperature is controlled within 40 degrees Celsius. The heating time is adjusted according to the thermal diffusivity of the material to which the crack area belongs; the greater the thermal diffusivity, the longer the heating time.

[0017] Immediately after heating is completed, the infrared thermal imager is activated to acquire a sequence of infrared thermal images of the crack area during natural cooling at a fixed frame rate. At the same time, the visible light camera is triggered to acquire a visible light image that is aligned with the start frame time of the infrared thermal image sequence.

[0018] Furthermore, the temperature difference between the crack region and the adjacent background region in each frame of the infrared thermal image sequence is calculated, and the image frame corresponding to the maximum temperature difference is taken as the key frame with the most significant thermal contrast.

[0019] Gaussian filtering was applied to the selected keyframe infrared thermal image and visible light image for noise reduction.

[0020] A contrast-limited adaptive histogram equalization algorithm is used to enhance the local contrast of the crack region in the keyframe infrared thermal image.

[0021] The region of interest (ROI) of the keyframe infrared thermal image is cropped with the crack as the center and denoted as the infrared ROI image. The infrared ROI image and the visible light image are spatially registered using a feature point-based registration method. Based on the registration result, the image region at the same position as the infrared ROI image is cropped from the visible light image and denoted as the visible light crack region image.

[0022] The infrared region of interest (ROI) image and the visible light crack region image are scaled to the same pixel size. The infrared ROI image is normalized based on the temperature value of the infrared ROI image, and the pixel values ​​of the visible light crack region image are linearly normalized to the [0,1] interval.

[0023] During the acquisition process, the acquisition time corresponding to each frame of infrared thermal image, as well as the ambient temperature and humidity parameters corresponding to that acquisition time, are recorded.

[0024] Furthermore, the logic for cropping the infrared region of interest image is as follows:

[0025] A crack instance segmentation model is used to extract pixel-level masks of cracks. The minimum bounding rectangle of the mask is calculated, and a preset pixel range is extended outward based on the minimum bounding rectangle to define the region of interest (ROI). This ROI ensures that the ROI completely includes the crack body and the heat-affected zone around the crack. When there are multiple cracks in the image, the corresponding ROI is defined independently for each crack. The defined ROI is then cropped from the corresponding infrared thermal image, and the cropped image is the infrared ROI image.

[0026] Furthermore, the logic for determining multimodal fusion features is as follows:

[0027] Based on the infrared region of interest image after spatial registration, cropping and scaling, the temperature gradient value of each pixel is extracted, the rate of change of temperature gradient in the horizontal and vertical directions is calculated, and the average temperature, maximum temperature, minimum temperature and temperature drop rate of the crack region are calculated, which together constitute the infrared thermal features.

[0028] On the visible light crack region image, a crack instance segmentation model is used to extract the pixel-level mask of the crack. After binarizing the pixel-level mask, the center line of the crack is extracted by the skeletonization algorithm. The crack pixel length is obtained by counting the number of skeleton pixels. The width at each position is calculated by searching the mask boundary along the normal direction of the skeleton line. The average width and maximum width of the crack are obtained. At the same time, the crack pixel area is obtained by counting the total number of pixels in the mask. Together, they constitute the visible light geometric features.

[0029] Furthermore, the logic for performing consistency checks is as follows:

[0030] Based on the temperature gradient and temperature drop rate in the infrared thermal features, and combined with the recorded ambient temperature and humidity parameters at the acquisition time as boundary conditions, a heat conduction inverse problem model is established. A thermal response database under different depth conditions is constructed through finite element analysis. The measured infrared thermal features are matched and inverted with the database to obtain the first estimate of the crack depth.

[0031] Based on the maximum crack width in the visible light geometric characteristics, a second estimate of the crack depth is obtained by using a pre-established empirical relationship between width and depth through linear regression or power function fitting; the empirical relationship between width and depth is derived from the statistical data of destructive tests on specimens of the same material type.

[0032] The crack depth at the detection location is measured using the ultrasonic pulse echo method or core drilling method to obtain a third estimate of the crack depth.

[0033] The first, second, and third estimates are compared, and their mean and standard deviation are calculated. When the ratio of the standard deviation to the mean is less than 10%, the estimate is determined to have high confidence, and the mean is used as the depth label for the crack region. When the ratio of the standard deviation to the mean is not less than 10%, a verification mechanism is triggered. The verification mechanism includes adjusting the boundary conditions of the inverse heat conduction model and repeating the above steps until the requirement is met, at which point the depth label is determined.

[0034] The infrared region of interest image and the visible light crack region image are associated and stored with their corresponding depth labels to form a labeled dataset for training deep learning models.

[0035] Furthermore, learning the mapping relationship specifically includes:

[0036] Infrared region of interest images and visible light crack region images were obtained from a labeled dataset. The two images were spatially registered and scaled to the same pixel size.

[0037] The infrared region of interest image and the visible light crack region image are stitched together along the channel dimension to form a dual-channel input image. The first channel is the registered infrared region of interest image, and the second channel is the visible light crack region image. At the same time, the depth label is used as the regression target value.

[0038] A residual network is used as the backbone network, including an initial convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer, as well as four stages of residual block groups, to extract deep features at different levels from a dual-channel input image.

[0039] Dual-path attention enhancement modules are embedded after the third and fourth stages of the backbone network. These modules are designed for the unique temperature field distribution caused by thermal diffusion in the crack region. They adaptively recalibrate the feature map through parallel channel attention paths and spatial attention paths: the channel attention path recalibrates the weights of each channel in the feature map to strengthen the thermal gradient feature channels that are strongly correlated with the crack depth; the spatial attention path recalibrates the weights of each spatial location in the feature map to focus on the crack body and the surrounding thermally affected area. The weights of the two recalibrated modules are then fused and multiplied with the feature map to output the enhanced crack region feature map.

[0040] The feature maps output from the second, third, and fourth stages of the backbone network are input into the multi-scale feature fusion module. The feature map of the second stage retains spatial details including crack edges and morphology, while the feature map of the fourth stage includes semantic information of the overall heat diffusion pattern. The feature maps of each stage are adjusted to the same spatial size through upsampling, the number of channels is unified through convolutional layers, and feature fusion is performed through element-wise addition to output a fused feature map containing multi-level semantic information.

[0041] Global average pooling is performed on the fused feature map. The pooled features are then sequentially input into the first fully connected layer and the second fully connected layer. The first fully connected layer reduces the feature dimension to 128 dimensions and connects it to the activation function. The second fully connected layer maps the 128-dimensional features to a one-dimensional crack depth prediction value.

[0042] Using mean squared error as the loss function, an optimizer is used to iteratively optimize the network parameters. During training, the change in the loss value on the validation set is monitored. When the validation set loss does not decrease for several consecutive training cycles, training is terminated early, and the model with the lowest validation set loss is saved as the final deep learning model.

[0043] Furthermore, obtaining the predicted crack depth value of the structure under test specifically includes:

[0044] In the target detection area of ​​the structure to be tested, the same acquisition platform and parameter settings as in step 1 are used to simultaneously acquire infrared thermal image sequences and visible light images of the surface where the crack is located.

[0045] Keyframes were determined from the acquired infrared thermal image sequence, and the same cropping, preprocessing, and registration methods were used to spatially register the infrared region of interest image of the structure under test with the visible light crack region image.

[0046] After spatial registration, the infrared region of interest image of the structure under test and the visible light crack region image are stitched together by channel dimension to form a dual-channel input image of the structure under test. This image is then input into the trained deep learning model to obtain the predicted crack depth value of the structure under test.

[0047] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0048] This invention fuses transient thermal wave signals captured by an infrared thermal imager with fine geometric information acquired by a visible light camera in a multimodal manner. By constructing a dual-channel deep learning model, it achieves a leap from observing shape to measuring depth. This not only solves the problems of traditional manual inspection's inability to acquire depth information and the limitations of ultrasonic methods due to on-site conditions, but also significantly enhances weak infrared thermal feature signals by introducing controllable thermal excitation and intelligent keyframe extraction technology. Furthermore, this invention utilizes heat conduction inversion, empirical formulas, and the results of localized destructive experiments to automatically generate high-confidence depth labels, providing high-quality fuel for the deep learning model and effectively solving the bottleneck of scarce high-quality labeled data in engineering practice. This invention encapsulates complex physical mechanisms within an intelligent model, enabling on-site inspectors to obtain crack depth in real time through simple image acquisition operations, greatly improving the intelligence level and inspection efficiency of structural health monitoring. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall method flow of the present invention;

[0050] Figure 2 This is a schematic diagram showing the results before and after the adjustment of the verification mechanism of this invention;

[0051] Figure 3 This is a schematic diagram of the consistency verification results of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0053] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0054] Example:

[0055] Please see Figures 1-3 The present invention provides a technical solution:

[0056] A crack depth prediction method based on infrared thermal imager and deep learning, comprising the following steps:

[0057] Step 1: At the same detection location, simultaneously acquire infrared thermal image sequence and visible light image of the crack area. The infrared thermal image sequence is a thermal image of the crack area cooling process acquired under thermal excitation.

[0058] In this embodiment, the infrared thermal imager and the visible light camera are fixed on the same detection platform to ensure that their relative positions remain constant during the acquisition process. To ensure that the two images have overlapping fields of view, the optical axes of the lenses of the two devices should be arranged in parallel. To obtain a more accurate spatial correspondence, preferably, the infrared thermal imager and the visible light camera are jointly calibrated before each detection task. The calibration method can adopt the Zhang Zhengyou calibration method, which calculates the intrinsic parameters (focal length, principal point coordinates, distortion coefficients) of the two cameras and the extrinsic parameters (rotation matrix and translation vector) between them by photographing a checkerboard calibration board. Through calibration, a precise mapping relationship between the pixel coordinates of the infrared image and the pixel coordinates of the visible light image can be established, providing prior parameters for subsequent spatial registration, thereby improving registration accuracy and efficiency.

[0059] Since the thermal contrast of cracks in their natural state is often very weak, active thermal excitation needs to be applied to the crack area before data collection. In this embodiment, a low-power hot air gun is preferably used as the thermal excitation device. Considering the thermal sensitivity of most civil engineering materials (such as concrete, asphalt, and stone), the heating temperature is strictly controlled below 40 degrees Celsius to avoid thermal damage to the structure.

[0060] During the heating process, an infrared thermal imager is used to monitor the temperature field changes on the surface of the crack area in real time at a high frequency (e.g., 10Hz to 30Hz, preferably 20Hz in this embodiment). The center point of the crack is selected as the monitoring point in the infrared thermal imager's view, and the temperature value at that point and its rate of increase over time are calculated in real time.

[0061] The criteria for determining the end of the heating phase and immediately stopping thermal excitation are the following two conditions:

[0062] Temperature threshold condition: The temperature at the center point of the crack area reaches a preset safe temperature threshold, such as 40℃. This safe temperature threshold is set primarily based on material safety considerations, ensuring that the heating process does not lead to material performance degradation or damage to the surface coating. Furthermore, this temperature value is lower than the thermal damage threshold of most civil engineering materials (such as concrete and asphalt), and also avoids interference from severe surface moisture evaporation, ensuring the non-destructive nature of the testing process and the reliability of the data.

[0063] Temperature rise rate inflection point condition: The temperature rise rate at the center of the crack region first shows a decreasing inflection point. This is because in the initial stage of heating, heat is mainly transferred to the interior of the structure by thermal conduction, and the surface temperature rises rapidly. When defects such as cracks block the heat flow conduction path, heat accumulates above the defects, causing a change in the surface temperature rise rate. By calculating the first derivative of the temperature-time curve in real time, the change in the rise rate can be monitored. When the rise rate values ​​of three consecutive sampling points are all less than the previous peak value, and the peak value decreases by more than 20%, a decreasing inflection point can be determined. This condition can adaptively determine the optimal heating time for different materials and crack depths, avoiding overheating or underheating.

[0064] The safe temperature threshold ensures safety, while the inflection point can intelligently capture the critical point with the best thermal contrast based on the defect characteristics of different materials. When one of the above two conditions is met, the heating stage is determined to be over and thermal excitation is stopped immediately.

[0065] The heating time is not fixed, but depends on the thermal diffusivity of the material in the crack region. Adjustments were made. Thermal diffusivity. The larger the surface area, the faster heat conducts within the material; therefore, the required heating time to create sufficient thermal contrast on the surface increases. The longer. For example, heating time. It can be based on empirical formulas Make estimates, among which This is the estimated crack depth (which can be estimated based on the width first). The adjustment coefficient is related to the power and form of the heat source. The initial value can be set to 1.0, and adjusted within the range of 0.5-2.0 based on actual results. For example, for a thermal diffusivity of... For ordinary concrete with an estimated crack depth of 50mm, the initial estimated heating time is approximately 55 seconds. This formula is only an estimate; the actual stopping time will be ultimately determined by the two real-time judgment conditions mentioned above.

[0066] Immediately after heating is complete, the infrared thermal imager and visible light camera are automatically triggered to collect data synchronously.

[0067] An infrared thermal imager continuously acquires a sequence of infrared thermal images of the crack area during natural convection cooling at a fixed frame rate (e.g., 30 frames / second). The acquisition duration should be set according to the cooling characteristics of the material, generally continuing until the temperature difference between the crack and the background area becomes indistinguishable or the crack temperature becomes the same as the background area. For example, continuous acquisition for 60 to 300 seconds, at a frame rate of 30 fps, is sufficient to capture subtle dynamics of temperature changes during cooling. A 120-second acquisition duration covers the entire process from the start of cooling to the temperature field stabilizing, ensuring that key frames with the most significant thermal contrast can be found. Based on experience from extensive pre-experiments on typical concrete specimens, 120 seconds is preferred for concrete materials in this embodiment. The precise acquisition timestamp corresponding to each frame of the infrared thermal image is recorded for subsequent analysis of the cooling rate.

[0068] To ensure strict correspondence with the initial state of the infrared thermal image sequence, a hardware trigger signal is used at the same moment the infrared thermal imager begins acquiring data to control the visible light camera to acquire a high-resolution visible light image. This image records the true texture, color, and geometry of the crack surface.

[0069] Since ambient temperature and humidity affect the heat dissipation rate of the structure and the accuracy of infrared thermometry, the ambient temperature (unit: °C) and relative humidity (unit: %) at the detection site must be recorded simultaneously during the acquisition process. These parameters can be measured using a handheld thermometer and hygrometer and input into the acquisition control software in real time via a data interface. They are then stored in association with each frame of infrared image data. These environmental parameters will serve as key boundary conditions for establishing the physical model of heat conduction in subsequent steps.

[0070] Step 2: Determine the keyframes in the infrared thermal image sequence and crop out the regions of interest (ROIs) of the keyframes. Spatially register and stitch the ROIs with the visible light image to obtain the multimodal fusion features of the dual-channel input image and the crack region. The multimodal fusion features include infrared thermal features and visible light geometric features.

[0071] The infrared thermographic sequence records the complete cooling process of the crack region from the end of heating to the return to temperature equilibrium. The temperature difference (i.e., thermal contrast) between the crack and the background varies in images at different times. To extract the most significant crack features, the image frames with the best thermal contrast need to be selected as keyframes from the sequence. In this embodiment, the method for determining the keyframes is as follows:

[0072] In the starting frame or any frame of the infrared thermal image sequence, a small area containing the crack is roughly selected as the region of interest by manually selecting a bounding box or by using a simple threshold segmentation algorithm.

[0073] Iterate through each frame of the infrared thermal image sequence, calculating the average temperature value of the crack body pixels within the region of interest and the average temperature value of the background region pixels adjacent to the crack. The background region can be a defect-free area parallel to the crack and at a certain distance (e.g., 10 to 20 pixels) from the crack edge.

[0074] Calculate the temperature difference for each frame, which is the absolute value of the average temperature value within the region of interest and the average temperature value of the background pixels adjacent to the crack, and plot the temperature difference over time. This curve is typically a single-peaked curve that first rises (due to heat accumulation at the defect) and then falls (due to heat dissipation). The frame corresponding to the maximum temperature difference is identified as the keyframe with the most significant thermal contrast.

[0075] It should be noted that if there are multiple independent cracks in the image sequence, a region of interest can be selected for each crack, and the above steps can be repeated to determine the corresponding keyframe for each crack.

[0076] To improve image quality and the accuracy of subsequent processing, the selected keyframe infrared thermal images and synchronously acquired visible light images were preprocessed, as follows:

[0077] Gaussian filtering is applied to the selected keyframe infrared thermal image and visible light image for denoising. The choice of Gaussian kernel size is crucial; a kernel that is too large will blur the crack edges, while a kernel that is too small will result in poor denoising. Preferably, the Gaussian kernel size is 3×3 or 5×5, depending on the image resolution and noise level. The standard deviation is automatically calculated based on the kernel size; for example, for a 3×3 kernel, the standard deviation is typically between 0.5 and 0.8.

[0078] The original contrast of infrared thermal images may be low. A contrast-limited adaptive histogram equalization algorithm is used to enhance the local contrast of crack areas in keyframe infrared thermal images. The preferred algorithm is the CLAHE algorithm, which has two key parameters: grid size and contrast limit. The grid size determines how many local regions the image is divided into for histogram equalization; a smaller grid results in stronger local contrast enhancement but may also introduce noise. Preferably, for an infrared image with a pixel size of 640×480, the grid size is set to 8×8, dividing the image into 64 small blocks of 80×60 pixels. The contrast limit is used to limit the height of the histogram, preventing noise from being excessively amplified. The contrast limit value is typically set between 2 and 4; preferably, it is 3.0, which effectively enhances crack contrast while suppressing background noise.

[0079] To reduce computational load and focus on the crack itself and its surrounding heat-affected zone, it is necessary to crop the region of interest (ROI) from the keyframe infrared thermal image centered on the crack. In this embodiment, a deep learning-based crack instance segmentation model is used to intelligently delineate the ROI. The specific logic is as follows:

[0080] The keyframe infrared thermal image is input into a pre-trained crack instance segmentation model, such as Mask R-CNN or YOLACT++, which outputs a pixel-level mask for each crack in the image. For each crack mask, its minimum bounding rectangle is calculated. To fully encompass the crack body and the surrounding heat-affected zone formed by impaired heat conduction, the minimum bounding rectangle needs to be extended outward by a certain number of pixels. This extension range is not a fixed value but is related to the crack size and the thermal conductivity of the material.

[0081] The expansion range can be dynamically calculated based on the maximum width of the crack (derived from the pixel-level mask), for example... ,in, It is an expansion factor, with a value ranging from 2 to 5. For macroscopic cracks with larger widths, the heat-affected zone is relatively wider. A larger value can be chosen (e.g., 4); for fine cracks, to avoid introducing too much background, A smaller value can be taken (e.g., 2.5). Preferably, The value is 3, which means that the range of pixels is expanded to approximately 3 times the maximum width of the crack in all directions. To a certain extent, this reflects the size of the crack, while the extent of the heat-affected zone is related to the crack size and material properties. (Using...) The dynamic expansion method is more scientific than fixed pixel expansion. It can adapt to cracks of different scales and ensure that the cropped region of interest can fully contain the effective information area without introducing noise due to the inclusion of too much irrelevant background.

[0082] The expanded rectangular region is cropped from the original infrared thermal image to obtain an infrared region of interest (ROI) image. When multiple cracks exist in the image, each crack is independently defined and cropped to obtain multiple ROI image samples.

[0083] Due to parallax between infrared and visible light cameras, the pixel coordinates of the same physical location in the two images do not correspond perfectly. Spatially registering the infrared region of interest image with the original visible light image is necessary to obtain visible light image patches corresponding to the same crack region. Because of the significant modal differences between infrared and visible light images, directly extracting common feature points is difficult. In this embodiment, an indirect registration strategy is employed. First, the visible light image is downsampled to make its resolution roughly equivalent to the infrared image. Then, using edge features or significant structural features (such as crack intersections or structure edges), at least four pairs of corresponding points are selected as control points on both images, either manually or automatically. Finally, a perspective transformation matrix is ​​calculated based on these control points. The calculated transformation matrix is ​​applied to the original, high-resolution visible light image, performing a geometric transformation to obtain a registered visible light image aligned with the viewpoint of the infrared image. Based on the region of interest coordinates defined on the infrared thermal image, the registered visible light image is directly cropped at the same location to obtain a visible light crack region image that spatially corresponds one-to-one with the infrared region of interest image.

[0084] To meet the input requirements of deep learning models, the two cropped images need to be processed, specifically:

[0085] The infrared region of interest image and the visible light crack region image are uniformly scaled to the same pixel size using an interpolation algorithm (e.g., bilinear or bicubic interpolation). Determining this size requires a trade-off between computational cost and information retention. In this embodiment, all image samples are uniformly scaled to 224×224 pixels, a commonly used input size for many classic deep learning models, achieving a good balance between computational efficiency and feature representation.

[0086] The essence of an infrared region of interest (ROI) image is a temperature matrix. To eliminate absolute temperature differences caused by varying ambient temperatures and heating conditions, the infrared ROI image needs to be normalized. The normalization formula is: ,in This is the original temperature value. This represents the normalized temperature value. and These represent the minimum and maximum temperatures within the region of interest, respectively. After normalization, the pixel values ​​are mapped to the [0,1] interval, representing the relative temperature distribution. Furthermore, the pixel values ​​of the visible light image are linearly normalized from the original [0,255] to the [0,1] floating-point interval.

[0087] Step 3: Based on multimodal fusion features, obtain multiple estimates of crack depth through at least two different depth estimation methods, perform consistency verification on multiple estimates, and when the consistency meets the preset requirements, determine the depth label of the crack region and construct a labeled dataset.

[0088] In this embodiment, the logic for determining the multimodal fusion features is as follows:

[0089] Based on the infrared region of interest image after spatial registration, cropping, and scaling, the following features are extracted:

[0090] Calculate the temperature gradient of each pixel in the image along the horizontal (x-direction) and vertical (y-direction). The gradient can be calculated using the Sobel operator or the central difference method. For example, for pixel (i,j), its horizontal gradient... Vertical gradient .in, This represents the normalized temperature value of the pixel located in the i-th row and j+1-th column of the same image. This represents the normalized temperature value of the pixel located in the i-th row and j-1-th column of the same image. This represents the normalized temperature value of the pixel located in the (i+1)th row and jth column of the same image. This represents the normalized temperature value of the pixel located in the (i-1)th row and jth column of the same image; from this, the gradient magnitude of each pixel is obtained. and gradient direction .

[0091] The maximum, mean, and variance of the gradient magnitudes within the entire infrared region of interest image are statistically analyzed. Simultaneously, the rates of change of the gradients in the horizontal and vertical directions are calculated; for example, the divergence or curl of the gradient field can be calculated to characterize the heat flux diffusion pattern.

[0092] Calculate the average, maximum, and minimum temperatures of the crack region (defined by the crack mask). Combine the keyframe acquisition timestamps and environmental parameters to calculate the instantaneous cooling rate corresponding to the keyframe moment. The cooling rate can be obtained by numerically differentiating the temperature-time curves of multiple frames before and after the keyframe. For example, taking the temperature sequence of the keyframe and two frames before and after it (a total of 5 frames), the cooling rate at the center point can be calculated using a five-point difference formula. ,in This is the inter-frame time interval. , , , These represent the acquisition times corresponding to the two frames before the keyframe, the previous frame, the next frame, and the two frames after the keyframe, respectively. , , , These represent the temperature values ​​at the corresponding times. It should be noted that using the five-point difference formula requires at least two valid frames before and after the keyframe. For frames near the start or end of an infrared thermal image sequence, if the five-point difference requirement cannot be met, a three-point difference formula or a two-point difference formula (i.e., first-order backward / forward difference) can be used as an alternative.

[0093] On the visible light crack region image, a crack instance segmentation model is used to extract the pixel-level mask of the crack, and the following features are extracted:

[0094] After binarizing the pixel-level mask, the centerline (skeleton) of the crack is extracted using a skeletonization algorithm (such as the Zhang-Suen thinning algorithm). The crack pixel length is obtained by counting the number of skeleton pixels. Combined with the spatial resolution of the image (e.g., how many pixels per millimeter), the pixel length and physical length of the crack can be calculated.

[0095] Search outwards along the normal direction of the skeleton line until the mask boundary is encountered, and calculate the crack width at that point. Statistically analyze the crack widths calculated at all normal locations to obtain the average crack width, maximum crack width, and standard deviation of the crack width.

[0096] The total number of pixels within the mask is counted, and combined with the image resolution, the area of ​​the crack pixels is obtained, which together constitute the visible light geometric features.

[0097] In this embodiment, the crack instance segmentation model used is based on a deep convolutional neural network instance segmentation architecture. Considering the diversity and complexity of crack morphology in engineering applications, and the practical difficulty of training a high-quality segmentation model requiring a large amount of labeled data, this invention preferably uses a model pre-trained on a large general dataset (such as the COCO dataset) as the basis, and then uses a small amount of labeled crack image data for fine-tuning.

[0098] Specifically, this embodiment selects Mask R-CNN as the basic framework for instance segmentation, with ResNet-50-FPN (Feature Pyramid Network) as its backbone network. This model has already learned rich general object features on the COCO dataset and has powerful feature extraction capabilities. We use approximately 2000 visible light images containing different types of cracks (such as transverse cracks, longitudinal cracks, and mesh cracks), which we collected and labeled ourselves, to perform transfer learning training on the pre-trained model. The training process is as follows:

[0099] Each crack in the acquired visible light images was precisely annotated with pixel-level polygons, generating a corresponding JSON format annotation file. COCO pre-trained weights were loaded, and the number of output classes for the model's classification layer was modified to 2 (background and cracks). Training employed a momentum-driven stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.001, a momentum factor of 0.9, and a weight decay of 0.0001. The batch size was set to 8. A two-stage training strategy was adopted. In the first stage, the parameters of the backbone network (ResNet-50-FPN) were frozen, and only the network head was trained for 50 epochs. In the second stage, all network layers were unfrozen, and the entire network was fine-tuned with a low learning rate of 0.0001 for 100 epochs. During training, the mean accuracy on the validation set was monitored, and the best-performing model weights were saved. After training, the model was integrated into the data processing pipeline. Given an image of a visible light crack region, the model can directly output the pixel-level mask, confidence score, and minimum bounding rectangle for each crack in the image.

[0100] This eliminates the need to train a large network from scratch, significantly reducing the requirement for massive amounts of labeled data and lowering engineering implementation costs. The low-level features (such as edges, textures, and corners) learned by the pre-trained model on general datasets exhibit excellent transferability, effectively improving the model's segmentation robustness under different lighting and background conditions. Through fine-tuning, the model can focus its general feature extraction capabilities on the specific target of cracks, thereby achieving more accurate segmentation results.

[0101] To obtain high-confidence depth labels, this embodiment employs three independent methods to verify the consistency of the depth of the same crack. The logic is as follows:

[0102] Method 1: Depth estimation based on the inverse problem of heat conduction

[0103] For the material of the structure being tested (e.g., C30 concrete), a pre-established database of its thermophysical properties (including density, specific heat capacity, and thermal conductivity) is created. Based on these parameters, a two-dimensional or three-dimensional heat conduction simulation model is built using finite element analysis software (such as ANSYS or COMSOL Multiphysics). Cracks of different depths (e.g., from 5 mm to 100 mm, with a step size of 2 mm) are set in the model, and thermal excitation and environmental boundary conditions (ambient temperature and humidity) consistent with those in field experiments are applied. The simulation calculates the temperature field variation over time on the surface of the crack region at each depth. From the simulation results, the same infrared thermal characteristics (such as maximum temperature difference, cooling rate, temperature gradient, etc.) are extracted to form a depth-thermal response characteristic database.

[0104] The measured and calculated infrared thermal feature vector is matched with simulated features in the database for similarity. The matching algorithm can employ the nearest neighbor algorithm or Euclidean distance minimization. The database is then used to find one or more sets of simulated features that are closest to the infrared thermal feature vector; the corresponding depth value is the first estimate of the crack depth, D1. For example, if the measured infrared thermal feature vector has the smallest Euclidean distance to the simulated feature in the database with a depth of 28 mm, then D1 = 28 mm.

[0105] Method 2: Depth estimation based on the width-depth empirical relationship

[0106] For the same material, an empirical relationship between the maximum width of the crack surface and the crack depth D is established in advance through destructive testing. For example, a batch of concrete test blocks with precast cracks of different depths are loaded, their maximum surface width is measured, and then the test blocks are broken open to measure the actual depth. The collected data (maximum width, crack depth) are then subjected to regression analysis.

[0107] Based on the characteristics of the data distribution, linear regression can be selected. Or a power function regression that better aligns with fracture mechanics theory. .

[0108] The maximum width of the crack obtained from the aforementioned calculation Substituting into the above empirical formula, we can quickly obtain the second estimate of the crack depth, D2.

[0109] Prepare a set of standard specimens with artificial cracks of different preset depths. For example, for concrete, 30 standard cubic specimens (150mm×150mm×150mm) can be made, and thin steel sheets of different thicknesses can be pre-embedded during pouring. These steel sheets are then removed before the concrete initially sets, forming cracks with depths of 10mm, 20mm, 30mm, 40mm, 50mm, and 60mm. Five specimens are prepared for each depth to ensure statistical reliability. For natural cracks in existing structures, 10-20 representative crack locations can be selected. First, the maximum width of the crack is measured, and then the true depth is obtained through core drilling.

[0110] For each specimen or core sample, accurately measure the maximum width of its surface cracks. (Unit: mm) and corresponding actual depth (Unit: mm). Record the measured data points as ( , ), r=1,2,…,N, where N is the total number of samples and r represents the sample index. For example, the following data points may be obtained through the above experiment: (0.2mm,12mm), (0.3mm,18mm), (0.5mm,28mm), (0.8mm,41mm), (1.2mm,55mm), etc.

[0111] Plot the collected data points on a coordinate system (horizontal axis is the maximum width). The vertical axis represents the crack depth (D), and the distribution trend of the data points is observed. According to fracture mechanics theory and engineering experience, crack depth and surface width typically exhibit a non-linear positive correlation. Therefore, this embodiment preferably uses the following two regression models for fitting:

[0112] Linear regression model: This model is suitable for specific materials or specific depth ranges where crack depth and width have an approximately linear relationship. The least squares method is used to determine parameters a and b. The goal of the least squares method is to find a set of a and b that minimizes the sum of squared residuals between the predicted and actual values ​​for all data points.

[0113] Assuming we use the above 5 data points and calculate using the least squares method (the specific calculation process can be implemented using tools such as Excel, MATLAB, or Python), we obtain a = 45.2 and b = 2.1. Then, the established linear empirical relationship is: This formula indicates that for this type of concrete material, for every 1 mm increase in the maximum surface width, the crack depth increases by approximately 45.2 mm.

[0114] Power function regression model:

[0115] This model is more consistent with fracture mechanics theory and can describe the physical law that width increases at an accelerated rate with depth. Taking the natural logarithm of both sides of the power function transforms it into a linear form: .make , , Then it is transformed into a linear regression model. At this point, the least squares method can also be used to transform the data points ( , Perform linear regression to solve for A and n. Finally, obtain the result through exponential calculation. .

[0116] Calculate the goodness of fit of the two models. , The closer a value is to 1, the stronger the model's ability to interpret the data. Compare the two models. Value, selection A higher-order model is used as the final breadth-depth empirical relation. For example, if linear regression... Power function regression If so, then the power function regression model is preferred.

[0117] Method 3: Depth estimation based on traditional nondestructive testing methods

[0118] To ensure absolute accuracy, a number of representative crack locations were selected, and traditional, engineering-approved testing methods were used for on-site measurements.

[0119] Ultrasonic pulse echo method: Ultrasonic transducers are placed on both sides of the crack to measure the sound time, and the crack depth is calculated based on the sound velocity in the concrete.

[0120] Core sampling method: With consent and if the structure allows, core samples are taken from the crack location, and the crack depth is measured directly on the core sample. This is a destructive testing method, but it offers the highest accuracy.

[0121] In this embodiment, core drilling is used to obtain D3 for key parts or 10% of the samples in the dataset; for the majority of the remaining samples, ultrasonic pulse echo method is used to obtain D3, in order to balance accuracy and cost.

[0122] The first, second, and third estimates are compared, and their mean and standard deviation are calculated. The coefficient of variation (COP), the ratio of the standard deviation to the mean, is calculated. When the COP is less than 10%, the estimate is considered high-confidence, and the mean is used as the depth label for the crack region. The corresponding infrared region of interest (ROI) image and the visible light crack region image are associated and stored, forming a labeled sample that can be used for supervised learning. When the number of labeled samples that can be associated and stored, forming a number of ROI images and visible light crack region images, is not less than 10%, it indicates a significant discrepancy between the estimation results of the three methods, and this sample is considered a low-confidence sample. At this point, a verification mechanism is triggered. The verification mechanism includes checking whether the image registration is accurate and whether the crack mask is accurate. The boundary conditions (such as the heat transfer coefficient) in the inverse heat conduction model are re-examined for reasonableness, and fine-tuned based on environmental parameters. D1 is recalculated; the measurement results of D3 are verified, and if necessary, a higher-precision core sampling method is used for re-measurement; the recalculated D1, D2, and D3 are then checked for consistency. If the coefficient of variation is still less than 10% after (maximum) three adjustments, the sample is marked as invalid and not included in the final dataset to avoid low-quality data polluting the model.

[0123] All high-confidence samples that pass the consistency check (infrared region of interest images, visible light crack region images, and depth labels) are uniformly formatted and stored. The dataset can be randomly divided into training, validation, and test sets according to a certain ratio, for example, 70% training, 15% validation, and 15% test, for subsequent training, optimization, and performance evaluation of deep learning models.

[0124] This embodiment selects representative test samples for statistical analysis to illustrate the effectiveness of the proposed consistency verification mechanism in screening data quality and the improvement effect of the review and adjustment process on the convergence of estimated values. Table 1 shows the initial estimated values ​​and their consistency verification results for 30 samples of different material types and crack types. Among them, 24 samples had an initial coefficient of variation of less than 10%, and directly passed the verification and generated high-confidence depth labels; 6 samples triggered the review mechanism because their initial coefficient of variation was greater than 10%. Table 2 details the changes in estimated values ​​and specific adjustment measures before and after the review and adjustment for these 6 samples that triggered the review, as follows:

[0125] Table 1: Schematic Table of Consistency Verification Results

[0126]

[0127] Table 2: Schematic diagram of review and adjustment results

[0128]

[0129] As shown in Table 1, 24 out of 30 samples (80%) had initial coefficients of variation less than 10%, directly passing the consistency check. Their final label values ​​were the arithmetic mean of the three estimated values. These samples covered various building materials such as C30-C50 concrete, asphalt mixture, granite, and marble, as well as three types of cracks: transverse, longitudinal, and network cracks, with depths ranging from 8.7 mm to 75.9 mm, fully demonstrating the wide applicability of the method of this invention to different materials and crack types. The remaining 6 samples (6 out of 8 samples excluding 25 from 19-28) had initial coefficients of variation between 10.99% and 25.27%, triggering the verification mechanism.

[0130] Table 2 details the verification and adjustment process for these six sample groups. Taking sample 19 as an example, this sample is a network crack in C30 concrete. The initial first, second, and third estimates were 16.8 mm, 19.3 mm, and 24.5 mm, respectively, with a coefficient of variation as high as 19.41%. Analysis revealed that the boundary conditions for the network crack were set too ideally by the heat transfer inversion model. By increasing the heat transfer coefficient by 15%, the first estimate was recalculated to 19.2 mm, while retaining the second estimate of 19.3 mm. The re-collected third estimate was 21.1 mm. After adjustment, the coefficient of variation decreased to 5.33%, successfully passing the verification. Sample 20 is a transverse crack in asphalt mixture. The initial third estimate of 37.2 mm deviated significantly from the initial first estimate of 28.5 mm and the initial second estimate of 32.7 mm. By re-measuring the third estimate of 30.8 mm through core sampling, the coefficient of variation decreased from 13.26% to 6.85%. The initial high dispersion of longitudinal cracks in sample 21 (granite) stemmed from neglecting the influence of material anisotropy on thermal diffusion. After correcting for thermal diffusivity, the first estimated value was adjusted from 41.3 mm to 43.6 mm, and the third estimated value was adjusted from 48.5 mm to 42.2 mm, reducing the coefficient of variation to 8.78%. For samples 22, 23, 24, and 26, measures such as image reregistration, environmental humidity correction, infrared sequence resampling, and camera calibration optimization were implemented, reducing the coefficient of variation to below 10%. Notably, despite re-measuring the third estimated value (adjusted from 15.8 mm to 12.1 mm), the coefficient of variation for the sandstone network cracks in sample 28 remained at 11.32%, failing to meet the inclusion criteria and ultimately being removed from the training dataset.

[0131] from Figure 1 It is clearly visible that before adjustment, the coefficients of variation of all eight groups of samples that triggered the review were above the 10% threshold (reaching a maximum of 25.27%). After targeted adjustments, seven of these groups successfully reduced their coefficients of variation to below 10% (the lowest dropping to 5.33%), with only sample 28 remaining above the threshold. This visual comparison intuitively demonstrates the effectiveness of the review mechanism, namely, through multiple interventions such as optimizing the boundary conditions of the physical model, improving measurement methods, and enhancing image processing accuracy, the vast majority of originally discrete estimates can converge to an acceptable range. Figure 2 The distribution of the three sets of estimates after adjustment is further shown. It can be seen that the estimated values ​​of each sample after adjustment are closely clustered and the dispersion is significantly reduced, indicating that the review and adjustment successfully eliminated the systematic bias and measurement error in the original estimates.

[0132] Combining Tables 1 and 2 above, Figure 2 and Figure 3It can be observed that heat conduction inversion is based on physical mechanisms but is affected by boundary conditions; the width-depth empirical formula is fast and simple but relies on statistical laws; and traditional detection methods are highly accurate but inefficient. Combining these three methods and passing consistency verification achieves complementary advantages and offsets disadvantages.

[0133] Step 4: Build a deep learning model, train it using a labeled dataset, and learn the mapping relationship between the dual-channel input image and the depth label.

[0134] In this embodiment, after obtaining the depth labels generated by high-confidence physical / geometric feature verification, step 4 returns to the problem itself and designs an end-to-end deep learning model with dual-channel images as input. The aim is to enable the network to automatically learn pixel-level features related to depth. The learning of the mapping relationship specifically includes:

[0135] Before training begins, the image-depth pairing dataset constructed in step 1 needs to be normalized and partitioned. In this embodiment, the dataset contains, for example, 5000 preprocessed sample pairs. Each sample pair consists of an infrared region of interest image and a corresponding visible light crack region image, along with the true depth label of the crack. To ensure the effectiveness of model training and the objectivity of evaluation, the dataset is randomly divided into three subsets according to the following proportions: the training set, accounting for 70% (3500 images), is used for actual update and optimization of model parameters; the validation set, accounting for 15% (750 images), is used for model performance monitoring, hyperparameter tuning, and early stopping mechanism judgment during training; and the test set, accounting for 15% (750 images), is used for unbiased evaluation of the model's generalization ability after final training, and is only used once after the model is finalized.

[0136] This embodiment designs a deep neural network specifically for predicting crack depth in infrared thermal images, abbreviated as IR-ADResNet. The network design fully considers the characteristics of crack regions in infrared thermal images, such as weak thermal gradient features, susceptibility to background interference, and multi-scale correlation between crack depth and heat diffusion patterns. By introducing an attention mechanism and a multi-scale feature fusion strategy, the model's ability to extract depth-related features is enhanced. To clearly illustrate the target network optimized during training, the components, hierarchical connections, and key dimensional parameters of the IR-ADResNet network are described in detail below:

[0137] The network takes a preprocessed dual-channel image, stitched together from an infrared region of interest image and a visible light crack region image, as input, with a uniform image size of 224×224 pixels. The backbone of the network is built on the ResNet-18 architecture and is responsible for extracting hierarchical depth features from the input image. Specifically, the input layer receives a normalized infrared image tensor of shape (Batch_Size, 224, 224, 1), followed by an initial convolutional layer using 64 7×7 kernels with a stride of 2 and padding of 3, resulting in an output feature map size of 112×112 and 64 channels. Next, a batch normalization layer, a ReLU activation function, and a 3×3 max-pooling layer are applied, reducing the output feature map size to 56×56 while maintaining the 64 channels. Afterward, the network sequentially passes through four stages of residual block groups. The first stage contains two basic residual blocks, with both input and output channels at 64, and the feature map size remaining unchanged at 56×56. The second stage consists of two basic residual blocks. In the first residual block, a 1×1 convolutional layer is added to the skip connections to match the dimension, increasing the number of channels to 128. The feature map size is reduced to 28×28 by setting the convolution stride to 2. The second residual block maintains the same number of channels (128) and size (28×28). The third stage also consists of two basic residual blocks. In the first residual block, the number of channels is increased to 256 through a 1×1 convolution, and the feature map size is reduced to 14×14 through a convolution with a stride of 2. The second residual block maintains the same number of channels (256) and size (14×14). The output feature map F3 from this stage is simultaneously fed into the subsequent DAEM and MFFM modules. The fourth stage comprises two basic residual blocks. The first residual block increases the number of channels to 512 through a 1×1 convolution and reduces the feature map size to 7×7 through a convolution with a stride of 2. The second residual block maintains the number of channels (512) and the size (7×7). The output feature map F4 from this stage is simultaneously fed into the subsequent DAEM and MFFM modules. Furthermore, the output feature map F2 (size 28×28, 128 channels) from Layer 2 in the second stage is directly fed into the MFFM module without DAEM processing to preserve more spatial detail.

[0138] After the third and fourth stages of the backbone network, dual-path attention enhancement modules (DAEMs) are embedded. These are used to input feature maps. For example, this module adaptively recalibrates the feature map using two parallel paths; here, any feature map can be represented as a three-dimensional tensor, denoted as . Where R represents the set of real numbers, H represents the feature map height, W represents the feature map width, and C represents the number of feature map channels. In the channel attention path, the input Q is first subjected to global average pooling to obtain a 1×1×C feature vector. Then, the pooled vector is sequentially fed into two fully connected layers. The first fully connected layer compresses the number of channels to C / 16 and applies the ReLU activation function. The second fully connected layer restores the number of channels to C and applies the Sigmoid activation function, thereby generating the channel attention weight vector. In the spatial attention path, the input Q is first subjected to average pooling and max pooling along the channel dimension, respectively, to obtain two H×W×1 feature maps. These two feature maps are then concatenated along the channel dimension to obtain an H×W×2 feature map. Next, a 7×7 convolution operation is performed on this concatenated feature map, outputting a single channel. Finally, a sigmoid activation function is applied to generate the spatial attention weight map. Finally, the channel weight vector Spatial weighting graph Element-wise multiplication (using a broadcast mechanism) yields a comprehensive attention map M. This comprehensive attention map M is then multiplied element-wise with the original input feature map Q to obtain the final enhanced feature map Q. After passing through the DAEM module, the feature map size and number of channels remain unchanged (H×W×C as input), but its feature response values ​​are adaptively adjusted to focus more on crack-related thermal gradient features. In addition, a dual-path attention enhancement module can be embedded after the second stage of the backbone network. Introducing attention in the second stage aims to leverage its high-resolution feature map to enhance the accurate perception of crack edges and micro-geometry. Introducing attention in the third and fourth stages is used to capture depth-related thermal diffusion semantic features, thereby achieving full-scale feature enhancement from fine geometry to deep semantics.

[0139] Following the DAEM module, the network integrates feature maps from different stages of the autonomous backbone network into the Multi-Scale Feature Fusion (MFFM) module. The inputs to this module include: the output F2 from Layer 2 (28×28, 128 channels), the output F3 from Layer 3 after DAEM processing (14×14, 256 channels), and the output F4 from Layer 4 after DAEM processing (7×7, 512 channels). The specific fusion operation of MFFM is as follows: First, bilinear interpolation upsampling is performed on F2 and F3 respectively, enlarging their spatial size to the same 7×7 as F4. Then, each is passed through a 1×1 convolutional layer, and its channel count is uniformly adjusted to 512, consistent with F4. F4 itself has a size of 7×7 and 512 channels, so no adjustment is needed. Subsequently, the three feature maps after the above processing are added element-wise to obtain the fused feature map. Its dimensions are 7×7 and it has 512 channels.

[0140] After obtaining the fused feature map, the network proceeds to the regression prediction head. First, the fused feature map... Global average pooling is performed to compress the spatial dimension, resulting in a 1×1×512 feature vector. This 512-dimensional feature vector is then input into the first fully connected layer, reducing the output dimension to 128. A ReLU activation function is then applied, and this layer introduces weight decay as a regularization term with a coefficient set to 0.0001 to constrain model complexity and prevent overfitting. Finally, the 128-dimensional feature vector is input into the second fully connected layer, outputting the final 1-dimensional crack depth prediction.

[0141] With a clearly defined network structure, this embodiment employs a systematic training strategy to optimize the aforementioned IR-ADResNet network. The core of the training is defining a loss function that accurately reflects the deviation between predicted and true values. This invention uses the mean squared error loss function, mathematically expressed as follows: Where G is the batch sample size. and Let be the predicted depth and the actual depth of the g-th sample, respectively. To efficiently optimize the network parameters, the Adam optimizer is chosen. This optimizer combines the advantages of momentum and adaptive learning rate. The initial learning rate is set to 0.001, and the exponential decay rate of the first moment estimation is used. Set to 0.9, the exponential decay rate estimated by the second moment. The learning rate is set to 0.999. To improve convergence and avoid getting stuck in local optima, a cosine annealing learning rate scheduling strategy is introduced. This allows the learning rate to smoothly decrease from the initial learning rate to the minimum learning rate according to a cosine function over the training cycle. In this embodiment, the minimum learning rate is set to 1e-6. Considering both GPU memory capacity and model convergence speed, the batch size is set to 16, and the total training cycles are preset to 50. To prevent the model from overfitting the training data and losing its generalization ability, an early stopping mechanism is implemented: after each training cycle, the model's loss value on the validation set is calculated. If the validation set loss does not decrease within 10 consecutive training cycles, training is automatically terminated, and the model with the lowest validation loss is saved as the final model. In addition, to further improve the robustness of the model, data augmentation strategies are used during training, including random horizontal flipping, random vertical flipping, and random rotation within a certain angle range (e.g., ±5°). These augmentation operations are only applied during training and are not used during validation and testing.

[0142] The preprocessed training set images are continuously input into the IR-ADResNet network in batches of 16. The network performs forward propagation to calculate the predicted depth values, and the loss function calculates the error between the predicted values ​​and the ground truth labels. The optimizer updates the weight parameters of each layer of the network based on the error through backpropagation. This process is repeated until the early stopping condition is met. Finally, the model weight file with the lowest loss on the validation set is saved for use in the subsequent on-site prediction application in step 5.

[0143] The network structure parameters and training strategies described above are reasonably set. The backbone network built on ResNet-18 achieves a good balance between parameter quantity and feature extraction capability. The compression ratio of 16 in the DAEM module is a classic setting widely used in attention mechanisms, which can effectively reduce computational complexity while fully learning the dependencies between channels. The MFFM module uniformly upsamples multi-scale features to the smallest size for fusion, which not only preserves the abstractness of high-level semantics but also fuses detailed information through feature addition, making the final features have both global context and local detail perception capabilities. In terms of training strategies, the Adam optimizer can adaptively adjust the learning rate to adapt to the update requirements of different parameters. Compared with step-down learning rate scheduling, cosine annealing makes the learning rate change smoother, which helps the model converge to local optima better in the later stages of training. The 10-cycle patience value of the early stopping mechanism provides the model with sufficient optimization space and effectively prevents overfitting. The data augmentation strategy simulates different shooting angles and flipping situations that may occur in actual detection, significantly enhancing the model's generalization ability to complex on-site conditions. Through the above systematic training process and refined parameter configuration, the IR-ADResNet network can fully learn the intrinsic mapping relationship between infrared thermal image features and crack depth, and finally obtain a high-precision and robust deep learning model.

[0144] Step 5: Acquire infrared thermal image sequence and visible light image of the structure to be tested, and obtain dual-channel input image after preprocessing. Input the dual-channel input image into the trained deep learning model to obtain the crack depth prediction value.

[0145] Once the trained and validated IR-ADResNet model is finalized and deployed, a rapid crack depth prediction system suitable for field environments will be established. For field application, the first step is to collect standardized data from the target area of ​​the structure under test.

[0146] In this embodiment, obtaining the predicted crack depth value of the structure under test specifically includes:

[0147] To obtain high-quality infrared thermal images, it is recommended to operate during periods of relatively stable ambient temperature, such as 1-2 hours before sunrise or after sunset. At these times, ambient temperature fluctuations are small and solar radiation interference is weak, which is conducive to obtaining clear thermal contrast. For areas where natural temperature differences are difficult to generate, such as indoor structures or thick, bulky components, active thermal excitation can be performed using a portable, safe heat source.

[0148] In this embodiment, a low-power hot air gun is used for active thermal excitation, and the heating parameters are set as follows: the heating temperature is controlled below 40℃, the heating distance is maintained at 30-50cm, and the heating time is dynamically adjusted according to the thermal diffusivity of the structural material, with the surface temperature of the crack area rising by 5-8℃ being appropriate. During heating, the heat source should be moved evenly to avoid local overheating; immediately after heating, an infrared thermal imager is used to collect a sequence of thermal images of the cooling process.

[0149] The data acquisition equipment should be a handheld or airborne infrared thermal imager with performance similar to that of the training data acquisition equipment. Its technical parameters should meet the following requirements: thermal sensitivity (NETD) less than 0.05℃@30℃, spatial resolution better than 1.0mrad, and pixel resolution not less than 640×480. In this embodiment, the FLIR T1050sc infrared thermal imager is preferred, with a resolution of 1024×768 and a thermal sensitivity of 0.02℃.

[0150] After acquiring the on-site infrared thermal image of the structure under test, the system will automatically execute a preprocessing pipeline that is completely identical to step 2 to ensure that the input data distribution is consistent with the training data:

[0151] For the acquired infrared thermal image sequence, the system automatically calculates the grayscale difference between the crack candidate region and its neighborhood in each frame, and identifies the image frame corresponding to the maximum grayscale difference as the keyframe with the most significant thermal contrast. This process is fully automated and requires no manual intervention. If multiple cracks exist in the sequence, the system can locate the individual keyframes for each crack.

[0152] Gaussian filtering was applied to the selected keyframes with a kernel size of 3×3 and a standard deviation of 0.8 to effectively suppress sensor noise while preserving crack edge details.

[0153] The same CLAHE algorithm as in step 2 was used to enhance the local contrast in the crack area, with the mesh size set to 8×8 and the contrast limit set to 3.0.

[0154] The crack location is automatically located based on edge detection or lightweight object detection models. The region of interest is cropped with the crack as the center and then uniformly scaled to 224×224 pixels using bilinear interpolation.

[0155] The pixel values ​​of the scaled image are linearly normalized from the original 0-65535 (if it is 14-bit or 16-bit raw data) or 0-255 (if it is an 8-bit image) to the range [0,1].

[0156] The processed infrared image is then input into the deployed IR-ADResNet model for forward propagation inference. The model loads the optimal weight file saved from training in step 4, and uses the deep features and nonlinear mapping relationships learned by the model to calculate the predicted depth of the crack in milliseconds, and outputs it directly in millimeters.

[0157] To ensure the reliability of the prediction results, the system can perform multiple data acquisitions and predictions on the same crack area (e.g., three consecutive acquisitions), and calculate the mean and standard deviation of the predicted values. If the ratio of the standard deviation to the mean is less than 10%, the prediction result is considered stable and reliable, and the mean is taken as the final depth value; if the ratio is greater than 10%, the system prompts the inspection personnel to re-acquire data or check for interference sources on site.

[0158] Preferably, to achieve a comprehensive assessment of crack defects, this embodiment further integrates a deep learning-based crack identification and geometric parameter quantization module. This module employs a YOLOv8-based instance segmentation model to accurately detect cracks and extract pixel-level masks from visible light images acquired at the same detection location. In this embodiment, the YOLOv8 model uses the same training strategy as in step 2, i.e., fine-tuning on a self-built crack dataset based on COCO pre-trained weights. Its input image size is 640×640, the confidence threshold is set to 0.5, and the IoU threshold is set to 0.45. The model can distinguish different crack instances and generate high-quality mask images, outputting a binary image where crack pixel values ​​are 1 and background pixel values ​​are 0.

[0159] After obtaining the crack mask, multi-parameter calculations are performed using digital image processing techniques. The specific process is as follows: First, the mask is binarized. Then, the Zhang-Suen skeletonization algorithm is used to iteratively remove edge pixels until a crack centerline with a single pixel width is obtained. The total number of skeleton pixels is counted to obtain the pixel length of the crack. The search continues along the normal direction of the skeleton line to both sides until the mask boundary is encountered, and the crack width at that point is calculated. In this embodiment, sampling points are selected at equal intervals along the skeleton line, with a sampling interval of 5 pixels. The width is calculated for each sampling point, and the maximum value after removing outliers is taken as the maximum pixel width of the crack. The pixel area of ​​the crack is obtained by directly counting the number of all pixels in the mask.

[0160] The above method can only obtain the numerical value of crack defects in pixels, but cannot directly obtain the actual length, width, and area of ​​the cracks. Therefore, it is necessary to convert the pixel size of the image crack defects into the corresponding actual physical size using a reference object method.

[0161] During image acquisition, a reference object of known size must be fixedly placed within the field of view. The reference object should meet the following requirements: a regular geometric shape with high contrast, such as a checkerboard calibration plate (e.g., a checkerboard with sides of 100mm), a standard coin, or a custom-made metal ruler. The reference object should be placed on or as close as possible to the plane where the crack is located to reduce perspective errors. In this embodiment, a ceramic checkerboard calibration plate with sides of 100mm is preferred, as its black and white grid edges are clear and easy for automatic detection.

[0162] Image processing algorithms are used to automatically detect the pixel size of a reference object in an image. For example, for a checkerboard calibration board, its corner points can be detected and the pixel distance between adjacent corner points can be calculated, with the average value taken as the pixel size of the reference object. The scaling factor is the ratio of the actual physical size of the reference object to its pixel size in the image, reflecting the actual physical size corresponding to each pixel.

[0163] For example, if the checkerboard calibration board used has a side length of 100mm, and the detected side length pixel value in the image is 200 pixels, then the scaling factor... = 100mm / 200pixel = 0.5mm / pixel, meaning that each pixel in the image represents an actual length of 0.5mm.

[0164] The pixel length of the crack ,width and area Multiply by the scaling factor respectively, i.e. , , ;in The actual length of the crack. The maximum width of the crack. Let be the area of ​​the crack.

[0165] The system summarizes the calculated predicted crack depth, actual length, maximum width, area, and other parameters, and automatically generates a structural crack detection report. The report includes: detection time, detection location, environmental parameters, crack image, crack depth value, crack geometric parameters, and preliminary structural safety assessment recommendations based on these parameters. Based on the report results, inspectors can comprehensively assess the structural health condition and formulate corresponding maintenance or repair strategies.

[0166] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0167] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0169] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A crack depth prediction method based on infrared thermal imager and deep learning, characterized in that, The specific steps include: Step 1: At the same detection location, simultaneously acquire infrared thermal image sequence and visible light image of the crack area. The infrared thermal image sequence is a thermal image of the crack area cooling process acquired under thermal excitation. Step 2: Determine the keyframes in the infrared thermal image sequence and crop out the region of interest of the keyframes. Spatially register and stitch the region of interest with the visible light image to obtain the multimodal fusion features of the dual-channel input image and the crack region. The multimodal fusion features include infrared thermal features and visible light geometric features. Step 3: Based on multimodal fusion features, obtain multiple estimates of crack depth using at least two different depth estimation methods, perform consistency checks on the multiple estimates, and when the consistency meets the preset requirements, determine the depth label of the crack region and construct a labeled dataset; Step 4: Build a deep learning model, train it using a labeled dataset, and learn the mapping relationship between the dual-channel input image and the depth label; Step 5: Acquire infrared thermal image sequence and visible light image of the structure to be tested, and obtain dual-channel input image after preprocessing. Input the dual-channel input image into the trained deep learning model to obtain the crack depth prediction value.

2. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 1, characterized in that, The infrared thermal imager and the visible light camera are fixed on the same detection platform, and the optical axes of the lenses of the two devices are arranged in parallel or jointly calibrated before acquisition to ensure that the two images have overlapping fields of view.

3. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 2, characterized in that, Before data collection, the crack area is heated using a thermal excitation device. During the heating process, the surface temperature of the crack area is monitored in real time using an infrared thermal imager. When the temperature at the center point of the crack area reaches a preset threshold or the rate of temperature rise first shows a downward inflection point, the heating stage is determined to be over and thermal excitation is stopped immediately. The thermal excitation device is a low-power hot air gun, and the heating temperature is controlled within 40 degrees Celsius. The heating time is adjusted according to the thermal diffusivity of the material to which the crack area belongs; the greater the thermal diffusivity, the longer the heating time. Immediately after heating is completed, the infrared thermal imager is activated to acquire a sequence of infrared thermal images of the crack area during natural cooling at a fixed frame rate. At the same time, the visible light camera is triggered to acquire a visible light image that is aligned with the start frame time of the infrared thermal image sequence.

4. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 3, characterized in that, Calculate the temperature difference between the crack region and the adjacent background region in each frame of the infrared thermal image sequence, and take the image frame corresponding to the maximum temperature difference as the key frame with the most significant thermal contrast. Gaussian filtering was applied to the selected keyframe infrared thermal image and visible light image for noise reduction. A contrast-limited adaptive histogram equalization algorithm is used to enhance the local contrast of the crack region in the keyframe infrared thermal image. The region of interest (ROI) of the keyframe infrared thermal image is cropped with the crack as the center and denoted as the infrared ROI image. The infrared ROI image and the visible light image are spatially registered using a feature point-based registration method. Based on the registration result, the image region at the same position as the infrared ROI image is cropped from the visible light image and denoted as the visible light crack region image. The infrared region of interest (ROI) image and the visible light crack region image are scaled to the same pixel size. The infrared ROI image is normalized based on the temperature value of the infrared ROI image, and the pixel values ​​of the visible light crack region image are linearly normalized to the [0,1] interval. During the acquisition process, the acquisition time corresponding to each frame of infrared thermal image, as well as the ambient temperature and humidity parameters corresponding to that acquisition time, are recorded.

5. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 4, characterized in that, The logic for cropping the infrared region of interest image is as follows: A crack instance segmentation model is used to extract pixel-level masks of cracks. The minimum bounding rectangle of the mask is calculated, and a preset pixel range is extended outward based on the minimum bounding rectangle to define the region of interest (ROI). This ROI ensures that the ROI completely includes the crack body and the heat-affected zone around the crack. When there are multiple cracks in the image, the corresponding ROI is defined independently for each crack. The defined ROI is then cropped from the corresponding infrared thermal image, and the cropped image is the infrared ROI image.

6. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 4, characterized in that, The logic for determining multimodal fusion features is as follows: Based on the infrared region of interest image after spatial registration, cropping and scaling, the temperature gradient value of each pixel is extracted, the rate of change of temperature gradient in the horizontal and vertical directions is calculated, and the average temperature, maximum temperature, minimum temperature and temperature drop rate of the crack region are calculated, which together constitute the infrared thermal features. On the visible light crack region image, a crack instance segmentation model is used to extract the pixel-level mask of the crack. After binarizing the pixel-level mask, the center line of the crack is extracted by the skeletonization algorithm. The crack pixel length is obtained by counting the number of skeleton pixels. The width at each position is calculated by searching the mask boundary along the normal direction of the skeleton line. The average width and maximum width of the crack are obtained. At the same time, the crack pixel area is obtained by counting the total number of pixels in the mask. Together, they constitute the visible light geometric features.

7. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 6, characterized in that, The logic for performing consistency checks is as follows: Based on the temperature gradient and temperature drop rate in the infrared thermal features, and combined with the recorded ambient temperature and humidity parameters at the acquisition time as boundary conditions, a heat conduction inverse problem model is established. A thermal response database under different depth conditions is constructed through finite element analysis. The measured infrared thermal features are matched and inverted with the database to obtain the first estimate of the crack depth. Based on the maximum crack width in the visible light geometric characteristics, a second estimate of the crack depth is obtained by using a pre-established empirical relationship between width and depth through linear regression or power function fitting; the empirical relationship between width and depth is derived from the statistical data of destructive tests on specimens of the same material type. The crack depth at the detection location is measured using the ultrasonic pulse echo method or core drilling method to obtain a third estimate of the crack depth. The first, second, and third estimates are compared, and their mean and standard deviation are calculated. When the ratio of the standard deviation to the mean is less than 10%, the estimate is determined to have high confidence, and the mean is used as the depth label for the crack region. When the ratio of the standard deviation to the mean is not less than 10%, a verification mechanism is triggered. The verification mechanism includes adjusting the boundary conditions of the inverse heat conduction model and repeating the above steps until the requirement is met, at which point the depth label is determined. The infrared region of interest image and the visible light crack region image are associated and stored with their corresponding depth labels to form a labeled dataset for training deep learning models.

8. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 4, characterized in that, Learning the mapping relationship specifically includes: Infrared region of interest images and visible light crack region images were obtained from a labeled dataset. The two images were spatially registered and scaled to the same pixel size. The infrared region of interest image and the visible light crack region image are stitched together along the channel dimension to form a dual-channel input image. The first channel is the registered infrared region of interest image, and the second channel is the visible light crack region image. At the same time, the depth label is used as the regression target value. A residual network is used as the backbone network, including an initial convolutional layer, a batch normalization layer, an activation function layer, and a max pooling layer, as well as four stages of residual block groups, to extract deep features at different levels from a dual-channel input image. Dual-path attention enhancement modules are embedded after the third and fourth stages of the backbone network. These modules are designed for the unique temperature field distribution caused by thermal diffusion in the crack region. They adaptively recalibrate the feature map through parallel channel attention paths and spatial attention paths: the channel attention path recalibrates the weights of each channel in the feature map to strengthen the thermal gradient feature channels that are strongly correlated with the crack depth; the spatial attention path recalibrates the weights of each spatial location in the feature map to focus on the crack body and the surrounding thermally affected area. The weights of the two recalibrated modules are then fused and multiplied with the feature map to output the enhanced crack region feature map. The feature maps output from the second, third, and fourth stages of the backbone network are input into the multi-scale feature fusion module. The feature map of the second stage retains spatial details including crack edges and morphology, while the feature map of the fourth stage includes semantic information of the overall heat diffusion pattern. The feature maps of each stage are adjusted to the same spatial size through upsampling, the number of channels is unified through convolutional layers, and feature fusion is performed through element-wise addition to output a fused feature map containing multi-level semantic information. Global average pooling is performed on the fused feature map. The pooled features are then sequentially input into the first fully connected layer and the second fully connected layer. The first fully connected layer reduces the feature dimension to 128 dimensions and connects it to the activation function. The second fully connected layer maps the 128-dimensional features to a one-dimensional crack depth prediction value. Using mean squared error as the loss function, an optimizer is used to iteratively optimize the network parameters. During training, the change in the loss value on the validation set is monitored. When the validation set loss does not decrease for several consecutive training epochs, training is terminated early, and the model with the lowest validation set loss is saved as the final deep learning model.

9. The crack depth prediction method based on infrared thermal imager and deep learning according to claim 8, characterized in that, Obtaining the predicted crack depth of the structure under test specifically includes: In the target detection area of ​​the structure to be tested, the same acquisition platform and parameter settings as in step 1 are used to simultaneously acquire infrared thermal image sequences and visible light images of the surface where the crack is located. Keyframes were determined from the acquired infrared thermal image sequence, and the same cropping, preprocessing, and registration methods were used to spatially register the infrared region of interest image of the structure under test with the visible light crack region image. After spatial registration, the infrared region of interest image of the structure under test and the visible light crack region image are stitched together by channel dimension to form a dual-channel input image of the structure under test. This image is then input into the trained deep learning model to obtain the predicted crack depth value of the structure under test.

Images