Road crack defect detection method and system based on multi-scale feature fusion

Abstract

The application discloses a road crack defect detection method and system based on multi-scale feature fusion, belongs to the technical field of road detection, acquires a visible light image sequence, an infrared thermal imaging sequence and three-dimensional laser point cloud data of a target road section, constructs a three-dimensional surface grid model, and extracts a curvature significant region set; the visible light image sequence is input into a double-branch heterogeneous encoder, multi-scale structural damage explicit features and surface continuity disturbance features are extracted, and the curvature significant region set is used as spatial attention prior to perform weighted enhancement; inter-modal iterative collaborative optimization is performed on the two enhanced features to obtain deep fusion features; different scale up-sampling paths are adaptively selected according to the local energy distribution of the deep fusion features to generate a crack probability map; the position, scale and confidence of the crack defect are determined according to the crack probability map and a crack geometric constraint condition to generate a detection result; and the application can realize high robustness and high accuracy detection of multi-scale cracks.

Description

Technical Field

[0001] This invention relates to the field of road inspection technology, specifically to a method and system for detecting road crack defects based on multi-scale feature fusion. Background Technology

[0002] Road crack defect detection is a crucial aspect of road maintenance and management, especially in semi-enclosed or concealed road sections such as tunnel entrances and exits, bridge pavement layers, and urban underpasses. Cracks often exhibit diverse shapes, large scale differences, and high overlap with background textures. Existing detection methods largely rely on single-scale visual features or convolutional structures with fixed receptive fields, making it difficult to simultaneously capture the global structural information of both subtle early microcracks and wide, penetrating cracks. In complex scenarios such as uneven lighting, road marking interference, and shadow coverage, conventional models are prone to misidentifying pseudo-textures as cracks. Even more challenging is that when cracks appear in water-covered areas or under low-light conditions at night, traditional visible light image-based detection methods are almost ineffective. While simply introducing thermal imaging or 3D point clouds, existing methods merely stitch together multi-source data as independent input channels, lacking explicit modeling of the cross-scale physical relationships between the multi-source data. This leads to mutual inhibition between different modalities at the feature level, weakening or even obliterating key potential hazards during the fusion process. The false negative rate remains high even in complex environments, seriously threatening driving safety and structural durability. Summary of the Invention

[0003] The purpose of this invention is to provide a road crack defect detection method and system based on multi-scale feature fusion to address the shortcomings of the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a road crack defect detection method based on multi-scale feature fusion, comprising: S100, acquire multi-source acquisition data of the target road section, the multi-source acquisition data including visible light image sequence, infrared thermal imaging sequence and three-dimensional laser point cloud data; S200, the infrared thermal imaging sequence is jointly registered with the three-dimensional laser point cloud data to construct a three-dimensional surface mesh model M with temperature attributes, and curvature field analysis is performed on M to extract the set of curvature significant regions R that reflect local geometric abrupt changes; S300, the visible light image sequence is input into a dual-branch heterogeneous encoder, wherein the first branch uses dilated convolutional pyramids to extract multi-scale structural damage explicit feature maps Fdam, and the second branch uses deformable convolution to extract surface continuity perturbation feature maps Fdis aligned with geometric deformation. R is used as a spatial attention prior to weight and enhance Fdam and Fdis respectively, to obtain enhanced structural damage explicit feature Fdam' and enhanced surface continuity perturbation feature Fdis'. S400 performs intermodal iterative collaborative optimization on Fdam' and Fdis'. In each iteration, Fdis' is restructured and recalibrated based on the current Fdam', and Fdam' is back-projected in detail based on the current Fdis', until the iteration converges and the deep fusion feature Ffuse is obtained. S500 adaptively selects upsampling paths of different scales based on the local energy distribution of Ffuse to generate a crack probability map P; S600: Based on the crack probability map P and the preset crack geometric constraints, determine the location, scale, and confidence level of crack defects in the target road section, and generate crack detection results.

[0005] Preferably, in step S200, constructing a three-dimensional surface mesh model M with temperature properties specifically includes: S210, Based on the three-dimensional spatial coordinate information contained in the three-dimensional laser point cloud data, an initial three-dimensional surface mesh is constructed using the Poisson surface reconstruction algorithm, and the initial three-dimensional surface mesh is subjected to Laplacian smoothing to obtain a reference geometric mesh model. S220, extract the temperature value matrix corresponding to each frame of the infrared thermal imaging sequence, and project the temperature value matrix onto each vertex of the reference geometric mesh model through the inverse distance weight interpolation algorithm, so that each vertex obtains the corresponding temperature attribute value, forming a temperature-attached mesh model. S230, The temperature-attached mesh model is input into a pre-trained adaptive temperature-geometry joint optimization network. The network uses the structural consistency between the vertex temperature gradient and the local curvature change as a constraint to iteratively optimize the vertex position and temperature value until the temperature field and the geometric field reach the edge alignment state, and outputs a three-dimensional surface mesh model M.

[0006] Preferably, in step S200, curvature field analysis is performed on M to extract a set of curvature-significant regions R that reflect local geometric abrupt changes, specifically including: S240, calculate the Gaussian curvature and mean curvature of each vertex in the three-dimensional surface mesh model M with temperature properties at multiple neighborhood scales, and use the standard deviation of the curvature values ​​at each scale as a measure of the multi-scale curvature change of the vertex to obtain the curvature change field Cvar. S250, the curvature change field Cvar and the temperature attribute value corresponding to each vertex in the three-dimensional surface mesh model M with temperature attribute are jointly normalized to construct a curvature-temperature joint saliency map Smap; S260, perform morphological connected component extraction on the curvature-temperature joint saliency map Smap, filter out isolated noise points with an area smaller than the preset minimum crack unit, and determine the projection region of the remaining connected components on the three-dimensional surface mesh model M as the curvature saliency region set R.

[0007] Preferably, in step S300, the first branch uses a dilated convolutional pyramid to extract a multi-scale structural damage explicit feature map Fdam, specifically including: Each frame of the visible light image sequence is input into the first branch, which contains at least three dilated convolution branches set in parallel. The dilation rate of each dilated convolution branch is in an increasing sequence. The feature map output by each dilated convolution branch is subjected to pixel-wise weighted fusion. The fused feature map is then multiplied with the set of curvature significant regions by a spatial mask to obtain the multi-scale structural damage explicit feature map Fdam.

[0008] Preferably, in step S300, the second branch uses deformable convolution to extract a surface continuity perturbation feature map Fdis aligned with geometric deformation. Specifically, this includes: inputting each frame of visible light image in the visible light image sequence into the second branch; the second branch first extracts an initial feature map through a standard convolutional layer, and then superimposes a deformable offset field on the initial feature map; the deformable offset field is generated by the set of curvature significant regions, causing the convolution sampling position to shift along the geometric direction of the curvature significant regions, thereby extracting the surface continuity perturbation feature map Fdis.

[0009] Preferably, the surface continuity perturbation feature map Fdis and the multi-scale structural damage explicit feature map Fdam are concatenated along the channel dimension, and the concatenated feature map is input to the channel attention module. The channel attention module performs global average pooling and global max pooling on each channel feature and then generates channel weights through a shared multilayer perceptron. The channel weights are multiplied with the concatenated feature map channel by channel to output the aligned multi-scale structural damage explicit features and surface continuity perturbation features.

[0010] Preferably, step S400 specifically includes: S410, perform edge gradient extraction on the enhanced surface continuity perturbation features to obtain the surface continuity perturbation gradient map, and perform edge gradient extraction of the same size on the enhanced structural damage explicit features to obtain the structural damage explicit gradient map. S420, calculate the gradient difference map between the surface continuity perturbation gradient map and the structural damage explicit gradient map, and generate a detail compensation mask based on the gradient difference map. In the detail compensation mask, the region where the gradient difference exceeds a preset difference threshold is assigned a first preset value, and the remaining region is assigned a second preset value. S430, the detail compensation mask is multiplied channel-by-channel and element-by-element by the enhanced surface continuity perturbation feature to obtain the detail guiding feature, and the detail guiding feature is added element-by-element by the enhanced structure damage explicit feature to obtain the updated enhanced structure damage explicit feature. S440, repeat steps S410 to S430 for a total of a preset number of iterations. Each iteration takes the currently updated explicit features of enhanced structural damage and the currently updated features of enhanced surface continuity perturbation as inputs, and outputs the explicit features of enhanced structural damage obtained from the last iteration as deep fusion features.

[0011] Preferably, step S500 specifically includes: S510, perform multi-branch parallel processing on the deep fusion features. The multi-branch includes a first upsampling branch, a second upsampling branch and a third upsampling branch. Each branch adopts upsampling operations with different magnification and follows dilated convolutional layers with different dilation rates to output three decoded feature maps of different scales. S520, calculate the local energy distribution map of the deep fusion feature. The local energy distribution map is obtained by calculating the sum of squares of feature values ​​within a sliding window, and a path selection weight map is generated based on the local energy distribution map. S530, the three decoded feature maps of different scales are weighted and fused with the path selection weight map. For each spatial location, the feature vectors of the location on the three decoded feature maps are multiplied by the corresponding weight values ​​and then added together to obtain the fused decoded feature map. S540, the fused decoded feature map is passed sequentially through a convolutional layer and a Sigmoid activation function layer to output a crack probability map. The value of each pixel in the crack probability map represents the probability that the location belongs to the crack category.

[0012] Preferably, step S600 specifically includes: S610, perform adaptive threshold segmentation on the crack probability map, mark the pixels with probability values ​​greater than the segmentation threshold as crack candidate pixels, and perform connected component analysis on the crack candidate pixels to extract several initial crack connected regions. S620, extract the skeleton of each initial crack connected region to obtain the corresponding crack centerline, and calculate the local width at each point on the crack centerline to obtain the crack width distribution curve. S630, each initial crack connected region is screened according to preset crack geometric constraints. The crack geometric constraints include a minimum length threshold, a maximum width threshold, and a range of length to width ratios. The initial crack connected regions that satisfy all geometric constraints are determined as valid crack regions. S640, calculate the confidence level of the effective crack region based on the probability value of the corresponding position of the effective crack region on the crack probability map, and output the position coordinates, crack length, crack width distribution and confidence level of each effective crack region to generate crack detection results.

[0013] This invention also provides a road crack defect detection system based on multi-scale feature fusion, comprising: Data acquisition module: acquires multi-source data of the target road section, including visible light image sequences, infrared thermal imaging sequences, and three-dimensional laser point cloud data; The geometry-temperature fusion modeling module: jointly registers the infrared thermal imaging sequence with the three-dimensional laser point cloud data to construct a three-dimensional surface mesh model M with temperature attributes, and performs curvature field analysis on M to extract the set of curvature-significant regions R that reflect local geometric abrupt changes; Dual-branch heterogeneous feature extraction module: The visible light image sequence is input into the dual-branch heterogeneous encoder, wherein the first branch uses dilated convolutional pyramid to extract multi-scale structural damage explicit feature map Fdam, and the second branch uses deformable convolution to extract surface continuity perturbation feature map Fdis aligned with geometric deformation. R is used as a spatial attention prior to weight and enhance Fdam and Fdis respectively, to obtain enhanced structural damage explicit feature Fdam' and enhanced surface continuity perturbation feature Fdis'. Intermodal iterative collaborative optimization module: Perform intermodal iterative collaborative optimization on Fdam' and Fdis'. In each iteration, Fdis' is restructured and recalibrated based on the current Fdam', and Fdam' is back-projected in detail based on the current Fdis', until the iteration converges and the deep fusion feature Ffuse is obtained. Dynamic routing decoding module: Adaptively selects upsampling paths of different scales based on the local energy distribution of Ffuse to generate a crack probability map P; Crack detection output module: Based on the crack probability map P and the preset crack geometric constraints, determine the location, scale and confidence level of crack defects in the target road section, and generate crack detection results.

[0014] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. This invention achieves significant technical effects in multiple dimensions by constructing a complete technical chain from multi-source data acquisition to crack detection output. First, at the data fusion level, this invention overcomes the limitations of traditional methods that simply stitch together multi-source data. It jointly registers infrared thermal imaging sequences with 3D laser point cloud data to construct a 3D surface mesh model with temperature attributes. Furthermore, it extracts a set of significant curvature regions through curvature field analysis, deeply fusing geometric abrupt change information with temperature anomaly information to form a high-precision spatial prior guidance. This prior guidance effectively suppresses interference from complex backgrounds such as road markings, shadow coverage, and uneven illumination, allowing subsequent feature extraction to focus on areas with genuine structural anomalies, significantly reducing the false detection rate. Second, at the feature extraction level, this invention designs a dual-branch heterogeneous encoder. It employs a dilated convolutional pyramid to extract explicit features of multi-scale structural damage, and deformable convolution to extract surface continuity perturbation features aligned with geometric deformation. These two feature types complement each other, capturing both the fine texture of subtle early microcracks and perceiving the global structural information of penetrating wide cracks, overcoming the limitation of single-scale features in handling cracks of different scales.

[0015] 2. This invention utilizes an intermodal iterative collaborative optimization mechanism to perform detailed back-projection of explicit features of enhanced structural damage, guided by enhanced surface continuity perturbation features. This achieves deep complementarity and fusion of structural and texture features, resulting in deeply fused features that possess both macroscopic structural integrity and microscopic detail richness. Building upon this, the invention introduces a heterogeneous feature decoder with a dynamic routing mechanism. Based on the local energy distribution of the deeply fused features, it adaptively selects upsampling paths at different scales, achieving adaptive and accurate reconstruction of cracks at different scales. This solves the problem of insufficient adaptability of traditional fixed upsampling methods in complex scenarios. Finally, the detection results are filtered through preset crack geometric constraints, effectively eliminating non-crack interference areas and ensuring high reliability and practicality of the output crack location, scale, and confidence level. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart of the method of the present invention.

[0018] Figure 2 This is a flowchart of the system modules of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1, please refer to Figure 1 As shown in this embodiment, the road crack defect detection method based on multi-scale feature fusion includes: S100, acquire multi-source acquisition data of the target road section, the multi-source acquisition data including visible light image sequence, infrared thermal imaging sequence and three-dimensional laser point cloud data.

[0021] In this embodiment, multi-source acquisition data of the target road section is first obtained. The target road section specifically refers to semi-enclosed or concealed road areas such as tunnel entrance / exit sections, bridge pavement layers, or urban underpasses. These areas are characterized by complex lighting conditions, strong background interference, and easily obscured crack features.

[0022] The multi-source acquired data includes visible light image sequences, infrared thermal imaging sequences, and three-dimensional laser point cloud data. To achieve synchronous acquisition of multi-source data, this embodiment employs a multi-sensor fusion acquisition platform mounted on a mobile acquisition vehicle. The platform includes: Visible light camera: Employs a high-resolution industrial camera to continuously acquire visible light image sequences of the target road section at a fixed frame rate, used to capture road surface texture, crack morphology, and surrounding environmental information; Infrared thermal imager: Synchronously triggered with a visible light camera, it collects infrared thermal imaging sequences at corresponding locations to reflect the characteristics of road surface temperature distribution, especially the temperature difference response between cracks and the surrounding concrete matrix in areas covered by water or under low light conditions at night. 3D laser scanner: Employs line-scan or area-array LiDAR to acquire 3D laser point cloud data of the target road section. The point cloud data contains precise 3D spatial coordinates of the road surface and surrounding structures, used to reconstruct the road surface geometry and local undulation features.

[0023] To ensure the alignment of multi-source data in both spatial and temporal dimensions, the acquisition platform also integrates a high-precision Global Navigation Satellite System (GNSS) receiver and an inertial measurement unit (IMU) to record the position and attitude information corresponding to each frame of acquired data. During data acquisition, the visible light camera, infrared thermal imager, and 3D laser scanner synchronously acquire data through a unified timestamp triggering mechanism, ensuring that the data of each modality are precisely corresponding in time.

[0024] After the data acquisition is completed, this embodiment further preprocesses the multi-source acquired data, including: Denoising and contrast enhancement are performed on visible light image sequences and infrared thermal imaging sequences; Based on the pose information provided by GNSS / IMU, coarse registration is performed between 3D laser point cloud data and 2D image sequences to eliminate spatial offset caused by differences in sensor installation positions. Remove abnormal data points caused by vehicle vibration or environmental obstruction.

[0025] After the above processing, spatially aligned and temporally synchronized multi-source acquisition data is obtained for subsequent steps. This multi-source acquisition strategy effectively overcomes the problem of insufficient information acquisition capability of a single sensor in complex road environments, providing a complete data foundation for subsequent fusion detection.

[0026] S200, the infrared thermal imaging sequence is jointly registered with the three-dimensional laser point cloud data to construct a three-dimensional surface mesh model M with temperature attributes, and curvature field analysis is performed on M to extract the set of curvature significant regions R that reflect local geometric abrupt changes.

[0027] In this embodiment, an initial three-dimensional surface mesh is first constructed based on the three-dimensional spatial coordinate information contained in the three-dimensional laser point cloud data using the Poisson surface reconstruction algorithm. The three-dimensional laser point cloud data is acquired by a three-dimensional laser scanner and contains a set of discrete point coordinates of the target road section's pavement and surrounding structures in three-dimensional space. Each point contains its horizontal, vertical, and triangular coordinate values ​​in space.

[0028] The specific implementation of the Poisson surface reconstruction algorithm is as follows: First, an octree structure is constructed based on the spatial distribution of all points in the point cloud, and the space is adaptively divided, recursively dividing the spatial region occupied by the point cloud into cubic units of unequal sizes. Then, the directed normal vector of each sampling point is calculated. The normal vector is obtained through principal component analysis, that is, the covariance matrix formed by the sampling point and its neighborhood points is decomposed into eigenvalues, and the eigenvector corresponding to the smallest eigenvalue is taken as the direction of the normal vector. The orientation of the entire point cloud is adjusted to make all normal vectors point to the same side. Next, a vector field is defined to represent the gradient direction at any position in space. This vector field is obtained by interpolation of the directed normal vectors of all sampling points. Then, the Poisson equation is solved. The Poisson equation describes that the Laplace operator of the indicator function is equal to the divergence of the vector field. The indicator function is obtained by solving this equation. Finally, the isosurface of the indicator function is extracted by the moving cube algorithm to generate a continuous initial three-dimensional surface mesh.

[0029] Since the initial 3D surface mesh may contain local noise and unevenness, this embodiment further performs Laplacian smoothing on the initial 3D surface mesh. Specifically, for each vertex in the initial 3D surface mesh, its updated position is equal to the original position of the vertex plus a smoothing factor multiplied by the sum of the offset vectors of all first-order neighboring vertices relative to the vertex. Here, first-order neighboring vertices refer to all vertices directly connected to the vertex through a mesh edge, and the smoothing factor is a preset value; in this embodiment, the smoothing factor is 0.3. The above calculation is repeated for all vertices in the initial 3D surface mesh to complete one smoothing iteration. This embodiment performs three consecutive smoothing iterations to eliminate acquisition noise while preserving the main geometric features of the mesh surface, ultimately obtaining a baseline geometric mesh model. This baseline geometric mesh model is used for subsequent temperature attribute attachment and geometric field analysis.

[0030] After obtaining the baseline geometric mesh model, this embodiment extracts the temperature value matrix corresponding to each frame of the infrared thermal imaging sequence. The infrared thermal imaging sequence is acquired by an infrared thermal imager, and each frame corresponds to a two-dimensional temperature value matrix. The value of each element in the matrix represents the temperature value of the corresponding pixel, in degrees Celsius.

[0031] To achieve the attachment of temperature attributes to the 3D mesh model, this embodiment employs an inverse distance weighted interpolation algorithm. The specific process is as follows: First, based on the spatial alignment relationship established in step S100, the spatial location range corresponding to each frame of the infrared thermal imaging sequence is determined, and a correspondence is established between the temperature value matrix of that frame and the vertices located within that spatial range in the reference geometric mesh model. For any vertex in the reference geometric mesh model, the projection coordinates of that vertex on the image plane are first calculated according to the camera imaging model. These projection coordinates are obtained by multiplying the vertex's 3D coordinates with the camera's intrinsic and extrinsic parameter matrices. Then, multiple pixels within a preset radius around the projection point are selected as reference points. The preset radius is set according to the image resolution; in this embodiment, the preset radius is 5 pixels. For each reference point, the Euclidean distance between it and the projection point is calculated, i.e., the square root of the sum of the squared differences in the horizontal and vertical coordinates. The weight of each reference point is set to the reciprocal of the square of the Euclidean distance between the reference point and the projection point.

[0032] The temperature value of each reference point is multiplied by its corresponding weight, and the sum is then divided by the total sum of all weights to obtain the temperature attribute value of that vertex. Through this interpolation calculation, each vertex in the baseline geometric mesh model is assigned a corresponding temperature attribute value, forming a temperature-attached mesh model. Each vertex in this model contains three-dimensional spatial coordinates and a temperature attribute value.

[0033] In this embodiment, the temperature-attached mesh model is input into a pre-trained adaptive temperature-geometry joint optimization network. The training process of the adaptive temperature-geometry joint optimization network is as follows: First, a training dataset is constructed, which contains multiple sets of 3D mesh samples with known temperature fields and geometric fields precisely aligned. Each set of samples includes a temperature-attached mesh model and its corresponding real-aligned mesh model. Second, the network structure is designed, which consists of alternating geometric update modules and temperature update modules connected in series. The geometric update module is based on a graph convolutional neural network, with the current vertex coordinates and vertex temperature value as input and the adjusted vertex coordinates as output. The temperature update module is also based on a graph convolutional neural network, with the adjusted vertex coordinates and current temperature value as input and the updated temperature value as output. Finally, the structural consistency between the temperature field and the geometric field is used as the loss function for training. The structural consistency loss function is composed of the average value of the cosine of the angle between the temperature gradient direction and the geometric normal vector change direction until the loss function converges.

[0034] During the inference phase, the network uses the structural consistency between vertex temperature gradients and local curvature changes as a constraint to perform alternating iterative optimization of vertex positions and temperature values. The specific optimization process is as follows: In the first iteration, the geometry update module, guided by the current vertex temperature gradient distribution, adjusts the vertex spatial positions to ensure that the geometric curvature changes are consistent with the temperature gradient direction. The temperature update module then recalculates the temperature values ​​of each vertex based on the adjusted geometry, ensuring that the temperature field distribution conforms to the heat conduction continuity constraint. This alternating optimization process is repeated until the average offset of vertex positions between two adjacent iterations is less than a preset displacement threshold and the average change in temperature values ​​is less than a preset temperature threshold, at which point the iteration stops. In this embodiment, the preset displacement threshold is 0.01 mm, and the preset temperature threshold is 0.1 degrees Celsius.

[0035] After the aforementioned iterative optimization, the temperature field and the geometric field reach edge alignment, meaning that the temperature abrupt change boundary and the geometric structure edge precisely coincide in space. The output mesh model at this point is a three-dimensional surface mesh model with temperature attributes. Each vertex in this model contains optimized three-dimensional spatial coordinates and temperature attribute values, achieving a deep fusion of geometric and temperature information.

[0036] After obtaining a 3D surface mesh model with temperature properties, this embodiment calculates the Gaussian curvature and mean curvature of each vertex in the model at multiple neighborhood scales. Specifically, for any vertex in the model, its neighborhood scale set is defined to include a first-order neighborhood, a second-order neighborhood, and a third-order neighborhood. The first-order neighborhood is the set of all vertices directly connected to the vertex, the second-order neighborhood is the set of all vertices within two hops of the vertex, and the third-order neighborhood is the set of all vertices within three hops of the vertex.

[0037] For each neighborhood scale, the Gaussian curvature and mean curvature of the vertex at that scale are calculated. The Gaussian curvature is calculated using discrete differential geometry: first, the normal vectors of each neighboring facet at the vertex are calculated; each neighboring facet is a triangular facet with the vertex as one of its vertices within its neighborhood. Then, by integrating the changes in the normal vectors of the neighboring faces, an approximate value of the Gaussian curvature of the vertex at that scale is obtained. The mean curvature is obtained by calculating the positional offset relationship between the vertex and its neighboring vertices. Specifically, the difference vectors between the vertex and all vertices in its neighborhood are calculated, the sum of the magnitudes of these difference vectors is obtained, and then divided by the number of neighboring vertices. The resulting value is taken as the mean curvature of the vertex at that scale.

[0038] After obtaining the Gaussian curvature and mean curvature at each scale, the standard deviation of the curvature values ​​at each scale is calculated. For Gaussian curvature, first calculate the arithmetic mean of the Gaussian curvature of the vertex at each scale, then calculate the square of the difference between the Gaussian curvature at each scale and the mean, sum the squared values ​​at each scale, divide by the number of neighboring scales, and then take the square root of the quotient to obtain the multi-scale standard deviation of the Gaussian curvature of the vertex. The multi-scale standard deviation of the mean curvature of the vertex is obtained using the same calculation method.

[0039] The multi-scale standard deviation of the Gaussian curvature of the vertex is added to the multi-scale standard deviation of the mean curvature, and the sum is used as a measure of the multi-scale curvature change of the vertex. The above calculation is performed on all vertices in the 3D surface mesh model with temperature properties to obtain the curvature change field, which records the curvature change amplitude information at each vertex.

[0040] After obtaining the curvature change field, this embodiment performs joint normalization processing on it and the temperature attribute value corresponding to each vertex in the three-dimensional surface mesh model with temperature attributes. Specifically, for any vertex in the three-dimensional surface mesh model with temperature attributes, its curvature change value is the value corresponding to that vertex in the curvature change field, and the temperature attribute value is the temperature value carried by that vertex in the model.

[0041] First, the curvature change value is normalized: traverse all vertices in the curvature change field and find the minimum and maximum values ​​of the curvature change value; for each vertex, subtract the minimum value from the curvature change value of that vertex and divide by the difference between the maximum and minimum values ​​to obtain the normalized curvature change value of that vertex.

[0042] Secondly, the temperature attribute values ​​are normalized. To highlight the temperature gradient variation region, this embodiment uses the degree of temperature gradient deviation from the local mean as the temperature significance index. Specifically, for each vertex, the average temperature of all vertices in its local neighborhood is calculated, where the local neighborhood is the first-order neighborhood of that vertex. Then, the absolute value of the difference between the temperature value of that vertex and the local average temperature value is calculated to obtain the temperature gradient deviation of that vertex. By traversing all vertices, the minimum and maximum values ​​of the temperature gradient deviation are found. For each vertex, the minimum value is subtracted from the temperature gradient deviation of that vertex, and then divided by the difference between the maximum and minimum values ​​to obtain the normalized temperature gradient deviation of that vertex.

[0043] Finally, a curvature-temperature joint significance map is constructed. The significance value for each vertex is calculated through weighted fusion: the normalized curvature change value of the vertex is multiplied by a preset first fusion weight, and the normalized temperature gradient deviation of the vertex is multiplied by a preset second fusion weight. The sum is used as the significance value of the vertex. In this embodiment, the first fusion weight is set to 0.6, and the second fusion weight is set to 0.4 to emphasize the dominant role of geometric curvature change in crack detection, while retaining the contribution of temperature anomaly regions. The significance value is positively correlated with the magnitude of curvature change and the degree of temperature gradient deviation from the local mean; that is, the more drastic the curvature change or the greater the temperature gradient deviation of a vertex, the higher its significance value. After calculating the significance values ​​for all vertices, a curvature-temperature joint significance map is formed.

[0044] This embodiment performs morphological connected component extraction based on an adaptive threshold on the curvature-temperature joint saliency map. First, the adaptive segmentation threshold is determined. This embodiment uses Otsu's method to calculate the optimal threshold: the saliency values ​​of all vertices in the curvature-temperature joint saliency map are used as a numerical statistical histogram, with the horizontal axis representing the saliency value interval and the vertical axis representing the number of vertices falling within that interval; all possible saliency values ​​are traversed as candidate thresholds. For each candidate threshold, vertices are divided into two categories: vertices with saliency values ​​greater than or equal to the candidate threshold and vertices with saliency values ​​less than the candidate threshold; the inter-class variance of the saliency values ​​of the two categories of vertices is calculated, and the candidate threshold corresponding to the largest inter-class variance is taken as the segmentation threshold.

[0045] Then, the curvature-temperature joint saliency map is binarized according to the segmentation threshold: vertices with saliency values ​​greater than or equal to the segmentation threshold are marked as candidate region vertices, and vertices with saliency values ​​less than the segmentation threshold are marked as non-target regions.

[0046] Next, morphological connected component extraction is performed on the binarized candidate region vertices. Specifically, on the mesh structure of the 3D surface mesh model with temperature attributes, a breadth-first search algorithm based on adjacency is adopted: starting from any unvisited candidate region vertex, the algorithm traverses the candidate region vertices connected to that vertex along the mesh edge. Each vertex is marked as visited and added to the current connected component until no further expansion is possible, forming a connected component; the above process is repeated until all candidate region vertices have been visited. Each connected component corresponds to a candidate crack region.

[0047] Finally, the extracted connected components are filtered by area. The minimum crack element area is preset to 0.5 square centimeters. All connected components are traversed, and the projected area of ​​each connected component on the 3D surface mesh model with temperature attributes is calculated. Isolated noisy connected components with a projected area of ​​less than 0.5 square centimeters are filtered out.

[0048] The projection regions of the remaining connected components onto the 3D surface mesh model with temperature properties are defined as a set of curvature saliency regions. Each region in this set corresponds to a potential crack defect location on the road surface. These regions simultaneously satisfy the characteristics of abrupt changes in geometric curvature and abnormal temperature distribution, providing accurate spatial prior guidance for fine feature extraction and fusion detection in subsequent steps.

[0049] S300, the visible light image sequence is input into a dual-branch heterogeneous encoder, wherein the first branch uses a dilated convolutional pyramid to extract a multi-scale structural damage explicit feature map Fdam, and the second branch uses deformable convolution to extract a surface continuity perturbation feature map Fdis aligned with geometric deformation. R is used as a spatial attention prior to weight and enhance Fdam and Fdis respectively, resulting in enhanced structural damage explicit feature Fdam' and enhanced surface continuity perturbation feature Fdis'.

[0050] In this embodiment, each frame of the visible light image sequence is input to the first branch. The first branch contains three dilated convolution branches arranged in parallel, with the dilation rate of each dilated convolution branch increasing sequentially. Specifically, this embodiment sets up three dilated convolution branches with dilation rates of 2, 4, and 6, respectively. Each dilated convolution branch contains a dilated convolution layer with a kernel size of 3x3, followed by a batch normalization layer and a linear rectified activation function layer.

[0051] For an input frame of visible light image, three dilated convolution branches extract features independently. Dilated convolution expands the receptive field by inserting zeros between kernel elements. With a dilation rate of 2, the kernel effectively covers a 5x5 area; with 4, the coverage is 9x9; and with 6, it's 13x13. The feature maps output by each branch have the same spatial size and number of channels; in this embodiment, the number of channels is set to 256.

[0052] A pixel-wise weighted fusion is performed on the feature maps output by each dilated convolution branch. Specifically, for each spatial location on the feature map, feature vectors from the three branches at that location are obtained, each feature vector having a dimension of 256. The weighted sum of the three feature vectors at that location is calculated, with the weight coefficients being learnable parameters, and the sum of the three weight coefficients being 1. These learnable parameters are automatically optimized during model training using the backpropagation algorithm, with initial values ​​set to one-third. The above weighted fusion operation is performed on all spatial locations on the feature map to obtain a fused feature map.

[0053] The fused feature map is then spatially masked and multiplied with the set of curvature salient regions. This step converts the set of curvature salient regions into a binary mask matrix with the same spatial dimensions as the fused feature map. In this binary mask matrix, pixels located inside the set of curvature salient regions have a value of 1, while pixels located outside the set have a value of 0. The fused feature map is then element-wise multiplied with the binary mask matrix, that is, each channel feature value at each position in the fused feature map is multiplied by the corresponding value in the mask matrix. This operation preserves the features inside the set of curvature salient regions while setting the feature values ​​of the outer regions to zero, thus injecting spatial prior knowledge into the feature extraction process. The resulting multi-scale structural damage explicit feature map, denoted as Fdam, is obtained.

[0054] In this embodiment, each frame of the visible light image sequence is input to the second branch. The second branch first extracts an initial feature map through a standard convolutional layer. The standard convolutional layer uses a 3x3 kernel size, a stride of 1, and padding of 1, with an output channel count of 256. This is followed by a batch normalization layer and a linear rectified activation function layer to obtain the initial feature map. A deformable offset field is then superimposed on the initial feature map. The deformable offset field is generated by the curvature saliency region set. Specifically, the curvature saliency region set is first converted into a guide map with the same spatial dimensions as the initial feature map. Pixels inside the curvature saliency region set in the guide map have a value of 1, while pixels outside the curvature saliency region set have a value of 0. The guiding map is input into an offset generation network, which contains two convolutional layers. The first layer is a 3x3 convolutional layer with 256 output channels, and the second layer is a 3x3 convolutional layer with 18 output channels, where 18 corresponds to the 2 offset coordinates multiplied by 9 sampling points required for each sampling point in the deformable convolution. The output of the offset generation network is an offset field, which represents the spatial offset that the deformable convolution sampling point corresponding to each position on the initial feature map should produce.

[0055] The offset field and the initial feature map are input together into a deformable convolutional layer. The kernel size of the deformable convolutional layer is 3x3, and its sampling positions are superimposed with the offset values ​​from the corresponding positions in the offset field on top of the standard grid positions, causing the convolutional sampling positions to shift along the geometric direction of the significantly curved regions. Specifically, for each position on the output feature map, the deformable convolutional layer determines the sampling point position on the input feature map based on the offset field. These sampling points are no longer limited to a regular 3x3 grid but instead cluster in the direction of the significantly curved regions. Through the above operations, a surface continuity perturbation feature map, denoted as Fdis, is extracted. This feature map has a stronger response capability in geometrically deformed regions.

[0056] After obtaining the surface continuity perturbation feature map Fdis and the multi-scale structural damage explicit feature map Fdam, this embodiment concatenates them along the channel dimension. Since Fdis and Fdam have the same spatial size and both have 256 channels, the concatenated feature map has 512 channels. The concatenated feature map is then input into the channel attention module. The channel attention module performs global average pooling and global max pooling on the features of each channel. The global average pooling operation calculates the arithmetic mean of all spatial location feature values ​​in each channel, outputting a vector of dimension 512, where each element corresponds to the average value of one channel. The global max pooling operation calculates the maximum value of all spatial location feature values ​​in each channel, outputting a vector of dimension 512, where each element corresponds to the maximum value of one channel.

[0057] The vectors output by global average pooling and global max pooling are input into a shared multilayer perceptron. The shared multilayer perceptron contains two fully connected layers: the first fully connected layer has an input dimension of 512 and an output dimension of 32, and the second fully connected layer has an input dimension of 32 and an output dimension of 512. After passing through the shared multilayer perceptron, two intermediate vectors of dimension 512 are obtained. These two intermediate vectors are added element-wise, and then passed through a sigmoid activation function to generate a channel weight vector of dimension 512, where each channel weight value is between 0 and 1.

[0058] The channel weight vector is multiplied channel-by-channel with the stitched feature map. Specifically, for the c-th channel in the stitched feature map, the feature values ​​of all spatial locations in that channel are multiplied by the value of the c-th element in the channel weight vector. This operation enhances the features of important channels and suppresses the features of unimportant channels. After channel-by-channel multiplication, the aligned multi-scale structural damage explicit features and surface continuity perturbation features are output, denoted as Fdamaligned and Fdisaligned. Both have the same spatial size as the input feature map and 256 channels.

[0059] In this embodiment, the set of curvature salient regions is converted into a binary spatial attention mask. The set of curvature salient regions contains multiple projected regions on a 3D surface mesh model with temperature properties. This step first maps these projected regions to the pixel coordinate system of the visible light image, generating a binary mask matrix with the same spatial size as the visible light image. In this binary mask matrix, pixels located inside the set of curvature salient regions have a value of 1, and pixels located outside the set of curvature salient regions have a value of 0. Gaussian blur expansion is performed on the binary spatial attention mask. Specifically, a Gaussian kernel is used to perform a convolution operation on the binary mask, with the kernel size set to 15x15 and the standard deviation σ set to 3. After the convolution operation, a gradual transition region is formed at the edge of the mask. Regions that were originally 1 diffuse outwards, with the diffusion degree decreasing as the distance increases. Regions that were originally 0 obtain a transition value between 0 and 1 at their adjacent 1 value regions. After Gaussian blur expansion, a continuous spatial attention weight map is obtained, denoted as Watt, with pixel values ​​ranging from 0 to 1. The regions inside and near the set of regions with significant curvature have higher weight values, while the weight values ​​of the outer regions that are far away tend to be 0.

[0060] In this embodiment, the continuous spatial attention weight map Watt is multiplied channel-by-channel and element-by-element by the multi-scale structural damage explicit feature map Fdam. The spatial size of the multi-scale structural damage explicit feature map Fdam is H times W, and the number of channels is C. In this embodiment, H and W are the height and width of the visible light image, and C is 256. The spatial size of the continuous spatial attention weight map Watt is H times W, and the number of channels is 1.

[0061] The specific operation of channel-by-channel, element-by-element multiplication is as follows: For each channel c in Fdam, the eigenvalues ​​of all spatial locations in that channel are multiplied by the weight values ​​of the corresponding spatial locations in Watt. That is, the eigenvalue of position (i,j) in the c-th channel of Fdam is multiplied by the weight value of position (i,j) in Watt. Since Watt has only a single channel, this weight value is broadcast to all C channels. This operation enhances the eigenvalues ​​located at positions corresponding to the set of regions with significant curvature in the explicit feature map of multi-scale structural damage, while suppressing the eigenvalues ​​located at positions corresponding to positions outside the set of regions with significant curvature. After multiplication, a preliminary enhanced explicit feature map of structural damage is obtained.

[0062] In this embodiment, the continuous spatial attention weight map Watt is multiplied channel-by-channel and element-by-element by the surface continuity perturbation feature map Fdis. The spatial size of the surface continuity perturbation feature map Fdis is H x W, and the number of channels is C. In this embodiment, H and W are the height and width of the visible light image, and C is 256. The spatial size of the continuous spatial attention weight map Watt is H x W, and the number of channels is 1.

[0063] For each channel c in Fdis, the feature values ​​of all spatial locations in that channel are multiplied by the weight values ​​of the corresponding spatial locations in Watt. This operation enhances the feature values ​​corresponding to locations in the set of regions with significant curvature in the surface continuity perturbation feature map, while suppressing the feature values ​​corresponding to locations outside the set of regions with significant curvature. The multiplication yields a preliminary enhanced surface continuity perturbation feature map. In this embodiment, the preliminary enhanced structural damage explicit feature map and the preliminary enhanced surface continuity perturbation feature map are input to their respective spatial pyramid pooling modules. The spatial pyramid pooling module contains four pooling windows of different scales, with pooling window sizes set to 1x1, 2x2, 3x3, and 6x6, respectively. Each pooling window performs an adaptive average pooling operation, downsampling the spatial size of the input feature map to match the pooling window size.

[0064] For the initial enhanced explicit feature map of structural damage, adaptive average pooling is performed through four pooling windows to obtain four pooled output feature maps of different sizes. Each pooling output feature map is upsampled to its original size H x W using bilinear interpolation. The four upsampled feature maps are then concatenated along the channel dimension. The number of channels in the concatenated feature map is 4 times the number of channels in the original map; in this embodiment, the original number of channels is 256, and the concatenated number of channels is 1024. The concatenated feature map is then input into a convolutional layer for channel compression. This convolutional layer uses a 1x1 kernel with 256 output channels, followed by a batch normalization layer and a linear rectified activation function layer, outputting the enhanced explicit feature of structural damage, denoted as Fdam'.

[0065] The initial enhanced surface continuity perturbation feature map is processed using the same identical operations to obtain the enhanced surface continuity perturbation feature, denoted as Fdis'. The spatial pyramid pooling module captures contextual information under different receptive fields through multi-scale pooling operations, so that the final output feature simultaneously contains local details and global structural information, enhancing the feature's ability to represent crack defects.

[0066] S400 performs intermodal iterative collaborative optimization on Fdam' and Fdis'. In each iteration, Fdis' is restructured and recalibrated using the current Fdam' as a guide, and detailed backprojection is performed on Fdam' using the current Fdis' as a guide, until the iteration converges to obtain the deep fusion feature Ffuse.

[0067] In this embodiment, edge gradient extraction is first performed on the enhanced surface continuity perturbation feature. The enhanced surface continuity perturbation feature is denoted as Fdis', with a spatial size of H x W and a channel number of C, which is 256 in this embodiment. Edge gradient extraction uses the Sobel operator for convolution operations. The Sobel operator contains two convolution kernels: a horizontal convolution kernel and a vertical convolution kernel. The horizontal convolution kernel is a 3x3 matrix, with values ​​of -1, -2, -1 in the first row, 0, 0, 0 in the second row, and 1, 2, 1 in the third row. The vertical convolution kernel is also a 3x3 matrix, with values ​​of -1, 0, 1 in the first row, -2, 0, 2 in the second row, and -1, 0, 1 in the third row.

[0068] For each channel in the enhanced surface continuity perturbation feature, the feature map of that channel is convolved with both horizontal and vertical convolution kernels to obtain the horizontal and vertical gradient maps. For each spatial location on the feature map, the square root of the sum of the squares of the horizontal and vertical gradient values ​​is calculated to obtain the gradient magnitude at that location. This calculation is performed on all spatial locations to obtain the gradient map for that channel. Performing this operation on all channels yields the surface continuity perturbation gradient map, denoted as Gdis, with a spatial size of H x W and the number of channels C.

[0069] Edge gradient extraction is performed on the enhanced structural damage explicit features using the exact same operation. The enhanced structural damage explicit features are denoted as Fdam', with a spatial size of H x W and C channels. The structural damage explicit gradient map, denoted as Gdam, is obtained through Sobel convolution, with a spatial size of H x W and C channels.

[0070] After obtaining the surface continuity perturbation gradient map Gdis and the structural damage explicit gradient map Gdam, this embodiment calculates the gradient difference map between the two. For each channel, Gdis and Gdam are subtracted element-wise; that is, for each spatial location, the difference between the surface continuity perturbation gradient value and the structural damage explicit gradient value is calculated, and the absolute value of the difference is taken as the gradient difference value at that location. The above calculation is performed on all spatial locations of all channels to obtain the gradient difference map, denoted as Ddiff, with a spatial size of H times W and the number of channels C.

[0071] A detail compensation mask is generated based on the gradient difference map. Specifically, for each spatial location in the gradient difference map Ddiff, the arithmetic mean of the gradient difference values ​​at that location across all C channels is calculated to obtain the average gradient difference value at that location. This calculation is performed on all spatial locations to obtain the average gradient difference map, denoted as Davg, with a spatial size of H x W and 1 channel. A preset difference threshold is set; in this embodiment, the preset difference threshold is 0.15. Each spatial location in the average gradient difference map is traversed, and the average gradient difference value is compared with the preset difference threshold. If the average gradient difference value at that location is greater than the preset difference threshold, the value at the corresponding location in the detail compensation mask is set to a first preset value; in this embodiment, the first preset value is 1. If the average gradient difference value at that location is less than or equal to the preset difference threshold, the value at the corresponding location in the detail compensation mask is set to a second preset value; in this embodiment, the second preset value is 0. The detail compensation mask, denoted as Mcomp, is obtained through the above operations, with a spatial size of H x W and 1 channel. The regions with a value of 1 in this detail compensation mask represent locations where the gradient difference between the surface continuity perturbation features and the explicit structural damage features is large, i.e., detail regions that are included in the surface continuity perturbation features but missing from the explicit structural damage features.

[0072] In this embodiment, the detail compensation mask Mcomp is multiplied channel-by-channel and element-by-element by the enhanced surface continuity perturbation feature Fdis'. Specifically, for each channel c in Fdis', the feature values ​​of all spatial locations in that channel are multiplied by the corresponding spatial values ​​in Mcomp. Since Mcomp has only one channel, this value is broadcast to all C channels. This operation extracts the feature values ​​located in regions with large gradient differences from the enhanced surface continuity perturbation feature, resulting in the detail guidance feature, denoted as Fdetail, with a spatial size of H times W and C channels.

[0073] The detailed guiding feature Fdetail is element-wise added to the enhanced structural damage explicit feature Fdam'. For each channel c, the feature values ​​of all spatial positions in channel c of Fdetail are added to the feature values ​​of the corresponding spatial positions in channel c of Fdam' to obtain the updated feature value. This operation is performed on all spatial positions of all channels to obtain the updated enhanced structural damage explicit feature, denoted as Fdamupdated, with a spatial size of H multiplied by W and the number of channels C. This operation injects the detailed information unique to the surface continuity perturbation feature into the structural damage explicit feature, so that the updated feature simultaneously contains macroscopic information of structural damage and microscopic details of surface continuity. Each iteration uses the currently updated enhanced structural damage explicit feature and the currently enhanced surface continuity perturbation feature as input. Specifically, the first iteration uses the initial Fdam' and Fdis' as input and outputs the first updated Fdamupdated. The second iteration uses the first updated Fdamupdated as the new enhanced structural damage explicit feature, inputting it together with Fdis', and outputs the second updated Fdamupdated.

[0074] In this embodiment, the preset number of iterations is 3. This number of iterations was determined experimentally to ensure sufficient feature fusion while avoiding over-smoothing. In each iteration, the detail compensation mask is recalculated based on the gradient difference between the current explicit features of structural damage and the surface continuity perturbation features, so that the detailed information supplemented in each iteration is gradually refined.

[0075] The enhanced structural damage explicit features obtained from the last iteration are output as deep fusion features, denoted as Ffuse. This deep fusion feature combines the complementary information of the structural damage explicit features and the surface continuity perturbation features. It retains the macroscopic structural damage response guided by the set of curvature-significant regions, while supplementing the microscopic details in the surface continuity perturbation features, providing a more complete feature representation for the subsequent generation of crack probability maps.

[0076] S500 adaptively selects upsampling paths of different scales based on the local energy distribution of Ffuse to generate a crack probability map P.

[0077] In this embodiment, the deep fusion feature Ffuse is input to a heterogeneous feature decoder with a dynamic routing mechanism. The spatial size of the deep fusion feature Ffuse is H times W, and the number of channels is C. In this embodiment, H and W are the height and width of the visible light image, and C is 256.

[0078] First, the deep fusion features are processed in parallel through multiple branches. These branches include a first upsampling branch, a second upsampling branch, and a third upsampling branch. Each branch employs an upsampling operation at a different magnification and follows dilated convolutional layers with different dilation rates.

[0079] The first upsampling branch performs a 2x upsampling operation on Ffuse, using bilinear interpolation to expand the spatial size from H x W to 2H x 2W. Following upsampling is a dilated convolutional layer with a 3x3 kernel size, a dilation rate of 1 (standard convolution), and 128 output channels.

[0080] The second upsampling branch performs a 4x upsampling operation on Ffuse, using bilinear interpolation to expand the spatial size from H x W to 4H x 4W. This is followed by a dilated convolutional layer with a 3x3 kernel size, a dilation rate of 2, and 128 output channels.

[0081] The third upsampling branch performs an 8x upsampling operation on Ffuse, using bilinear interpolation to expand the spatial size from H x W to 8H x 8W. This is followed by a dilated convolutional layer with a 3x3 kernel size, a dilation rate of 4, and 128 output channels.

[0082] Each branch's dilated convolutional layer is followed by a batch normalization layer and a linear rectified activation function layer. After the above processing, the first upsampling branch outputs the first decoded feature map, denoted as D1, with a spatial size of 2H x 2W and 128 channels; the second upsampling branch outputs the second decoded feature map, denoted as D2, with a spatial size of 4H x 4W and 128 channels; and the third upsampling branch outputs the third decoded feature map, denoted as D3, with a spatial size of 8H x 8W and 128 channels. The three decoded feature maps form a pyramid structure in spatial scale, corresponding to feature representations of different granularities.

[0083] This embodiment calculates the local energy distribution map of the deep fusion feature Ffuse. The local energy distribution map is obtained by calculating the sum of squares of feature values ​​within a sliding window. Specifically, for each channel in Ffuse, the sliding window size is set to 7x7 with a step size of 1. For each spatial location on the feature map, a 7x7 neighborhood window is taken centered at that location, and the sum of squares of all feature values ​​within the window is calculated. This sum of squares is used as the local energy value at that location in that channel. The above calculation is performed on all channels to obtain the local energy map for each channel.

[0084] The local energy maps of each channel are summed along the channel dimension. That is, for each spatial location, the local energy values ​​of all channels at that location are added together to obtain the total local energy value at that location. This calculation is performed on all spatial locations to obtain a local energy distribution map, denoted as Elocal, with a spatial size of H times W and one channel.

[0085] A path selection weight map is generated based on the local energy distribution map. Since the spatial dimensions of the three decoded feature maps are 2H x 2W, 4H x 4W, and 8H x 8W respectively, which differ from the spatial dimension H x W of Ffuse, this embodiment first upsamples the local energy distribution map Elocal to the same spatial dimension as each decoded feature map. Specifically, using bilinear interpolation, Elocal is upsampled to 2H x 2W to obtain Elocal2; upsampled to 4H x 4W to obtain Elocal4; and upsampled to 8H x 8W to obtain Elocal8.

[0086] For the upsampled local energy distribution map, three energy thresholds are set: low energy threshold, medium energy threshold, and high energy threshold. In this embodiment, the low energy threshold is set to 0.3, the medium energy threshold is set to 0.6, and the high energy threshold is set to 0.9. The value of each energy threshold is between 0 and 1, and is obtained by normalizing the local energy distribution map.

[0087] The path selection weight graph contains three weight channels, corresponding to the first upsampling branch, the second upsampling branch, and the third upsampling branch, respectively. For each spatial location, the weight allocation method is determined based on the value of that location on the local energy distribution map: if the local energy value is less than or equal to the low energy threshold, the weight of the first branch at that location is 1, the weight of the second branch is 0, and the weight of the third branch is 0; if the local energy value is greater than the low energy threshold and less than or equal to the medium energy threshold, the weight of the first branch at that location is 0, the weight of the second branch is 1, and the weight of the third branch is 0; if the local energy value is greater than the medium energy threshold and less than or equal to the high energy threshold, the weight of the first branch at that location is 0, the weight of the second branch is 0, and the weight of the third branch is 1; if the local energy value is greater than the high energy threshold, the weight of the first branch at that location is one-third, the weight of the second branch is one-third, and the weight of the third branch is one-third.

[0088] The above weight assignment is performed for each spatial location to obtain a path selection weight map, denoted as Wroute. The spatial dimensions of this weight map correspond to the corresponding decoding features. Figure 1 The formula contains three channels, each channel corresponding to a branch weight value, and the sum of the three weight values ​​at each spatial location is 1.

[0089] This embodiment performs a weighted fusion of the three decoded feature maps with different scales and the path selection weight map. Since the three decoded feature maps have different spatial dimensions, this step requires processing the weight map corresponding to each size separately.

[0090] For the first upsampling branch, the spatial size of the decoded feature map D1 is 2H x 2W. The weight channel corresponding to the first branch in the path selection weight map is denoted as Wroute1, and its spatial size is also 2H x 2W. D1 and Wroute1 are multiplied channel-by-channel and element-by-element. That is, for each channel in D1, the feature values ​​of all spatial locations in that channel are multiplied by the weight values ​​of the corresponding spatial locations in Wroute1 to obtain the weighted first decoded feature map, denoted as D1weighted.

[0091] For the second upsampling branch, the spatial size of the decoded feature map D2 is 4H x 4W. The weight channel corresponding to the second branch in the path selection weight map is denoted as Wroute2, with a spatial size of 4H x 4W. Multiplying D2 and Wroute2 channel by channel and element by element yields the weighted second decoded feature map, denoted as D2weighted.

[0092] For the third upsampling branch, the spatial size of the decoded feature map D3 is 8H x 8W. The weight channel corresponding to the third branch in the path selection weight map is denoted as Wroute3, with a spatial size of 8H x 8W. Multiplying D3 and Wroute3 channel by channel and element by element yields the weighted third decoded feature map, denoted as D3weighted.

[0093] The three weighted decoded feature maps are spatially aligned and then summed. Since the three feature maps have different spatial dimensions, D1weighted is first upsampled to 8H x 8W using bilinear interpolation, D2weighted is upsampled to 8H x 8W using bilinear interpolation, and D3weighted remains at its original size. Then, the three upsampled and aligned feature maps are summed element-wise to obtain the fused decoded feature map, denoted as Dfuse, with a spatial dimension of 8H x 8W and 128 channels.

[0094] In this fused decoding feature map, the feature value of each spatial location comes from a weighted combination of different branches. Regions with higher local energy tend to select branches with larger receptive fields to capture the global structure of large-scale cracks; regions with lower local energy tend to select branches with smaller receptive fields to preserve the fine texture of small-scale cracks.

[0095] In this embodiment, the fused decoding feature map Dfuse is passed sequentially through a convolutional layer and a Sigmoid activation function layer to output a crack probability map.

[0096] First, Dfuse is input into a convolutional layer. This convolutional layer uses a 3x3 kernel with a stride of 1 and padding of 1, with 128 input channels and 1 output channel. This convolutional layer compresses the 128-channel feature map into a single-channel feature map, where the feature value at each spatial location represents the original response intensity of that location belonging to the crack category.

[0097] The single-channel feature map output from the convolutional layer is input to the Sigmoid activation function layer. The Sigmoid activation function is calculated as follows: for an input value x, the output value is 1 divided by the sum of 1 plus e raised to the power of -x. This function maps any real number to a probability value between 0 and 1. After processing by the Sigmoid activation function layer, a crack probability map, denoted as P, is obtained, with a spatial size of 8H x 8W and one channel. The value of each pixel in the crack probability map is between 0 and 1, representing the probability that the location belongs to the crack category. The closer the probability value is to 1, the higher the probability that a crack exists at that location; the closer the probability value is to 0, the higher the probability that the location is background. The crack probability map P is used as the final output for subsequent steps to combine with preset crack geometric constraints to determine the location, scale, and confidence level of crack defects in the target road section, and to generate crack detection results.

[0098] S600: Based on the crack probability map P and the preset crack geometric constraints, determine the location, scale, and confidence level of crack defects in the target road section, and generate crack detection results.

[0099] In this embodiment, adaptive threshold segmentation is first performed on the crack probability map P. The crack probability map P has a spatial size of 8H by 8W and one channel. Each pixel has a probability value between 0 and 1, representing the probability that the location belongs to the crack category.

[0100] Adaptive threshold segmentation employs Otsu's method to calculate the optimal segmentation threshold. Specifically, the probability distribution of all pixels in the crack probability map P is statistically analyzed, and a histogram is constructed. The horizontal axis of the histogram represents the probability value interval; in this embodiment, 0 to 1 is divided into 256 equally spaced intervals. The vertical axis represents the number of pixels falling within each interval. All possible probability values ​​are traversed as candidate thresholds. For each candidate threshold, pixels are divided into two categories: the first category consists of pixels with a probability value greater than or equal to the candidate threshold, and the second category consists of pixels with a probability value less than the candidate threshold. The inter-class variance of the probability values ​​of the two categories of pixels is calculated, and the candidate threshold corresponding to the largest inter-class variance is taken as the segmentation threshold, denoted as τseg. Pixels in the crack probability map P with a probability value greater than the segmentation threshold τseg are marked as 1, and pixels with a probability value less than or equal to the segmentation threshold τseg are marked as 0, resulting in a binary segmentation map, denoted as Bseg, with a spatial size of 8H x 8W and one channel.

[0101] Connectivity analysis is performed on the candidate crack pixels marked as 1 in the binary segmentation image Bseg. An eight-neighbor connectivity rule is used, meaning that two pixels are considered connected if they are adjacent in the horizontal, vertical, or diagonal direction. Specifically, the extraction method is as follows: traverse every pixel in the binary segmentation image. When a pixel with a value of 1 that has not been visited is encountered, use that pixel as the starting point and employ a breadth-first search algorithm to traverse all connected pixels with a value of 1 along the eight-neighbor direction. These pixels are grouped into the same connected region and marked as visited. This process is repeated until all pixels have been visited. Several initial crack connected regions are extracted through this operation, denoted as Rinit={r1,r2,…,rk}, where k is the number of initial crack connected regions. Each initial crack connected region contains a set of continuous spatial coordinates.

[0102] This embodiment extracts the skeleton of each initial crack connected region to obtain the corresponding crack centerline. The skeleton extraction employs a morphological thinning algorithm, with the following steps: For each initial crack connected region, it is represented as a binary image, where pixels within the region have a value of 1, and pixels outside the region have a value of 0. Iterative thinning is performed using a sequence of structuring elements, consisting of eight 3x3 templates in different directions, each corresponding to a pixel deletion condition. In each iteration, all boundary pixels within the region are traversed. If a pixel satisfies the deletion condition defined by the current structuring element, it is marked as to be deleted. After one round of traversal, the values ​​of all pixels to be deleted are set to 0. The above process is repeated using the eight structuring elements until no pixels are deleted in two consecutive iterations. The final retained pixels constitute the crack centerline, denoted as Cline.

[0103] For each point on the crack centerline Cline, calculate the local width at that point. Specifically, for a point p on the centerline, draw the normal direction of the centerline through point p, and extend it to both sides until it encounters the boundary of the initial crack connectivity region. The distance between the boundary points on both sides of the normal direction is taken as the local width value at that point. Perform the above calculation on all points on the centerline to obtain the crack width distribution curve, denoted as Wcurve. This curve records the mapping relationship between the position of each point on the centerline and its corresponding local width.

[0104] This embodiment filters the connected regions of each initial crack based on preset crack geometric constraints. The preset crack geometric constraints include a minimum length threshold, a maximum width threshold, and a range of length-to-width ratios.

[0105] First, the crack length of each initial crack connected region is calculated. The crack length is defined as the number of pixels on the crack centerline Cline multiplied by the actual physical size of each pixel. In this embodiment, since the spatial size of the crack probability map P is 8H x 8W, the actual physical size of each pixel is obtained through calibration by the acquisition device. In this embodiment, the actual physical size of each pixel is 0.5 mm. Multiplying the number of pixels on the centerline by 0.5 mm gives the crack length, denoted as Lcrack. Next, the maximum width of each initial crack connected region is calculated. All local width values ​​in the crack width distribution curve Wcurve are traversed, and the maximum value is taken as the maximum width of the crack, denoted as Wmax. The ratio of the crack length to the maximum width is calculated, denoted as Rlw = Lcrack / Wmax.

[0106] In this embodiment, the minimum length threshold is set to 10 mm, the maximum width threshold is set to 20 mm, and the length-to-width ratio is set to a range of 3 to 100. For each initial crack connected region, the following is determined: if the crack length is greater than or equal to 10 mm, the maximum width is less than or equal to 20 mm, and the length-to-width ratio is between 3 and 100, then the initial crack connected region is determined to satisfy all geometric constraints.

[0107] Initial crack connected regions that satisfy all geometric constraints are defined as valid crack regions, denoted as Rvalid={r1,r2,…,rm}, where m is the number of valid crack regions. Initial crack connected regions that do not satisfy any geometric constraints are considered false detection regions or non-crack interference regions and are discarded. In this embodiment, the confidence level of a valid crack region is calculated based on the probability value of the corresponding position of the valid crack region on the crack probability map P. For each valid crack region, the probability values ​​of all pixels within that region on the crack probability map P are obtained, and the arithmetic mean of these probability values ​​is calculated. This average value is used as the confidence level of the valid crack region, denoted as Conf, with a value ranging from 0 to 1. The closer the confidence level is to 1, the higher the confidence that the crack region belongs to a real crack.

[0108] Output the location coordinates, crack length, crack width distribution, and confidence score for each valid crack region to generate crack detection results. The location coordinates are represented by the minimum bounding rectangle of the valid crack region, specifically the x and y coordinates of the top-left corner vertex of the rectangle, as well as the width and height of the rectangle. All coordinate values ​​are in actual physical dimensions, in millimeters. The crack length is the calculated Lcrack value, in millimeters. The crack width distribution is the calculated Wcurve, recording the mapping relationship between the position of each point on the center line and its corresponding local width, in millimeters. The confidence score is the calculated Conf value, dimensionless.

[0109] The above information is organized according to a preset format to generate crack detection results, which are then used for data entry and visualization in the subsequent road maintenance management system.

[0110] Example 2, please refer to Figure 2 As shown in this embodiment, the road crack defect detection system based on multi-scale feature fusion includes: Data acquisition module: acquires multi-source data of the target road section, including visible light image sequences, infrared thermal imaging sequences, and three-dimensional laser point cloud data; The geometry-temperature fusion modeling module: jointly registers the infrared thermal imaging sequence with the three-dimensional laser point cloud data to construct a three-dimensional surface mesh model M with temperature attributes, and performs curvature field analysis on M to extract the set of curvature-significant regions R that reflect local geometric abrupt changes; Dual-branch heterogeneous feature extraction module: The visible light image sequence is input into the dual-branch heterogeneous encoder, wherein the first branch uses dilated convolutional pyramid to extract multi-scale structural damage explicit feature map Fdam, and the second branch uses deformable convolution to extract surface continuity perturbation feature map Fdis aligned with geometric deformation. R is used as a spatial attention prior to weight and enhance Fdam and Fdis respectively, to obtain enhanced structural damage explicit feature Fdam' and enhanced surface continuity perturbation feature Fdis'. Intermodal iterative collaborative optimization module: Perform intermodal iterative collaborative optimization on Fdam' and Fdis'. In each iteration, Fdis' is restructured and recalibrated based on the current Fdam', and Fdam' is back-projected in detail based on the current Fdis', until the iteration converges and the deep fusion feature Ffuse is obtained. Dynamic routing decoding module: Adaptively selects upsampling paths of different scales based on the local energy distribution of Ffuse to generate a crack probability map P; Crack detection output module: Based on the crack probability map P and the preset crack geometric constraints, determine the location, scale and confidence level of crack defects in the target road section, and generate crack detection results.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A road crack defect detection method based on multi-scale feature fusion, characterized in that: include: S100, acquire multi-source acquisition data of the target road section, the multi-source acquisition data including visible light image sequence, infrared thermal imaging sequence and three-dimensional laser point cloud data; S200, the infrared thermal imaging sequence is jointly registered with the three-dimensional laser point cloud data to construct a three-dimensional surface mesh model M with temperature attributes, and curvature field analysis is performed on M to extract the set of curvature significant regions R that reflect local geometric abrupt changes; S300, the visible light image sequence is input into a dual-branch heterogeneous encoder, wherein the first branch uses dilated convolutional pyramids to extract multi-scale structural damage explicit feature map Fdam, and the second branch uses deformable convolution to extract surface continuity perturbation feature map Fdis aligned with geometric deformation. R is used as a spatial attention prior to weight and enhance Fdam and Fdis respectively, to obtain enhanced structural damage explicit feature Fdam' and enhanced surface continuity perturbation feature Fdis'. S400 performs intermodal iterative collaborative optimization on Fdam' and Fdis'. In each iteration, Fdis' is restructured and recalibrated based on the current Fdam', and Fdam' is back-projected in detail based on the current Fdis', until the iteration converges and the deep fusion feature Ffuse is obtained. S500 adaptively selects upsampling paths of different scales based on the local energy distribution of Ffuse to generate a crack probability map P; S600: Based on the crack probability map P and the preset crack geometric constraints, determine the location, scale, and confidence level of crack defects in the target road section, and generate crack detection results.

2. The road crack defect detection method based on multi-scale feature fusion according to claim 1, characterized in that: In step S200, a three-dimensional surface mesh model M with temperature properties is constructed, specifically including: S210, Based on the three-dimensional spatial coordinate information contained in the three-dimensional laser point cloud data, an initial three-dimensional surface mesh is constructed using the Poisson surface reconstruction algorithm, and the initial three-dimensional surface mesh is subjected to Laplacian smoothing to obtain a reference geometric mesh model. S220, extract the temperature value matrix corresponding to each frame of the infrared thermal imaging sequence, and project the temperature value matrix onto each vertex of the reference geometric mesh model through the inverse distance weight interpolation algorithm, so that each vertex obtains the corresponding temperature attribute value, forming a temperature-attached mesh model. S230, The temperature-attached mesh model is input into a pre-trained adaptive temperature-geometry joint optimization network. The network uses the structural consistency between the vertex temperature gradient and the local curvature change as a constraint to iteratively optimize the vertex position and temperature value until the temperature field and the geometric field reach the edge alignment state, and outputs a three-dimensional surface mesh model M.

3. The road crack defect detection method based on multi-scale feature fusion according to claim 2, characterized in that: In step S200, curvature field analysis is performed on M to extract a set of curvature-significant regions R that reflect local geometric abrupt changes, specifically including: S240, calculate the Gaussian curvature and mean curvature of each vertex in the three-dimensional surface mesh model M with temperature properties at multiple neighborhood scales, and use the standard deviation of the curvature values ​​at each scale as a measure of the multi-scale curvature change of the vertex to obtain the curvature change field Cvar. S250, the curvature change field Cvar and the temperature attribute value corresponding to each vertex in the three-dimensional surface mesh model M with temperature attribute are jointly normalized to construct a curvature-temperature joint saliency map Smap; S260, perform morphological connected component extraction on the curvature-temperature joint saliency map Smap, filter out isolated noise points with an area smaller than the preset minimum crack unit, and determine the projection region of the remaining connected components on the three-dimensional surface mesh model M as the curvature saliency region set R.

4. The road crack defect detection method based on multi-scale feature fusion according to claim 1, characterized in that: In step S300, the first branch uses dilated convolutional pyramids to extract a multi-scale structural damage explicit feature map Fdam, specifically including: Each frame of the visible light image sequence is input into the first branch, which contains at least three dilated convolution branches set in parallel. The dilation rate of each dilated convolution branch is in an increasing sequence. The feature map output by each dilated convolution branch is subjected to pixel-wise weighted fusion. The fused feature map is then multiplied with the set of curvature significant regions by a spatial mask to obtain the multi-scale structural damage explicit feature map Fdam.

5. The road crack defect detection method based on multi-scale feature fusion according to claim 1, characterized in that: In step S300, the second branch uses deformable convolution to extract a surface continuity perturbation feature map Fdis aligned with geometric deformation. Specifically, it includes: inputting each frame of visible light image in the visible light image sequence into the second branch; the second branch first extracts an initial feature map through a standard convolutional layer; and then superimposes a deformable offset field on the initial feature map. The deformable offset field is generated by the set of curvature saliency regions, causing the convolution sampling position to shift along the geometric direction of the curvature saliency region, thereby extracting the surface continuity perturbation feature map Fdis.

6. The road crack defect detection method based on multi-scale feature fusion according to claim 5, characterized in that: The surface continuity perturbation feature map Fdis and the multi-scale structural damage explicit feature map Fdam are concatenated along the channel dimension. The concatenated feature map is then input into the channel attention module. The channel attention module performs global average pooling and global max pooling on each channel feature and generates channel weights through a shared multilayer perceptron. The channel weights are then multiplied with the concatenated feature map channel by channel to output the aligned multi-scale structural damage explicit features and surface continuity perturbation features.

7. The road crack defect detection method based on multi-scale feature fusion according to claim 1, characterized in that: Step S400 specifically includes: S410, perform edge gradient extraction on the enhanced surface continuity perturbation features to obtain the surface continuity perturbation gradient map, and perform edge gradient extraction of the same size on the enhanced structural damage explicit features to obtain the structural damage explicit gradient map. S420, calculate the gradient difference map between the surface continuity perturbation gradient map and the structural damage explicit gradient map, and generate a detail compensation mask based on the gradient difference map. In the detail compensation mask, the region where the gradient difference exceeds a preset difference threshold is assigned a first preset value, and the remaining region is assigned a second preset value. S430, the detail compensation mask is multiplied channel-by-channel and element-by-element by the enhanced surface continuity perturbation feature to obtain the detail guiding feature, and the detail guiding feature is added element-by-element by the enhanced structure damage explicit feature to obtain the updated enhanced structure damage explicit feature. S440, repeat steps S410 to S430 for a total of a preset number of iterations. Each iteration takes the currently updated explicit features of enhanced structural damage and the currently updated features of enhanced surface continuity perturbation as inputs, and outputs the explicit features of enhanced structural damage obtained from the last iteration as deep fusion features.

8. The road crack defect detection method based on multi-scale feature fusion according to claim 1, characterized in that: Step S500 specifically includes: S510, perform multi-branch parallel processing on the deep fusion features. The multi-branch includes a first upsampling branch, a second upsampling branch and a third upsampling branch. Each branch adopts upsampling operations with different magnification and follows dilated convolutional layers with different dilation rates to output three decoded feature maps of different scales. S520, calculate the local energy distribution map of the deep fusion feature. The local energy distribution map is obtained by calculating the sum of squares of feature values ​​within a sliding window, and a path selection weight map is generated based on the local energy distribution map. S530, the three decoded feature maps of different scales are weighted and fused with the path selection weight map. For each spatial location, the feature vectors of the location on the three decoded feature maps are multiplied by the corresponding weight values ​​and then added together to obtain the fused decoded feature map. S540, the fused decoded feature map is passed sequentially through a convolutional layer and a Sigmoid activation function layer to output a crack probability map. The value of each pixel in the crack probability map represents the probability that the location belongs to the crack category.

9. The road crack defect detection method based on multi-scale feature fusion according to claim 1, characterized in that: Step S600 specifically includes: S610, perform adaptive threshold segmentation on the crack probability map, mark the pixels with probability values ​​greater than the segmentation threshold as crack candidate pixels, and perform connected component analysis on the crack candidate pixels to extract several initial crack connected regions. S620, extract the skeleton of each initial crack connected region to obtain the corresponding crack centerline, and calculate the local width at each point on the crack centerline to obtain the crack width distribution curve. S630, each initial crack connected region is screened according to preset crack geometric constraints. The crack geometric constraints include a minimum length threshold, a maximum width threshold, and a range of length to width ratios. The initial crack connected regions that satisfy all geometric constraints are determined as valid crack regions. S640, calculate the confidence level of the effective crack region based on the probability value of the corresponding position of the effective crack region on the crack probability map, and output the position coordinates, crack length, crack width distribution and confidence level of each effective crack region to generate crack detection results.

10. A road crack defect detection system based on multi-scale feature fusion, used to implement the road crack defect detection method based on multi-scale feature fusion as described in any one of claims 1-9, characterized in that: include: Data acquisition module: Acquires multi-source data of the target road section, including visible light image sequences, infrared thermal imaging sequences, and three-dimensional laser point cloud data; The geometry-temperature fusion modeling module: jointly registers the infrared thermal imaging sequence with the three-dimensional laser point cloud data to construct a three-dimensional surface mesh model M with temperature attributes, and performs curvature field analysis on M to extract the set of curvature-significant regions R that reflect local geometric abrupt changes; Dual-branch heterogeneous feature extraction module: The visible light image sequence is input into the dual-branch heterogeneous encoder, wherein the first branch uses dilated convolutional pyramid to extract multi-scale structural damage explicit feature map Fdam, and the second branch uses deformable convolution to extract surface continuity perturbation feature map Fdis aligned with geometric deformation. R is used as a spatial attention prior to weight and enhance Fdam and Fdis respectively, to obtain enhanced structural damage explicit feature Fdam' and enhanced surface continuity perturbation feature Fdis'. Intermodal iterative collaborative optimization module: Perform intermodal iterative collaborative optimization on Fdam' and Fdis'. In each iteration, Fdis' is restructured and recalibrated based on the current Fdam', and Fdam' is back-projected in detail based on the current Fdis', until the iteration converges and the deep fusion feature Ffuse is obtained. Dynamic routing decoding module: Adaptively selects upsampling paths of different scales based on the local energy distribution of Ffuse to generate a crack probability map P; Crack detection output module: Based on the crack probability map P and the preset crack geometric constraints, determine the location, scale and confidence level of crack defects in the target road section, and generate crack detection results.

Images