A pavement disease detection method and system based on multi-modal signal fusion

Abstract

The present application relates to the technical field of road engineering detection, and particularly relates to a pavement disease detection method and system based on multi-modal signal fusion. The method comprises the following steps: acquiring multi-source detection signals of a road pavement; pre-processing and spatio-temporal alignment of the multi-source detection signals; feature extraction of the pre-processed and aligned multi-source detection signals; two-stage fusion of the extracted multi-source features to obtain comprehensive features; output of disease detection results based on the comprehensive features and calculation of road condition indexes. The present application comprehensively utilizes the complementary advantages of multi-source information, realizes comprehensive and accurate detection and quantitative evaluation of pavement surface and internal diseases, effectively overcomes the limitations of single sensor detection and environmental interference problems, realizes high-precision and real-time detection of asphalt pavement surface and internal diseases under complex weather and multi-working condition conditions, and improves road inspection efficiency and the scientificity of maintenance decision-making.

Description

Technical Field

[0001] This invention relates to the field of road engineering inspection technology, specifically to a method and system for detecting pavement defects based on multimodal signal fusion. Background Technology

[0002] In the process of road maintenance and operation management, asphalt pavement distress detection is a crucial foundation for determining maintenance plans, controlling maintenance costs, and ensuring driving safety. Current technologies often rely on manual inspections or single-sensor / single-modal detection methods. For example, some rely solely on visual images of the pavement for crack and pothole identification, while others use ground-penetrating radar (GPR) for internal structure detection, or only employ three-dimensional laser point clouds for pavement morphology measurement.

[0003] However, the above-mentioned single-source data detection method has the following problems: (1) Visual image detection is easily affected by light, shadow, rain, snow, road pollution, etc., and it is difficult to identify hidden defects inside the road surface; (2) GPR can reflect changes in internal structure, but it is insufficient in expressing texture-type defects such as fine surface cracks, and is easily affected by noise and environmental interference. (3) Point clouds can quantify road surface morphology and pothole depth, but have limited semantic expression of disease types, and data alignment and fusion are difficult; (4) Domain offset caused by different road surface scenarios (dry / wet, different materials, different construction quality) will significantly reduce the generalization ability of deep learning models in real roads; (5) In real engineering projects, the labeling cost is high, and the distribution of disease samples is small and unbalanced, which makes the model prone to bias towards large-class diseases and the detection accuracy is unstable.

[0004] Therefore, there is a need for a detection method and system that can integrate multi-source signals such as vision, point cloud and GPR under complex weather and multi-condition working conditions to achieve high-precision and real-time identification of road surface and internal defects, and output defect size parameters and road condition index (PCI). Furthermore, the system should be specifically optimized for different defect types and PCI assessments in terms of multimodal fusion and loss function design. Summary of the Invention

[0005] To address the shortcomings of existing methods and the needs of practical applications, this invention provides a pavement distress detection method based on multimodal signal fusion, comprising the following steps: Acquire multi-source detection signals of the road surface; preprocess and spatiotemporally align the multi-source detection signals; extract features from the preprocessed and aligned multi-source detection signals respectively; perform two-stage fusion of the extracted multi-source features to obtain comprehensive features; output the disease detection results and calculate the road condition index based on the comprehensive features.

[0006] Optionally, the multi-source detection signals include visual image signals, three-dimensional laser point cloud signals, and ground-penetrating radar electromagnetic signals.

[0007] Optionally, feature extraction is performed on the preprocessed and aligned multi-source detection signals separately, including the following steps: Visual image signals are input into a residual deep sub-domain adaptive network, using ResNet-50 as the backbone network to extract visual features. GPR electromagnetic signals are input into a network composed of a one-dimensional convolutional network and a bidirectional long short-term memory network (Bi-LSTM) to extract GPR features. Three-dimensional laser point cloud signals are input into a PointNet++ network to extract point cloud features. The residual deep sub-domain adaptive network embeds a sub-domain adaptive layer before the fully connected layer of ResNet-50, and minimizes the feature distribution difference between the source domain dataset and the target domain dataset through an inter-domain distribution alignment loss function to improve the model's generalization ability. The residual deep sub-domain adaptive network embeds a concomitant objective function, which is a center loss function, to make similar disease features cluster towards the center and improve feature discrimination.

[0008] Optionally, the two-stage fusion of the extracted multi-source features includes the following steps: The first stage of fusion involves concatenating the multi-source features, assigning weights to each modality feature, and then performing a weighted summation to obtain a primary fusion vector. The second stage of fusion involves inputting the primary fusion vector into a semantic-level fusion network, and mapping it in a unified semantic space through an attention mechanism or cross-scale feature fusion to obtain the comprehensive features.

[0009] Optionally, when assigning weights, at least one of the disease category information or road condition information of the current sample may be referenced.

[0010] Optionally, during the training phase, a hybrid loss function is used for optimization; the hybrid loss function includes at least the focal Tversky loss used to optimize disease region segmentation.

[0011] Optionally, the hybrid loss function further includes modality weighted guided loss, which guides the network to adjust the weight allocation of different modal features under different disease type samples.

[0012] Optionally, the hybrid loss function further includes a multimodal geometric consistency loss, used to constrain the consistency of the spatial location and size parameters of the same disease predicted based on different modal signals.

[0013] Optionally, the hybrid loss function further includes a road condition index constraint loss, used to constrain the deviation between the predicted road condition index and the target road condition index.

[0014] Secondly, this invention also provides a pavement distress detection system based on multimodal signal fusion, comprising: a data acquisition module for synchronously acquiring multi-source detection signals of the road surface; a data preprocessing and alignment module, communicatively connected to the data acquisition module, for preprocessing and spatiotemporally aligning the multi-source detection signals; a multi-source feature extraction module, communicatively connected to the data preprocessing and alignment module, for extracting features from the preprocessed and aligned multi-source detection signals respectively; a multimodal feature fusion module, communicatively connected to the multi-source feature extraction module, for fusing the extracted multi-source features to obtain comprehensive features; and a distress detection and assessment output module, communicatively connected to the multimodal feature fusion module, for outputting distress detection results and calculating a road condition index based on the comprehensive features. This pavement distress detection system based on multimodal signal fusion is compact in structure and stable in performance, capable of stably executing the pavement distress detection method based on multimodal signal fusion provided by this invention, further enhancing the overall applicability and practical application capability of this invention.

[0015] This invention simultaneously acquires and fuses multi-source detection signals from road surfaces, comprehensively utilizing the surface texture perception capabilities of visual image signals, the internal structural anomaly detection capabilities of ground-penetrating radar electromagnetic signals, and the spatial morphology quantification capabilities of 3D laser point cloud signals. This enables the simultaneous detection of both surface and internal road surface defects, overcoming the limitations of single-sensor detection ranges and significantly improving the comprehensiveness and accuracy of detection. Through data preprocessing and alignment modules, the multi-source detection signals are spatiotemporally synchronized, ensuring consistent temporal and spatial correspondence between the data upon which subsequent feature extraction and fusion are based. This lays the foundation for solving the difficulties of multi-source data fusion and improves the accuracy and reliability of the fusion process.

[0016] Furthermore, features are extracted using networks designed for different signal characteristics. Examples include residual networks for visual images, convolutional and long short-term memory networks for ground-penetrating radar time-series signals, and hierarchical sampling networks for 3D point clouds. This allows for a more thorough extraction of the disease information contained in various signals, providing a high-quality feature foundation for subsequent fusion. A two-stage fusion architecture is employed. First, at the feature level, preliminary fusion is achieved through adaptive weight allocation, dynamically adjusting the contribution of each modality feature based on different disease types or road conditions. Then, at the semantic level, deeper association and mapping are achieved through methods such as attention mechanisms. This process effectively integrates complementary information and suppresses redundancy or noise interference, thereby generating more robust comprehensive features and solving the problem of poor detection stability caused by environmental interference.

[0017] Furthermore, the introduced hybrid loss training submodule performs end-to-end optimization of the model using a total loss function, including FocalTverskyLoss. FocalTverskyLoss effectively focuses on small-area or difficult-to-detect defect samples, improving the detection accuracy of targets such as small cracks. Modal weight-guided loss guides the adaptive weight allocation module to learn weight allocation strategies that conform to prior knowledge of different defect types, making the fusion process more targeted. Multimodal geometric consistency loss constrains the consistency of spatial parameters predicted by different modalities for the same defect, improving the accuracy of defect size and location estimation, thus providing reliable input for subsequent accurate calculation of the road condition index. The road condition index constraint loss directly correlates defect detection results with the final engineering evaluation index, optimizing the entire detection network through backpropagation. This not only ensures accuracy in individual defect identification but also better meets engineering specifications at the overall road condition assessment level, ultimately achieving high-precision and highly robust pavement defect detection and quantitative road condition assessment. Attached Figure Description

[0018] Figure 1 A flowchart of a pavement distress detection method based on multimodal signal fusion is provided in an embodiment of the present invention; Figure 2 This is a framework diagram of a road surface defect detection system based on multimodal signal fusion, provided for an embodiment of the present invention. Detailed Implementation

[0019] Specific embodiments of the present invention will now be described in detail. It should be noted that the embodiments described herein are for illustrative purposes only and are not intended to limit the invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. In other instances, well-known circuits, software, or methods have not been specifically described to avoid obscuring the invention.

[0020] Throughout this specification, references to "an embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "in an embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. Moreover, those skilled in the art will understand that the illustrations provided herein are for illustrative purposes and are not necessarily drawn to scale.

[0021] Please see Figure 1This invention provides a method for detecting pavement defects based on multimodal signal fusion, comprising the following steps: S1. Acquire multi-source detection signals of the road surface.

[0022] In this embodiment, the multi-source detection signal includes: (1) Road surface visual image signals, used to reflect road surface texture information, cracks, potholes, looseness and other surface defects details; (2) Three-dimensional laser point cloud signal is used to characterize the spatial morphology information of the road surface and quantify the pothole depth, disease outline and elevation change; (3) GPR electromagnetic signals are used to reveal abnormalities in the internal structure of the road surface, such as hidden defects such as voids and interlayer separation.

[0023] Furthermore, multiple sources of signals are collected simultaneously by drones and vehicle-mounted detection platforms, and the three types of signals are aligned in time and space through GNSS / RTK timestamps, so that each road surface location corresponds to the three types of signal data, providing an accurate correspondence for subsequent feature fusion.

[0024] S2. Preprocess and align the multi-source detection signals in time and space.

[0025] In the embodiments, various signals are subjected to noise reduction, formatting, and geometric alignment.

[0026] (1) Noise reduction processing: The visual image signal is denoised using adaptive Gaussian filtering to suppress the effects of illumination noise and sensor noise on crack details; GPR electromagnetic signals employ wavelet threshold denoising to suppress background noise and random disturbances, highlighting abnormal responses of the internal structure. The 3D point cloud signal is denoised by voxel filtering and downsampling to remove outliers and reduce computational load.

[0027] (2) Alignment processing: Based on GNSS / RTK timestamps and spatial coordinates, visual images, point clouds, and GPR waveforms are geometrically aligned according to the same location on the same road segment to form multimodal sample units that can be used for fusion.

[0028] S3. Perform feature extraction on the preprocessed and aligned multi-source detection signals respectively.

[0029] S3.1 Visual signal feature extraction (DSAN+ResNet-50).

[0030] In this embodiment, the visual feature extraction network adopts an improved residual deep sub-domain adaptive network (DSAN), and the backbone network is ResNet-50.

[0031] 1) Domain Shift Suppression: A sub-domain adaptation layer is embedded before the ResNet-50 fully connected layer. This minimizes the feature distribution difference between the source domain (public dataset) and the target domain (real road data) through inter-domain distribution alignment loss, thereby improving the model's cross-scene generalization ability. The inter-domain distribution alignment loss... Satisfying the formula:

[0032] Among them, D s For the source domain dataset, D t Given the target domain dataset, f(x) is the feature vector extracted by ResNet-50, and M... s M t , where represents the number of samples in the source domain and the target domain, respectively, and N represents the number of training samples in a single batch.

[0033] 2) Enhanced Feature Discrimination: An accompanying objective function is embedded in the network, and a center loss is used as a supplement to the classification loss. This causes similar disease features to cluster towards the center, while dissimilar features move away from each other, improving the fine-grained discrimination ability for diseases such as cracks, pits, and loose areas. The center loss satisfies the formula:

[0034] Among them, c yi The feature center vector of the first type of disease is updated iteratively in each round of training. m is the number of trained samples for this class.

[0035] The total loss is the weighted sum of the classification loss and the center loss, and the total loss function is:

[0036] Multi-scale feature aggregation: A fusion structure of lateral connectivity and deconvolution upsampling is adopted to achieve effective aggregation of features at different levels. The specific process is as follows: Feature extraction: Extract features from four layers (C2, C3, C4, and C5) of ResNet-50, corresponding to resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. Dimensionality reduction and alignment: Dimensionality reduction of C3, C4, and C5 is performed using 1×1 convolution and the number of channels is unified to 256; Upsampling fusion: Deconvolutional upsampling (4×4 kernel, stride 2) is used to upsample and fuse to C2 step by step to obtain a multi-scale fused feature map, which takes into account both fine cracks, small targets and semantic expression.

[0037] S3.2GPR signal feature extraction (1D-CNN+Bi-LSTM).

[0038] In this embodiment, the GPR signal is a one-dimensional time-series waveform, and a progressive fusion network combining local pattern extraction and time-dependent modeling is used. First, convolution operations are performed on the one-dimensional waveform to extract local patterns, and then a bidirectional long short-term memory network (Bi-LSTM) is used to model the time-series features to extract the structured signal features of internal defects.

[0039] 1) Local pattern extraction: A 3-layer stacked 1D convolutional network is used. Layer 1: Kernel size 3, input channel 1, output channel 64, stride 1, padding method same, activation function ReLU; Layer 2: Kernel size 5, input channels 64, output channels 128, stride 1, padding method same, activation function ReLU; Layer 3: Kernel size 7, input channels 128, output channels 256, stride 1, padding method same, activation function ReLU; Phase 1: Feature-level fusion In this embodiment, the ReLU activation function is uniformly used for visual features, GPR features, and point cloud features; a BatchNorm layer and a Dropout layer (dropoutrate=0.2) are added after each convolutional layer to suppress overfitting.

[0040] Temporal feature aggregation: The 256-dimensional feature sequence output by 1D convolution is input into the Bi-LSTM network. The Bi-LSTM is a 2-layer structure with 128 hidden units in each layer. The bidirectional outputs are concatenated to obtain a 256-dimensional temporal aggregated feature vector. This vector contains both local waveform patterns and global temporal dependencies, which can accurately characterize the signal features of internal defects such as voids and interlayer delamination.

[0041] S3.3 Point Cloud Signal Feature Extraction (PointNet++).

[0042] In this embodiment, the point cloud signal feature extraction adopts the PointNet++ network structure, which captures the three-dimensional morphological structure features of the road surface through hierarchical sampling and local region feature learning, and outputs a 256-dimensional point cloud feature vector to characterize spatial features such as pothole depth and deformation contour.

[0043] Specifically, a single frame of 3D laser point cloud is denoted as... , where x i ,y i ,z i Let represent the spatial coordinates of the i-th point, and N be the number of points in the point cloud of this frame.

[0044] The PointNet++ feature extraction process in this embodiment includes three steps: hierarchical sampling, grouping, and local feature learning (PointNet-based Feature Learning), and progressively extracts spatial morphological features from local to global in a multi-layered structure.

[0045] Stratified sampling and grouping.

[0046] Let a single frame of 3D laser point cloud be denoted as point set P, where each point contains spatial coordinates (x, y, z). In the l-th layer of the network, the input point set is... The corresponding feature is denoted as From farthest point sampling (FPS) Select several center points to form a set of center points. For each center point, within a given radius r l The system searches for neighboring points within a given range to form a local point set. Then, it constructs relative coordinates to emphasize the local geometry; the formula for relative coordinates is:

[0047] in, In the lth layer, with Centered on, with radius r l The j-th neighboring point within the neighborhood; This represents the relative coordinates of a neighboring point with respect to the center point.

[0048] Local feature learning: Within each local region, relative coordinates and point features are used as input. A small multilayer perceptron (MLP) network is used to extract the intermediate features of each point, denoted as... Then, pooling (e.g., max pooling) is performed on all point features within a local region to obtain an aggregated feature vector for that local region. The formula for point-level feature extraction is: The formula for local aggregation features is: ,in, The · symbol represents taking the maximum value for each feature dimension for all points in that local area. After the above operations, each center point obtains a local feature vector, which constitutes the point feature set of the next layer.

[0049] Multi-layer structure and global feature generation.

[0050] Multi-scale features are extracted layer by layer through a multi-level sampling-grouping-local feature learning module. For example, the feature dimensions of the three-layer structure are 64, 128, and 256 respectively. The point-level feature set obtained in the last layer can be denoted as {f}. k}, where each f k It is a 256-dimensional vector.

[0051] To obtain the global point cloud features of the entire road segment, global pooling (e.g., taking the maximum or average value of all point features in each dimension) is used to compress these point-level features into a 256-dimensional global feature vector f. pc , ,in (·) indicates that the feature of all points is taken as the maximum value in each dimension. This 256-dimensional global point cloud feature vector is the output of the PointNet++ module, which is used in the subsequent multimodal fusion stage to jointly characterize the depth of potholes, the contour of defects, and the three-dimensional morphological features of the road surface with visual features and GPR features.

[0052] S4. Perform two-stage fusion of the extracted multi-source features to obtain comprehensive features.

[0053] In the first stage of fusion, multi-source features are concatenated, weights are assigned to each modality feature, and then a weighted sum is obtained to obtain a primary fusion vector. When assigning weights, at least one of the following can be referenced: the disease category information or road condition information of the current sample, to achieve dynamic and adaptive feature-level fusion. Specifically, this includes: 1) Normalization: Perform L2 normalization on the three types of features respectively to eliminate dimensional differences; 2) Stitching: Stitching is performed in the order of vision → GPR → point cloud to obtain a 768-dimensional joint feature vector; 3) Adaptive Weight Allocation: The adaptive weight allocation module is set as a two-layer fully connected network: input dimension 768, first hidden layer number of units 384, second layer output 3 weight values, activation function Sigmoid; the output weights are normalized by Softmax to obtain ω. v ω g ω p And satisfy: ω v +ω g +ω p =1.

[0054] The feature-level fusion vector F is obtained by weighting and summing the features of each modality according to their respective weights. fuse =ω v ·f v +ωg·f g +ω p ·f p , where f v For visual features, f g For GPR features, f p These are point cloud features.

[0055] Note: When detecting internal defects such as cavities, the weight allocation module can increase the GPR weight; when detecting cracks, it can increase the visual weight; when detecting pit depth, it can increase the point cloud weight, thus achieving adaptive fusion.

[0056] In a preferred embodiment, considering the significant differences in the dependence of different disease types on the three modalities of vision, GPR, and point cloud, this invention introduces a disease category-driven modal weight guidance mechanism based on the adaptive weight allocation module. Specifically, for each disease type y, a pre-defined modal weight prior vector corresponding to that disease is pre-set: ,in , , , representing the expected weights of the visual modality, GPR modality, and point cloud modality under this disease type, respectively. For example, for internal voids and interlayer separation diseases, the GPR modality weights are... Set to the maximum value; for surface crack defects, set the visual modal weights to the maximum value. The maximum value is used; for pothole defects, the point cloud modal weights are set to the maximum value. Set it to the maximum value.

[0057] During the training phase, this invention obtains the disease category y from the fusion features through a lightweight classification head or prior labels, and encodes the disease category into a low-dimensional vector. This vector is then concatenated with visual features, GPR features, and point cloud features and input into the adaptive weight allocation module. This allows the module to simultaneously perceive disease category information when performing feature weighting. Furthermore, this embodiment constructs a modal weight guided loss function to constrain the modal weight vector output by the adaptive weight allocation module. The deviation from the prior vector of disease type is in the form of: .

[0058] By introducing the modal weights into the total loss to guide the loss, the GPR modal weights are automatically increased in internal defect samples, the visual modal weights are automatically increased in surface crack samples, and the point cloud modal weights are automatically increased in pit samples. This strengthens the contribution of each modality to its dominant defect type, significantly reduces the missed detection rate of internal defects and the 3D size estimation error, and improves the adaptability of multimodal fusion to complex working conditions.

[0059] In another embodiment, the input to the adaptive weight allocation module includes not only a concatenated vector of visual features, GPR features, and point cloud features, but also a scene feature vector representing road conditions (such as dry / wet conditions, weather conditions, road age, etc.). These scene features are obtained by encoding parameters such as weather, temperature, humidity, and traffic load. By simultaneously introducing defect category information and road condition information, the adaptive weight allocation module can dynamically adjust the contribution of each modality according to different road segment scenarios, maintaining stable defect detection and PCI assessment performance even under complex conditions such as rain, snow, nighttime, and severe pollution.

[0060] In the second stage of fusion, the primary fusion vector is input into a semantic-level fusion network (such as a network based on the Transformer architecture), and mapped in a unified semantic space through attention mechanism or cross-scale feature fusion to obtain comprehensive features that can fully represent disease information.

[0061] In this embodiment, the first-stage fusion result is input into the improved Transformer detection framework for semantic-level fusion: The Transformer encoder has 6 layers, each containing 8 attention heads, and the hidden layer dimension is 768. It mines the global semantic association of cross-modal features through a multi-head self-attention mechanism to obtain cross-modal semantic representation. A category-aware attention module is embedded in the Transformer decoder to dynamically adjust the attention weights according to the preset disease categories, so as to achieve accurate mapping of fused features in a unified semantic space and output disease classification and localization results.

[0062] Alternatively, the fused features can be input into the CSPNEXt module of the improved YOLOv9 detection framework to enhance semantic expressive capabilities through cross-scale feature fusion and adaptive feature selection, thereby achieving real-time target detection.

[0063] In a preferred embodiment, in order to ensure that the spatial position and size estimation of the three modalities of vision, GPR and point cloud on the same lesion are consistent, thereby improving the accuracy of the 3D parameters of the lesion and the PCI calculation results after multimodal fusion, the present invention introduces a multimodal geometric consistency constraint mechanism during the training phase.

[0064] Specifically, while maintaining GNSS / RTK temporal and spatial alignment and cross-modal collaborative data augmentation (synchronous rotation and translation), this invention obtains the prediction center coordinates c of the same defect in each modality from the visual branch, point cloud branch, and GPR branch, respectively. v ,c p ,c g and the size parameter vector s v ,s p ,s g Constructing a multimodal geometric consistency loss: , where c i ,c j The coordinates of the disease center predicted by the i-th and j-th modes are s, respectively. i ,s j , which are the predicted disease size parameter vectors, These are the weighting coefficients for positional consistency and size consistency.

[0065] By minimizing the geometric consistency loss, the position and size estimates of the three modalities—visual, point cloud, and GPR—on the same lesion are kept highly consistent. This effectively eliminates local deviations caused by single-modal noise, occlusion, or echo interference, making the three-dimensional size parameters of the lesion after multimodal fusion more stable and accurate, and providing more reliable input for subsequent PCI deduction calculations based on lesion size.

[0066] S5. Output the disease detection results based on the comprehensive features and calculate the road condition index.

[0067] The detection head network decodes the comprehensive features and outputs the type, location coordinates (such as the center point of the bounding box), and dimensional parameters (such as length, width, depth, and area) of the defects. Subsequently, based on the formulas and rules specified in the "Highway Technical Condition Assessment Standard" JTG5210-2018, the road condition index (PCI) is automatically calculated using the output defect information.

[0068] During the model training phase, a hybrid loss function is used to perform end-to-end optimization of the network involved in the above methods. The hybrid loss function includes at least the FocalTversky Loss for optimizing disease region segmentation. Additionally, the hybrid loss function may optionally include a modal weight-guided loss to guide the network in adjusting the weight allocation of different modal features under different disease type samples; a multimodal geometric consistency loss to constrain the consistency of the spatial location and size parameters of the same disease predicted based on different modal signals; and a road condition index constraint loss to constrain the deviation between the predicted road condition index and the target road condition index. By jointly optimizing these loss terms, the trained model achieves significant improvements in detection accuracy, robustness, and engineering applicability.

[0069] In this embodiment, a total loss function consisting of FocalLoss, FocalTverskyLoss, modal weighted guided loss, PCI constraint loss, and multimodal geometric consistency loss is used.

[0070] (1) FocalLoss is used in the classification and detection part. FocalLoss is used to improve the attention to disease categories in small samples; DiceLoss is used to optimize the prediction accuracy of disease area boundaries.

[0071] The FocalLoss formula is: , where α m For class weights, a weight of 0.7 is set for small sample classes and a weight of 0.3 is set for large sample classes; =2; Let be the predicted probability of the model for the m-th class.

[0072] Furthermore, the disease region segmentation part uses FocalTverskyLoss instead of the traditional DiceLoss. For the disease segmentation task, let Y be the mask of the actual disease region. To predict the disease area mask, the Tversky index (TI) is defined as follows: ,in This represents the number of pixels at the intersection of the actual and predicted regions. Indicates the number of missed pixels. Indicates the number of false positives. These are the weighting coefficients for controlling the severity of penalties for missed detections and false detections, respectively.

[0073] FocalTverskyLoss is defined as: , Used for focusing on difficult-to-detect samples, this embodiment preferably sets... Compared to the traditional DiceLoss, FocalTverskyLoss can assign higher gradient weights to small disease areas and difficult-to-detect diseases, significantly reducing the risk of small targets such as fine cracks and small pits being submerged by the background, and providing a more refined gradient at the disease boundary, which is beneficial to improving the accuracy of PCI deduction based on disease area and length.

[0074] To constrain the differences between the visual weights, GPR weights, and point cloud weights output by the adaptive weight allocation module and the prior modal weights corresponding to each disease type, this invention introduces modal weight guided loss. Its form is: This loss guides the network to increase the GPR modal weights in the internal defect scenario, the visual modal weights in the surface crack scenario, and the point cloud modal weights in the pit scenario, thereby significantly reducing the missed detection rate of internal defects and improving the accuracy of 3D size estimation.

[0075] To ensure consistency in the spatial location and size estimation of multimodal data on the same defect, this invention introduces a multimodal geometric consistency loss L. align Its definition is: This loss suppresses position and size deviations caused by single-mode noise and improves the stability of multi-mode fused disease size parameters.

[0076] In another preferred embodiment, in order to make the disease detection results output by the model closer to the actual engineering situation at the level of overall road section technical condition assessment, the present invention introduces the PCI calculation results of the "Highway Technical Condition Assessment Standard" (JTG5210-2018) into the network training process to construct PCI constraint loss.

[0077] For training samples, the target PCI value (PCI) is obtained by manual evaluation or traditional detection methods according to the JTG5210-2018 standard. true The disease type, location, and three-dimensional size parameters output by the model of this invention are input into the PCI calculation module, and the predicted PCI value is calculated according to the standard formula. pred Define the PCI constraint loss as: The PCI constraint loss, along with FocalLoss, FocalTverskyLoss, modal weight guidance loss, and multimodal geometric consistency loss, constitute the total loss function, which is then used for end-to-end joint training of the multimodal feature extraction network, adaptive weight allocation module, and detection network.

[0078] By introducing PCI constraint loss, this invention not only optimizes the detection and segmentation accuracy of individual defects, but also further aligns the PCI assessment results of the entire road segment with the manual assessment values. This effectively reduces the systematic deviation of PCI under complex weather and multi-condition conditions, and improves the engineering usability of road condition assessment results.

[0079] In summary, the total loss function L of the present invention total for: ,in, These are the weighting coefficients for each loss item. By reasonably setting the above weights, this invention has significant advantages in solving problems such as uneven disease categories, easy omission of small-target diseases, imbalance of multimodal contributions, and PCI assessment bias.

[0080] In this embodiment, a transfer learning strategy is adopted: first, the model is pre-trained on a publicly available multimodal road defect dataset, and then fine-tuned on a small amount of real road data to reduce annotation costs and improve the model's generalization ability.

[0081] Specifically, training optimization and generalization improvement include: Regularization: Add Dropout (rate=0.1) to the fully connected layer and the fusion module. Cross-modal collaborative data augmentation: synchronously rotate visual images and point clouds (±15°), synchronously translate GPR signals and visual images (≤20 pixels), and maintain multimodal spatiotemporal consistency; Early stopping strategy: Using validation set mAP as the monitoring metric, if there is no improvement for 5 consecutive rounds, training is stopped and the optimal weights are saved.

[0082] Then, output the defect type (cracks, pits, loose areas, cavities, interlayer separation, etc.), two-dimensional location coordinates (based on the geodetic coordinate system, accuracy ≤5cm), and three-dimensional dimension parameters: Crack length error ≤ 0.1m, crack width error ≤ 0.5mm; The depth error of the pit is ≤0.05m, and the area error of the pit is ≤0.1m. 2 ; Void volume error ≤ 0.01m 3 ; Interlayer separation area error ≤ 0.2m 2 .

[0083] In this embodiment, the PCI is calculated according to the "Highway Technical Condition Assessment Standard" (JTG5210-2018), and the PCI calculation satisfies the following formula:

[0084] Among them, D i W is the deduction value for the i-th type of disease. i PCI is the weighting coefficient for the i-th type of disease, with a value range of 0-100.

[0085] In the reasoning and application stage, this invention calculates the road condition index (PCI) based on the disease type, two-dimensional location coordinates and three-dimensional size parameters output by the model, in accordance with the "Highway Technical Condition Assessment Standard" (JTG5210-2018). The output PCI value ranges from 0 to 100, which is used to provide a quantitative basis for road maintenance decisions.

[0086] During the training phase, this invention further uses the PCI calculation process to obtain the predicted PCI value, compares it with the target PCI value obtained by manual evaluation or traditional methods, and constructs the PCI constraint loss L. PCI The PCI bias is backpropagated as an additional training signal to the multimodal feature extraction and fusion network to reduce the overall road segment PCI assessment error.

[0087] In summary, by introducing a modal weight guidance mechanism driven by disease category and road condition during the multimodal feature fusion stage, and by superimposing FocalTverskyLoss, modal weight guidance loss, multimodal geometric consistency loss, and PCI constraint loss during training, this invention significantly enhances the adaptability to internal diseases, small-target diseases, and complex working conditions while maintaining the overall simplicity of the detection framework. It also achieves a high degree of consistency with the results of standard and manual assessment at the overall PCI evaluation level, demonstrating a comprehensive technical effect and engineering application value that is significantly superior to existing technologies.

[0088] Please see Figure 2In an embodiment, to efficiently execute the pavement distress detection method based on multimodal signal fusion provided by the present invention, the present invention also provides a pavement distress detection system based on multimodal signal fusion, comprising: a data acquisition module for synchronously acquiring multi-source detection signals of the road surface; a data preprocessing and alignment module, communicatively connected to the data acquisition module, for preprocessing and spatiotemporally aligning the multi-source detection signals; a multi-source feature extraction module, communicatively connected to the data preprocessing and alignment module, for extracting features from the preprocessed and aligned multi-source detection signals respectively; a multimodal feature fusion module, communicatively connected to the multi-source feature extraction module, for fusing the extracted multi-source features to obtain comprehensive features; and a distress detection and evaluation output module, communicatively connected to the multimodal feature fusion module, for outputting distress detection results and calculating a road condition index based on the comprehensive features.

[0089] The data acquisition module is used to simultaneously acquire multi-source detection signals of the road surface. In one embodiment, the data acquisition module specifically includes a visual image acquisition unit, a 3D laser scanning unit, and a ground-penetrating radar (GPR) detection unit. The visual image acquisition unit acquires visual image signals of the road surface through a high-resolution camera to capture texture information such as surface cracks and potholes. The 3D laser scanning unit acquires 3D laser point cloud signals of the road surface through laser radar scanning to quantify the spatial morphology of the road surface and the 3D contours of defects. The GPR detection unit acquires ground-penetrating radar electromagnetic signals of the road surface by emitting and receiving electromagnetic waves to detect hidden defects such as cavities and interlayer delamination within the road surface. The above three units are mounted on a unified hardware platform (such as a vehicle-mounted or UAV platform) and employ Global Navigation Satellite System Real-Time Dynamic Differential (GNSS / RTK) technology to assign high-precision spatial coordinates and timestamps to each frame of data acquired.

[0090] In addition to the visual image acquisition unit, the three-dimensional laser scanning unit, and the ground-penetrating radar detection unit, the data acquisition module may also optionally include other types of sensors, such as an infrared thermal imager to acquire road surface temperature distribution signals to help identify thermal anomaly areas caused by internal defects.

[0091] The data preprocessing and alignment module communicates with the data acquisition module to preprocess and spatiotemporally align multi-source detection signals. Spatiotemporal alignment primarily utilizes the timestamps from the GNSS / RTK real-time dynamic differential technology to synchronize visual image signals, 3D laser point cloud signals, and ground-penetrating radar electromagnetic signals in time and space. Specifically, the processor parses the GNSS / RTK timestamps to associate visual images, laser point cloud frames, and ground-penetrating radar waveform data packets acquired at the same time and geographical location, forming multimodal sample pairs that correspond one-to-one in time and space, laying the foundation for subsequent precise feature-level fusion.

[0092] Regarding preprocessing operations, in one embodiment, the data preprocessing and alignment module specifically includes a visual image processing unit, a ground-penetrating radar signal processing unit, and a point cloud processing unit. The visual image processing unit performs adaptive Gaussian filtering on the visual image signal to suppress the impact of factors such as uneven illumination and sensor noise on image quality. The ground-penetrating radar signal processing unit performs wavelet thresholding on the ground-penetrating radar electromagnetic signal to filter out background noise and random interference, highlighting the reflection characteristics of internal structural anomalies. The point cloud processing unit performs voxel filtering and downsampling on the 3D laser point cloud signal to remove outlier noise points and reduce data volume, improving the efficiency of subsequent processing.

[0093] The multi-source feature extraction module communicates with the data preprocessing and alignment module to extract features from the preprocessed and aligned multi-source detection signals. This module includes a visual feature extraction submodule, a ground-penetrating radar feature extraction submodule, and a point cloud feature extraction submodule. The visual feature extraction submodule is based on a residual network architecture and embeds a sub-domain adaptive layer to extract visual features from visual image signals. In specific implementation, ResNet-50 can be used as the backbone network, and a sub-domain adaptive layer can be embedded before its fully connected layer. This suppresses domain shift by minimizing the feature distribution difference between the source domain (such as public datasets) and the target domain (real road data), thereby improving the model's generalization ability in different road scenarios. At the same time, features at different levels can be aggregated through lateral connections and deconvolution upsampling to take into account both the detailed information of small targets such as fine cracks and the semantic information of deep features.

[0094] The ground-penetrating radar feature extraction submodule includes a one-dimensional convolutional network and a bidirectional long short-term memory network (Bi-LSTM) to extract temporal structure features from the ground-penetrating radar electromagnetic signals. Specifically, the one-dimensional convolutional network (such as a 3-layer stacked 1D-CNN) first extracts local patterns from the waveform signal; then, the Bi-LSTM network models the temporal dependencies of the extracted local features, and finally outputs a temporal structure feature vector that integrates local and global information. This feature can effectively characterize the signal patterns of internal defects such as cavities and interlayer delamination.

[0095] The point cloud feature extraction submodule is based on a point cloud processing network using hierarchical sampling and local feature learning to extract 3D morphological spatial features from 3D laser point cloud signals. Specifically, a PointNet++ network structure can be used. This network selects center points through farthest point sampling, then groups the neighborhood of each center point, extracts local region features using a multilayer perceptron, and aggregates features through operations such as max pooling. Through this multilayer "sampling-grouping-feature learning" process, the network can capture the 3D geometry of the road surface from local to global perspectives, ultimately outputting a feature vector representing spatial features such as pothole depth and deformation contours.

[0096] The multimodal feature fusion module is communicatively connected to the multi-source feature extraction module to fuse the extracted multi-source features and obtain comprehensive features. This module adopts a two-stage fusion architecture, including a first-stage fusion unit and a second-stage fusion unit. The first-stage fusion unit performs feature-level fusion of visual features, temporal structure features, and three-dimensional morphological space features to obtain a primary fusion vector. In one embodiment, the first-stage fusion unit includes an adaptive weight allocation module. The processor concatenates the visual features, temporal structure features, and three-dimensional morphological space features to form a concatenated vector, which serves as the input to the adaptive weight allocation module. The adaptive weight allocation module (e.g., a two-layer fully connected network) is configured to output the weight values ​​corresponding to the visual features, temporal structure features, and three-dimensional morphological space features, and performs a weighted summation of each feature according to its corresponding weight to output the primary fusion vector. This adaptive weighting mechanism allows the network to dynamically adjust the contribution of each modal feature based on the degree of modal dependence of different diseases; for example, assigning higher weights to ground-penetrating radar features when detecting internal cavities, and assigning higher weights to visual features when detecting surface cracks.

[0097] In addition to the basic inputs mentioned above, the adaptive weight allocation module may optionally include at least one of the following: a disease category encoding vector or a road condition feature vector. The disease category encoding vector can be predicted from the fused features using a lightweight classification head, or provided by the training sample labels; the road condition feature vector can be obtained by encoding parameters such as weather, temperature, and road age. By introducing this additional information, the adaptive weight allocation module can more accurately allocate modal weights, further improving the targeting and robustness of the fusion process.

[0098] The second-stage fusion unit communicates with the first-stage fusion unit and performs semantic-level fusion on the primary fusion vector to obtain comprehensive features. In one embodiment, the second-stage fusion unit is a semantic fusion module based on the Transformer architecture. The processor inputs the primary fusion vector into the Transformer encoder and mines the global semantic associations of cross-modal features through a multi-head self-attention mechanism, achieving high-dimensional mapping of features in a unified semantic space. In another optional embodiment, the second-stage fusion unit is a semantic fusion module based on a single-stage object detection network, such as inputting the primary fusion vector into the CSPNEXt module of an improved YOLOv9 network, enhancing semantic expressiveness through cross-scale feature fusion. Semantic-level fusion can further integrate complementary information and suppress redundant or conflicting information, generating more discriminative comprehensive features.

[0099] The road condition detection and assessment output module communicates with the multimodal feature fusion module to output road condition detection results and calculate the road condition index based on comprehensive features. This module typically includes a detection head (such as a classification and regression head) to output the type of road condition, bounding box location (two-dimensional coordinates), and dimensional parameters (such as length, width, depth, and area). Further, this module is configured to calculate the road condition index based on the output road condition type, location, and dimensional parameters, according to the "Highway Technical Condition Assessment Standard" JTG5210-2018. The processor has built-in PCI calculation logic that automatically calculates the PCI value reflecting the overall technical condition of the road segment according to the deduction rules and weights for various types of road conditions specified in the standard.

[0100] During the model training phase, the disease detection and assessment output module includes a hybrid loss training submodule, which is configured to compute the total loss function to optimize the entire network. The total loss function includes at least the FocalTversky Loss, used to optimize disease region segmentation. FocalTversky Loss effectively alleviates class imbalance by assigning higher loss weights to hard-to-classify pixels (such as small disease areas), improving the detection accuracy for small-target diseases like fine cracks. Its calculation formula can be constructed based on the Tversky index, and hard-to-classify samples can be focused using a focusing parameter γ (e.g., γ=4 / 3).

[0101] In one embodiment, the total loss function further includes a modal weighting guided loss. This loss constrains the difference between the weight values ​​output by the adaptive weighting module and preset modal prior vectors for different disease types. For example, for "internal cavitation" disease, the weight of the ground-penetrating radar mode in the prior vector should be higher; for "surface cracks," the weight of the visual mode should be higher. By minimizing this loss, the network is guided to learn a weighting strategy that conforms to prior knowledge, making the fusion process more targeted.

[0102] In another embodiment, the total loss function also includes a multimodal geometric consistency loss. This loss is used to constrain the consistency between the spatial location and size parameters of the same lesion predicted based on different modal signals. Specifically, the processor obtains the predicted location and size of the same lesion from the visual branch, point cloud branch, and ground-penetrating radar branch (if the architecture supports it), or from different mappings of the fused features, calculates the differences between them (such as Euclidean distance, size vector difference), and uses this difference as the consistency loss. This loss helps to eliminate the positioning and size estimation bias caused by single-modal noise, improving the accuracy of the final output three-dimensional parameters.

[0103] In another embodiment, the total loss function also includes a road condition index constraint loss. This loss constrains the deviation between the predicted road condition index calculated based on the defect detection results from the model output and the target road condition index (such as the true value assessed manually). By backpropagating this assessment-level error to the network, the model can be further optimized from the perspective of overall road condition assessment, making the detection results not only accurate individually but also more in line with engineering specifications in terms of overall assessment.

[0104] Optionally, a model training and optimization module may be included, which is communicatively connected to the multi-source feature extraction module, the multimodal feature fusion module, and the disease detection and assessment output module. The model training and optimization module is configured to perform cross-modal collaborative data augmentation, transfer learning, and regularization operations. For example, during data augmentation, synchronized rotation and translation transformations are performed on visual images and point clouds, and synchronized time offsets are performed on ground-penetrating radar signals to maintain spatiotemporal consistency among multimodal data. The transfer learning strategy can utilize publicly available datasets to pre-train the feature extraction sub-network, and then fine-tune it on a small amount of labeled real data to reduce annotation costs and improve generalization ability.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the present invention.

Claims

1. A method for detecting pavement defects based on multimodal signal fusion, characterized in that, Includes the following steps: Acquire multi-source detection signals from the road surface; The multi-source detection signals are preprocessed and spatiotemporally aligned. Feature extraction is performed on the preprocessed and aligned multi-source detection signals respectively; The extracted multi-source features are fused in a two-stage process to obtain comprehensive features. Based on the comprehensive features, the damage detection results are output and the road condition index is calculated.

2. The pavement distress detection method based on multimodal signal fusion according to claim 1, characterized in that, The multi-source detection signals include visual image signals, three-dimensional laser point cloud signals, and ground-penetrating radar electromagnetic signals.

3. The pavement distress detection method based on multimodal signal fusion according to claim 1, characterized in that, Feature extraction is performed on the preprocessed and aligned multi-source detection signals, including the following steps: Visual image signals are input into a residual depth sub-domain adaptive network, and ResNet-50 is used as the backbone network to extract visual features. GPR electromagnetic signals are input into a network composed of a one-dimensional convolutional network and a bidirectional long short-term memory network Bi-LSTM to extract GPR features. The 3D laser point cloud signal is input into the PointNet++ network to extract point cloud features; The residual deep sub-domain adaptive network embeds a sub-domain adaptive layer before the fully connected layer of ResNet-50, and minimizes the feature distribution difference between the source domain dataset and the target domain dataset through the inter-domain distribution alignment loss function to improve the model's generalization ability. The residual deep sub-domain adaptive network is embedded with a companion objective function, which is a center loss function, so as to make similar disease features cluster towards the center and improve feature discrimination.

4. The pavement distress detection method based on multimodal signal fusion according to claim 1, characterized in that, The two-stage fusion of extracted multi-source features includes the following steps: First-stage fusion: The multi-source features are concatenated, weights are assigned to each modality feature, and then a weighted sum is obtained to obtain the primary fusion vector; The second stage of fusion involves inputting the primary fusion vector into a semantic-level fusion network, and mapping it in a unified semantic space through an attention mechanism or cross-scale feature fusion to obtain the comprehensive features.

5. The pavement distress detection method based on multimodal signal fusion according to claim 4, characterized in that, When assigning weights, at least one of the following should be considered: the disease category information or the road condition information of the current sample.

6. The pavement distress detection method based on multimodal signal fusion according to claim 1, characterized in that, During the training phase, a hybrid loss function is used for optimization; the hybrid loss function includes at least the focal Tversky loss used to optimize the segmentation of diseased regions.

7. The pavement distress detection method based on multimodal signal fusion according to claim 6, characterized in that, The hybrid loss function also includes modality weighted guided loss, which guides the network to adjust the weight allocation of different modal features under different disease type samples.

8. The pavement distress detection method based on multimodal signal fusion according to claim 6, characterized in that, The hybrid loss function also includes multimodal geometric consistency loss, which is used to constrain the consistency of the spatial location and size parameters of the same disease predicted based on different modal signals.

9. The pavement distress detection method based on multimodal signal fusion according to claim 6, characterized in that, The hybrid loss function also includes a road condition index constraint loss, which is used to constrain the deviation between the predicted road condition index and the target road condition index.

10. A pavement distress detection system based on multimodal signal fusion, characterized in that, include: The data acquisition module is used to simultaneously acquire multi-source detection signals from the road surface; The data preprocessing and alignment module is communicatively connected to the data acquisition module and is used to preprocess and spatiotemporally align the multi-source detection signals. The multi-source feature extraction module is communicatively connected to the data preprocessing and alignment module, and is used to extract features from the preprocessed and aligned multi-source detection signals respectively. The multimodal feature fusion module is communicatively connected to the multi-source feature extraction module and is used to fuse the extracted multi-source features to obtain comprehensive features; The disease detection and assessment output module is communicatively connected to the multimodal feature fusion module and is used to output disease detection results and calculate the road condition index based on the comprehensive features.

Images