Multi-modal fusion tunnel structure apparent disease identification and risk assessment system

Abstract

The invention relates to the technical field of civil engineering tunnel structure safety monitoring and intelligent detection, in particular to a multi-modal fusion tunnel structure apparent disease identification and risk assessment system, which comprises an image acquisition module used for acquiring continuous images of the inner wall of a tunnel lining; a laser point cloud acquisition module; a structure sensor acquisition module; a data synchronization and preprocessing module; the multi-modal feature extraction module is used for performing depth feature extraction on the image, the point cloud and the sensor data; the heterogeneous feature fusion and disease identification module is used for fusing each modal feature and outputting a disease type identification result; and the risk assessment module is used for carrying out size estimation and parameterized expression on the identified diseases. The problems that in an existing tunnel inspection technology, the detection means is single, appearance and internal information cannot be considered, and the disease size is difficult to quantify automatically are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of civil engineering tunnel structure safety monitoring and intelligent detection technology, and particularly relates to a tunnel structure disease automatic identification and risk assessment system fusing visual perception and structural sensing data. BACKGROUND

[0002] During long-term service, underground structures such as tunnels are often affected by train vibration, geological stress and environmental factors, and are prone to diseases such as cracks, lining misalignment, concrete spalling, water seepage and the like. If these diseases are not discovered and repaired in a timely manner, the safety of the tunnel structure will be endangered. At present, the operation safety detection of tunnels mainly relies on manual inspection or regular inspection by inspection vehicles, which cannot realize real-time monitoring of the tunnel, and the detection results are largely dependent on the experience of workers, and there are deficiencies in inspection efficiency, stability, real-time performance and accuracy. Some tunnels are equipped with strain gauges, displacement meters and other sensors for structural health monitoring, but single sensing data cannot directly reflect the detailed characteristics of apparent diseases, and still requires manual comparison and analysis.

[0003] In recent years, with the development of computer vision and deep learning technology, methods for automatically detecting tunnel diseases using images and laser scanning have emerged. For example, tunnel inspection robots usually carry high-definition cameras or three-dimensional laser scanners to collect tunnel surface images and point cloud data, and identify apparent defects such as cracks and seepage water through algorithms. However, such single-modal data detection methods have limitations: on the one hand, relying only on video images or laser point clouds can only detect visible diseases on the surface of the tunnel, and cannot detect hidden diseases inside the structure; on the other hand, existing methods mostly stop at defect identification or positioning, and cannot accurately quantify the geometric dimensions of the disease, such as crack width and depth, which requires subsequent manual measurement and calibration.

[0004] To improve inspection efficiency and accuracy, multi-sensor fusion intelligent detection equipment has begun to be applied to tunnel and rail transit facility maintenance. For example, there are inspection vehicles that integrate line laser radar, high-definition cameras, infrared thermography and other sensors, and through AI analysis of 2D / 3D multi-source data, they can accurately identify diseases such as pipe piece cracks and seepage water as small as 0.2mm while moving, and can detect multiple abnormalities even with a small number of defect samples. Multi-modal deep learning technology using heterogeneous sensing data is expected to significantly improve the robustness and accuracy of defect detection. However, existing inspection systems still mainly focus on the fusion of apparent images and point cloud information, and lack joint analysis of structural strain, displacement and other sensor data, which limits the comprehensive perception of potential risks and the quantitative assessment of disease severity.

[0005] In summary, there is an urgent need for a multimodal fusion system that combines high-definition images, 3D laser point clouds, and structural monitoring sensor data to achieve automatic identification of common defects in tunnel structures and service risk assessment, thereby overcoming the shortcomings of existing technologies in terms of real-time performance, comprehensiveness, and quantitative accuracy. Summary of the Invention

[0006] Based on the above description, the present invention provides a multimodal fusion tunnel structure appearance defect identification and risk assessment system to solve the problems of single detection methods, inability to take into account both appearance and internal information, and difficulty in automatically quantifying defect size in existing tunnel inspection technologies.

[0007] On the one hand, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a multimodal fusion tunnel structure apparent defect identification and risk assessment system, comprising:

[0008] Image acquisition module, used to acquire continuous images of the inner wall of the tunnel lining;

[0009] The laser point cloud acquisition module is used to acquire three-dimensional point cloud data of the tunnel cross-section.

[0010] The structural sensor acquisition module is used to acquire structural state parameters such as strain, displacement, temperature, and humidity.

[0011] The data synchronization and preprocessing module performs time alignment, coordinate transformation, and noise filtering on the aforementioned multi-source data.

[0012] The multimodal feature extraction module performs deep feature extraction on images, point clouds, and sensor data;

[0013] The heterogeneous feature fusion and disease identification module fuses features from various modalities and outputs disease type identification results;

[0014] The risk assessment module estimates the size and parametrically represents the identified defects.

[0015] Based on the above technical solutions, the accuracy and quantitative analysis capabilities of tunnel defect identification are significantly improved. Through the collaborative acquisition and fusion of image, laser point cloud, and structural sensor data, comprehensive perception and complementary enhancement of defect information are achieved, effectively overcoming the limitations of single-modal data affected by illumination, occlusion, or local errors. The system possesses an efficient data alignment and preprocessing mechanism, ensuring the consistency of multi-source data in time and space, and improving subsequent identification accuracy. By integrating deep learning feature extraction and heterogeneous data fusion strategies, common defect types such as cracks and misalignments can be accurately identified. Simultaneously, by combining point cloud and image information, precise quantification of geometric dimensions is achieved, providing maintenance units with quantifiable and traceable defect assessment data, demonstrating high engineering practical value.

[0016] Based on the above technical solution, the present invention can be further improved as follows.

[0017] Furthermore, the multimodal feature extraction module includes:

[0018] A two-dimensional convolutional neural network submodule is used to extract texture and edge features such as cracks, peeling, and water stains in images;

[0019] The point cloud feature extraction submodule uses a 3D convolutional network or PointNet structure to extract geometric structural features from the point cloud.

[0020] The sensor sequence analysis submodule uses a one-dimensional convolutional network or a recurrent neural network to process time-series data such as strain and displacement.

[0021] Furthermore, after extracting features, the multimodal feature extraction module encodes them into high-dimensional feature vectors through modal embedding for subsequent fusion.

[0022] Furthermore, the heterogeneous feature fusion and disease identification module employs an attention mechanism for intermodal feature weighting, preferably including a multi-head cross-attention layer or a modal self-attention fusion structure.

[0023] Furthermore, the fusion module further integrates a feature pyramid structure to achieve a fusion representation of multi-scale disease features, thereby enhancing the detection capability for cracks and misalignments of different sizes.

[0024] Furthermore, the risk assessment module includes:

[0025] The crack geometry estimation submodule is used to calculate crack length, width, and depth by combining image scale and point cloud depth;

[0026] The misalignment height analysis submodule is used to extract the elevation difference of point clouds of adjacent cross sections and verify it in combination with displacement sensor readings;

[0027] The detachment area measurement submodule is used to calculate the area and volume of the point cloud concave region based on its boundary.

[0028] The submodule for analyzing water stain area and humidity changes is used to fuse image segmentation results with humidity temporal gradients to estimate the risk of water seepage.

[0029] The structural damage characteristics output from the crack geometry estimation submodule, the misalignment height analysis submodule, the detachment area measurement submodule, and the water stain area and humidity change analysis submodule are comprehensively weighted to form a risk index, and the risk level is classified according to the set threshold range.

[0030] Among them, the risk index Calculate using the following formula:

[0031] ;

[0032] In the formula:

[0033] The crack characteristic factor is represented by the crack length. ,width With depth Comprehensive calculations yielded the following results:

[0034] ;

[0035] in Reference limits for design permitting;

[0036] The misalignment characteristic factor is defined as the difference in point cloud elevation between adjacent cross sections. With sensor displacement reading Normalized combination:

[0037] ;

[0038] This represents the volume factor of the detachment, taking the volume of the concave region. Compared with reference volume The ratio;

[0039] The humidity factor of water stains is determined by the area of ​​water seepage. With humidity change rate Normalization yields:

[0040] ;

[0041] These are the weighting coefficients for each damage feature, determined based on field experience or training data;

[0042] When the risk index Exceeding the set threshold When this occurs, the system automatically determines it to be in an early warning state, and classifies it according to the risk level range. The system outputs three risk levels: "stable," "developing," or "deteriorating."

[0043] Furthermore, the visualization display module includes a monitoring terminal connected to the data processing and analysis module. The monitoring terminal marks the location, type, and size information of defects on the three-dimensional model or two-dimensional unfolded diagram of the tunnel structure, and issues an alarm signal when the size of the defect exceeds the limit.

[0044] Furthermore, the result output and visualization module highlights the location of the disease based on the three-dimensional tunnel model or the two-dimensional unfolded diagram, and displays the disease type, size and risk level in the form of labels.

[0045] Furthermore, the system has a sensor threshold alarm function. When real-time data exceeds the preset range or the size of the lesion exceeds the limit, it will automatically trigger an audible and visual alarm or remotely push alarm information.

[0046] Secondly, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a multimodal disease evolution monitoring and trend assessment system, including a multi-source data acquisition module, a multi-time series data archiving and alignment module, a disease evolution analysis module, a trend identification and risk classification module, and a result output module.

[0047] The multi-source data acquisition module is used to collect multimodal monitoring data of the tunnel structure, including image data of the inner surface of the tunnel, three-dimensional point cloud data of the tunnel cross section, and structural response data of key parts of the tunnel structure.

[0048] The multi-temporal data archiving and alignment module is used to perform temporal and spatial alignment and fusion of the image data, point cloud data and structural response data collected at different times to form a temporal fusion dataset that can be used for comparative analysis of disease evolution.

[0049] The disease evolution analysis module is used to extract evolutionary characteristic parameters of tunnel disease areas based on the time-series fusion dataset, including disease morphology expansion trend, structural response change trend, and evolution rate index; the trend identification and risk classification module is used to identify the development trend of the disease according to the evolutionary characteristic parameters and determine the risk level of the disease based on preset rules; the result output module is used to output the results of tunnel disease monitoring and trend assessment.

[0050] Based on the above technical solutions, this system fully integrates multi-source heterogeneous data such as images, point clouds, and structural responses, systematically solving the technical problems of traditional tunnel defect monitoring, such as one-sided perception, unclear trends, and strong subjectivity in risk assessment. Through the multi-source data acquisition module, the system can simultaneously perceive the external morphology and internal structural responses of defects, comprehensively covering the surface and internal state of the tunnel structure. The multi-time-series data archiving and alignment module achieves high-precision registration of data sampled from different modalities and times, constructing a unified data baseline for defect evolution and providing support for accurate comparison. The defect evolution analysis module can mine the morphological changes and structural response fluctuations of key areas over time, quantifying the rate of defect expansion and mechanical degradation trends. The trend identification and risk classification module combines rule algorithms and trend parameters to automatically determine the direction and severity of defect development, improving the objectivity and accuracy of early warnings. Finally, the results output module enables visual display and report generation, supporting decision-making units to intuitively grasp the defect evolution situation and formulate targeted maintenance strategies. The overall system has technical advantages such as intelligence, high precision, and strong real-time performance, significantly improving the scientific nature and efficiency of tunnel defect operation and maintenance.

[0051] Furthermore, the multi-source data acquisition module includes: several high-definition industrial cameras installed between tunnel lining ring segments as image acquisition units, with each camera spaced 30-60 meters apart along the longitudinal direction of the tunnel; a 360° rotating laser scanner configured on the tunnel maintenance passage or inspection carrier as a laser point cloud acquisition unit to acquire three-dimensional point cloud data of the tunnel cross-section and internal cavity space; and several structural sensor nodes deployed at vulnerable locations to acquire structural response data.

[0052] The sensor nodes include strain sensors, displacement sensors, and temperature and humidity sensors; all images, point clouds, and sensor data acquired by the multi-source data acquisition module are bound with precise timestamps and location information tags, and the consistency of multimodal data in time is ensured through a global clock synchronization module.

[0053] Furthermore, the multi-time-series data archiving and alignment module includes:

[0054] The image registration and archiving submodule is used to perform temporal archiving and geometric alignment processing on tunnel inner wall images acquired at different times from the same monitoring location;

[0055] The point cloud model alignment submodule is used to preprocess and spatially align the tunnel 3D point cloud data acquired in multiple monitoring cycles. After filtering, noise reduction and voxelization resampling of the original point cloud, the iterative nearest point ICP algorithm or the FPFH feature matching combined with the RANSAC algorithm is used to unify the point cloud models of each period into the same coordinate system.

[0056] The structural sensing data synchronization submodule is used to perform unified timeline alignment and association indexing of time-series data from structural sensors such as strain, displacement, temperature and humidity with corresponding image and point cloud data. It associates sensing data with image / point cloud records at the corresponding time based on sensor ID, installation location coordinates and acquisition timestamp.

[0057] Compared with the prior art, the technical solution of this application has the following beneficial technical effects:

[0058] 1. This system integrates three modules: image acquisition, laser point cloud acquisition, and structural sensor acquisition, enabling coordinated perception of the external morphology and internal response of tunnel defects. The image acquisition module accurately captures texture information such as cracks and water stains on the lining surface; the laser point cloud module provides high-precision three-dimensional structural data; and the structural sensor module records indicators reflecting internal stress and environmental conditions, such as strain, displacement, temperature, and humidity. These three perception modules cover the three-dimensional dimensions of tunnel structure—"representation-structure-environment"—significantly enhancing the accuracy and completeness of defect identification, providing multi-view, high-resolution raw input for subsequent analysis, and greatly improving the system's adaptability to diverse defect types.

[0059] 2. To address the differences in sampling frequency, temporal precision, and spatial reference system among multimodal data sources, the system incorporates a dedicated data synchronization and preprocessing module. This module features a high-precision timestamp alignment mechanism, a unified spatial coordinate algorithm, and a noise filtering process. It can uniformly encode image, point cloud, and sensor data in both temporal and spatial dimensions, effectively resolving temporal misalignment and spatial mismatch issues among multi-source data. This ensures that subsequent analyses are based on consistent time points and spatial locations for comparison. This processing chain significantly improves the scientific rigor and reproducibility of disease evolution analysis, guaranteeing the accuracy of fused features and the timeliness of trend judgments.

[0060] 3. This system utilizes a dedicated multimodal feature extraction module, employing targeted deep network structures to achieve high-dimensional feature encoding of image, point cloud, and sensor data. Specifically, this includes using 2D convolutional neural networks to extract image texture and edges, PointNet or 3D convolutional structures to extract point cloud geometric features, and recurrent or 1D convolutional networks to process time-series data of structural responses. Different modal features are embedded and unified into high-dimensional vectors, which are then input into the subsequent recognition model. This strategy improves the abstractness and semantic consistency of feature representation, providing a high-quality input foundation for subsequent disease type identification and size calculation.

[0061] 4. This scheme employs a multi-head cross-attention mechanism and a modal self-attention fusion structure to achieve deep complementarity and significant information enhancement between heterogeneous features. Combined with a feature pyramid network structure, it can also perform multi-level fusion representation of disease features at different scales, making it particularly suitable for the detection and identification of various structural diseases such as cracks, misalignments, and erosion. This fusion mechanism not only effectively suppresses the impact of redundant or interfering information on recognition accuracy but also adaptively highlights key disease features, thereby improving the system's recognition accuracy and robustness in complex tunnel environments and reducing false alarms and missed alarms.

[0062] 5. The system has a built-in risk assessment module that can perform precise parametric processing on identified defects. Crack length, width, and depth are derived jointly from images and point clouds; misalignment height is double-checked using cross-sectional elevation differences and displacement data; and the volume of detachment and the area of ​​water seepage can also be quantitatively characterized through depression area reconstruction and humidity gradient analysis. The defect size data output by this module provides a quantitative basis for structural safety assessment, trend modeling, and maintenance decisions, possessing strong engineering practicality. Furthermore, it integrates with the visualization module to visually display the location, type, and risk level of defects on a 3D model, enabling intelligent early warning functionality. Attached Figure Description

[0063] Figure 1 This is a structural block diagram of the multimodal fusion tunnel structure apparent defect identification and risk assessment system provided in Embodiment 1 of the present invention;

[0064] Figure 2 This is a data processing flowchart of the multimodal fusion tunnel structure apparent defect identification and risk assessment system provided in Embodiment 1 of the present invention;

[0065] Figure 3 This is a schematic diagram of the multimodal fusion model structure of the multimodal fusion tunnel structure apparent defect identification and risk assessment system provided in Embodiment 1 of the present invention;

[0066] Figure 4 The flowchart shows the multimodal data preprocessing and alignment process of the multimodal fusion tunnel structure apparent defect identification and risk assessment system provided in Embodiment 1 of the present invention.

[0067] Figure 5 This is a diagram of the multimodal feature extraction and fusion recognition network structure of the multimodal fusion tunnel structure apparent defect identification and risk assessment system provided in Embodiment 1 of the present invention.

[0068] Figure 6 This is a flowchart of the spatiotemporal alignment of multimodal data in a multimodal disease evolution monitoring and trend assessment system provided in Embodiment 2 of the present invention.

[0069] Figure 7This is a structural diagram of the disease evolution analysis module of a multimodal disease evolution monitoring and trend assessment system provided in Embodiment 2 of the present invention;

[0070] Figure 8 This is a logic diagram of trend identification and risk classification for a multimodal disease evolution monitoring and trend assessment system provided in Embodiment 2 of the present invention. Detailed Implementation

[0071] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0073] Terminology Explanation:

[0074] To facilitate understanding of the system described in this invention, several key terms appearing in the specification and claims are defined and explained. The content of this section does not constitute a limitation on the scope of the claims, but is used to clarify the meaning of terms, implementation methods, and parameter setting methods, thereby ensuring the clarity and implementability of this invention.

[0075] 1. Multimodal Feature Extraction Module. This term refers to the functional unit that extracts features from tunnel images, point cloud data, and structural sensor data. Preferably, this module includes:

[0076] Image feature subnetwork: For example, based on a two-dimensional convolutional neural network (ResNet-18, EfficientNet or improved UNet), the input is a tunnel inner wall image with a resolution of not less than 1920×1080, and the output is a 512-dimensional image feature vector;

[0077] Point cloud feature subnetwork: For example, based on PointNet++, 3D convolutional network (3D-CNN) or KPConv, the input is a 3D point cloud with no less than 50,000 points per meter, and the output is a 256-dimensional point cloud feature vector;

[0078] Sensor feature sub-network: For example, a combination structure based on one-dimensional convolutional network (1D-CNN) and recurrent neural network (LSTM / GRU), with inputs such as time-series signals such as strain, displacement, temperature and humidity, and outputs a 128-dimensional sensor feature vector.

[0079] The output feature vectors of the above three types of sub-networks are concatenated or weighted at a unified interface to form a high-dimensional disease representation input.

[0080] 2. Heterogeneous Feature Fusion and Disease Identification Module: This term refers to the functional unit that fuses and identifies the aforementioned multimodal features. An attention-weighted mechanism is preferred, and its mathematical expression is:

[0081] ;

[0082] in, Let i be the feature vector of the i-th mode. For its corresponding weight coefficient, satisfy In a specific implementation, this module may further include:

[0083] Multilayer perceptrons (MLPs) are used for unified dimension mapping;

[0084] Multi-head cross-attention layers or modal self-attention layers are used for dynamic weighting;

[0085] The feature pyramid network is used to fuse disease features at different scales; the output is a probability distribution for disease type identification and a vector for estimating geometric parameters.

[0086] 3. Risk Index

[0087] The risk index is a parameter used to quantify the severity of disease, defined as a weighted combination of crack geometry factor, misalignment height factor, detachment volume factor, and water stain humidity factor.

[0088] ;

[0089] in:

[0090] The crack geometry factor is calculated by normalizing the crack length, width, and depth.

[0091] The misalignment factor is obtained by combining the point cloud elevation difference and displacement sensor readings.

[0092] The shedding factor is obtained from the ratio of the indentation volume to the reference volume.

[0093] The water stain humidity factor is obtained by normalizing the seepage area and the rate of change of humidity.

[0094] Let be the weighting coefficient, satisfying .

[0095] Weighting coefficients can be obtained in two ways:

[0096] Data-driven approach: Based on labeled historical disease datasets, an optimization objective function that minimizes the false positive rate and false negative rate is used to determine the target.

[0097] Experience-based assignment method: When samples are lacking, domain experts can set values ​​based on engineering experience, such as 0.4 for cracks, 0.3 for misalignment, 0.2 for detachment, and 0.1 for water stains.

[0098] threshold The selection method is as follows:

[0099] The optimal classification threshold can be obtained by analyzing the ROC curve.

[0100] Alternatively, the allowable values ​​can be set with reference to existing standards such as the "Technical Specifications for Highway Tunnel Operation".

[0101] when When the system detects a risk level as an early warning, it classifies the risk level into "stable," "developing," and "deteriorating" based on the threshold range.

[0102] 4. Input-output relationship

[0103] The input and output relationships of each submodule in this invention are as follows:

[0104] Image subnetwork input: image frame, output: 512-dimensional vector;

[0105] Point cloud subnetwork input: point set, output: 256-dimensional vector;

[0106] Sensor subnetwork input: time series data; output: 128-dimensional vector.

[0107] The fusion module outputs: an 896-dimensional high-dimensional feature vector;

[0108] The risk assessment module outputs: a vector of geometric parameters of the disease, a risk index value, and a risk level label.

[0109] 5. Alternative solutions: Although this specification provides typical implementations, the invention is not limited to specific network or sensor models. For example:

[0110] The image feature subnetwork can be replaced by any two-dimensional convolutional neural network with equivalent feature extraction capabilities;

[0111] The point cloud feature subnetwork can be replaced with sparse convolution or other point cloud learning structures;

[0112] The sensor feature subnetwork can be replaced with a Transformer-based temporal modeling network.

[0113] The above substitutions do not change the basic principle of this invention.

[0114] Example 1:

[0115] like Figures 1-5 As shown, the multimodal fusion tunnel structure apparent defect identification and risk assessment system mainly includes:

[0116] The system consists of three parts: a multimodal data acquisition unit, a data processing and fusion unit, and a result display unit. The multimodal data acquisition unit includes visual perception equipment and structural monitoring sensors: a high-definition camera and a laser scanner are used to acquire images of the tunnel's internal structure and 3D point cloud data, respectively; sensors such as strain gauges, displacement gauges, thermometers, and hygrometers are placed in the tunnel lining structure and the environment to continuously collect parameters such as strain, displacement, temperature, and humidity of the tunnel structure.

[0117] The data processing and fusion unit typically consists of an industrial computer equipped with a high-performance GPU, running deep learning analysis software to process heterogeneous data from cameras, LiDAR, and various sensors in real time. The results display unit includes a visual interface on a monitoring computer or mobile terminal to present the analyzed damage information and provide alarm functions.

[0118] Specifically, in the implementation of this invention, the data acquisition and preprocessing module, to achieve multi-angle, multi-dimensional identification and high-precision geometric quantification of tunnel structural defects, constructs a multi-modal data acquisition framework covering the entire cross-section and taking into account both appearance and internal structure. This framework mainly includes two categories: visual perception devices (images and laser point clouds) and a structural monitoring sensor network (strain, displacement, temperature, humidity, etc.), and completes the fusion and preprocessing of multi-source data under a time synchronization and spatial registration mechanism.

[0119] In terms of visual information acquisition, high-definition camera arrays can be installed at fixed inspection points along the longitudinal direction according to the tunnel's geometric characteristics, or deployed on mobile carriers such as automated inspection vehicles and maintenance robots. The cameras use wide-angle industrial lenses combined with LED supplementary lighting components to cope with complex environmental conditions such as insufficient tunnel illumination and high background noise. The preferred acquisition resolution is greater than 1920×1080 pixels, with an image frame rate of 5–10 fps, ensuring clear imaging and continuous acquisition of defects such as surface cracks, seepage, and detachment even while in motion.

[0120] Three-dimensional laser scanning devices (such as rotating lidar or line laser + rotating mechanism) are used to perform 360° continuous scanning of the tunnel cross-section, generating high-density point cloud data. In a typical configuration, the laser point cloud sampling frequency is no less than 10 Hz, the horizontal angular resolution is no less than 0.2°, and the point cloud spatial density is controlled at no less than 50,000 points per meter of tunnel length. During the mobile inspection platform's data acquisition process, point cloud trajectory positioning is completed through a synchronous inertial measurement unit (IMU) and wheel odometers, and global registration is performed using the SLAM algorithm to construct a unified spatial coordinate system. In a fixed installation mode, each laser device needs to be calibrated with a unified map through laser extrinsic parameter calibration.

[0121] For structural condition monitoring, the sensor network preferably employs fiber optic or resistive strain gauges, LVDT / eddy current displacement sensors, and integrated temperature / humidity acquisition modules. Sensing nodes are deployed along the tunnel lining at key locations (such as joints, boundaries, and vulnerable areas), with a sampling frequency range of 0.1–10 Hz. Transmission methods include RS485, CAN bus, or LoRa / NB-IoT wireless communication. All acquisition modules achieve timestamp consistency through a clock synchronization module (such as NTP or GPS), and data is uniformly accessed by the edge computing unit or uploaded to the cloud processing center.

[0122] The system first performs unified spatiotemporal alignment processing on the aforementioned multi-source data. Specifically, this includes:

[0123] Time synchronization mechanism: Based on a unified timestamp system (such as UNIX time), combined with high-precision time synchronization signals (GPS, Beidou, IEEE1588, etc.) or the system's internal master-slave clock alignment mechanism, to ensure that image frames, laser frames and sensor sampling data are collected at the same physical moment;

[0124] Spatial registration mechanism: The image and point cloud are mapped through camera-radar extrinsic parameter calibration (using planar target or corner point calibration methods), and the point cloud coordinates are transformed to the image viewpoint to achieve pixel-level alignment; the sensor spatial deployment information and laser map are spatially calibrated to form a mapping table of structural attributes and geometric models, so as to achieve accurate correspondence between structural response data and geometric entities.

[0125] During the data preprocessing stage, the system performs the following operations on the raw data of each modality:

[0126] 1) Image processing:

[0127] Perform distortion correction (based on camera intrinsic model), Gamma adjustment, Retinex enhancement, and median filtering for noise reduction;

[0128] Local contrast enhancement and multi-scale edge extraction are used to pre-enhance disease characteristics such as cracks, edge peeling, and water stains;

[0129] If there is lens shake or lighting drift, the system will perform image stabilization through motion compensation and tone matching modules.

[0130] 2) Point cloud processing:

[0131] Outlier removal (based on statistical filters) and voxel grid filtering are performed on the original point cloud.

[0132] Perform coordinate transformation to uniformly convert to the standard tunnel center coordinate system;

[0133] Pre-extract the structural section outline and rough abnormal areas (such as depressions, misalignments, etc.) in the slice view.

[0134] 3) Sensor data processing:

[0135] Low-pass filtering (Butterworth, wavelet denoising), zero-point drift correction, and moving average are applied to time-series signals.

[0136] Normalization transformation is performed to unify the range of strain and displacement to the interval [-1,1].

[0137] Temperature and humidity parameters are introduced as structural state correction factors (for environmental compensation, to suppress false strain judgments caused by temperature differences).

[0138] Furthermore, to further enhance the fusion effect, the system registers the image and point cloud within the same map to generate a joint view tensor, and interpolates it using structural parameters to create a structure-visual alignment tensor, which is then used as the multimodal feature input for the neural network model. This joint input constitutes the key data foundation for the disease identification and quantification model.

[0139] The completeness of the above data acquisition and preprocessing process ensures the accuracy, robustness, and real-time performance of subsequent feature extraction, fusion analysis, and identification quantification processes, which is a prerequisite for realizing the multimodal fusion capability of this invention.

[0140] Specifically, regarding multimodal feature extraction, to achieve intelligent identification and geometric dimensionality quantification of various types of defects in tunnel structures, this invention constructs a deep learning feature extraction subsystem oriented towards multimodal heterogeneous inputs within the data processing module. This system sets up three types of feature extraction channels—image sub-network, point cloud sub-network, and sensor temporal sequence sub-network—based on the dimensional structure, information attributes, and temporal / spatial distribution characteristics of different modal data. It outputs structured, high-dimensional defect representation vectors through a unified interface, providing fundamental support for subsequent multimodal fusion and identification decisions.

[0141] 1) Visual modality feature extraction

[0142] The visual modality mainly includes RGB image data and laser point cloud data of the tunnel surface, which are used to characterize the surface texture (cracks, water stains, erosion) and geometric deformation (depression, misalignment, defects), respectively.

[0143] ① ImageNet Subnetwork Design: Image data input format is A lightweight, multi-scale two-dimensional convolutional neural network (such as ResNet-18, EfficientNet, or an improved version of UNet) is used to extract low-texture disease regions, edge morphological features, and structurally degraded regions from images. The specific structure includes:

[0144] The initial convolutional layer (Conv + ReLU + BN) is used to extract edge textures;

[0145] Multi-scale residual blocks (or depthwise separable convolutions) extract local-global structure;

[0146] Dilated convolutions or attention enhancement modules (such as SE-Block / CBAM) highlight areas such as long cracks and discolored water stains; outputting high-dimensional feature maps. Preserve the spatial structure.

[0147] ② Point cloud sub-network (PointNet / 3D-CNN) design: The point cloud data format is... This includes coordinates and reflection intensity values. To improve processing efficiency, this invention provides two paths:

[0148] Path A (Projection Method): The point cloud is mapped by spherical / cylindrical unfolding or slice depth map projection, and converted into a pseudo-image form. Input the data into a 2D CNN network, aligned with the image channels;

[0149] Path B (Original Point Processing): Using PointNet++, KPConv or Sparse 3D CNN, local geometric feature extraction and global pooling are performed directly on the point set to extract three-dimensional geometric anomalies such as misalignment and peeling.

[0150] Output feature tensor ,in The number of points or blocks.

[0151] Through the two visual channels mentioned above, the system can extract multi-level perceptual information such as changes in structural geometric boundaries, surface texture degradation features, and deep concave and convex abnormal regions.

[0152] 2) Structural sensing modal feature extraction

[0153] Structural sensing modes mainly include strain Displacement ,temperature ,humidity Continuous time-series signals reflect changes in structural operating status and the evolution trend of potential defects.

[0154] ① SensorNet temporal subnetwork design:

[0155] Each type of sensor data is represented in time series as follows: The system employs a one-dimensional convolutional neural network (1D-CNN) and a long short-term memory network (LSTM) in parallel to extract its frequency and temporal features:

[0156] The 1D-CNN module is used to extract periodic fluctuations (such as vibrations), abrupt changes (such as instantaneous cracks), and gradual trends (such as subsidence).

[0157] The LSTM / GRU module extracts cross-time-period dependencies and predicts the evolution trend of structural states;

[0158] Multi-channel sensor data can be nested and fused, and the weights between modes can be adjusted through an attention mechanism;

[0159] Final output sensor feature vector Preserve time dependencies and indexes of abnormal event locations.

[0160] ② Multi-sensor collaborative feature enhancement strategy:

[0161] To address the problem of combined strain-temperature variation, a system is constructed. Temporal duality feature pair;

[0162] For events involving sudden stress changes and increased humidity, the system introduces a composite index to identify the "thermal stress-induced microcrack" mode.

[0163] Consistency between spatial location and visual perception modality is one of the conditions for fusion.

[0164] 3) Feature extraction interface and output specifications

[0165] The feature tensors output by all subnetworks uniformly adopt a multidimensional representation structure. and includes the following attributes:

[0166] Coordinate binding: All features are bound to spatial location indexes (image pixels, point cloud indexes, sensor deployment IDs);

[0167] Time tags: Image frames, point cloud frames, and sensor data within a time window are fused to form a sliding window multimodal feature block;

[0168] Modality identification: Each feature class is accompanied by a modality label and a confidence index, which are used for weighting in subsequent fusion strategies;

[0169] Output formats: Images / point clouds are in tensor format, and sensors are in vector / sequence format. All support parallel processing and asynchronous fusion.

[0170] The aforementioned multimodal feature extraction subnetwork is deployed on a neural network platform (such as PyTorch or TensorFlow) and supports GPU-accelerated inference. Existing tunnel defect detection models can be loaded through pre-training and transfer learning, and fine-tuned and optimized on target tunnel field data to further improve the model's adaptability and robustness.

[0171] Specifically, in this invention, a heterogeneous data fusion and defect identification module is constructed to achieve high-precision identification and geometric parameter quantification of various types of defects in tunnel structures, such as cracks, misalignments, spalling, and water stains. This module, as the core component of the entire identification system, integrates multimodal sensing information, identifies defect types, and quantifies defect parameters. It is suitable for structural health monitoring and intelligent diagnosis of various tunnel structures, including subways, highways, and water conservancy projects.

[0172] The heterogeneous feature fusion and recognition module includes the following main parts:

[0173] 1) Heterogeneous Feature Fusion Network

[0174] This module first fuses the different modal features output from the image subnetwork, point cloud subnetwork, and structure sensor subnetwork. Let the image features be... Point cloud features are The sensor features are The goal of the fusion module is to construct a unified fusion feature vector. ,in This represents the total feature dimension after fusion.

[0175] The converged network includes:

[0176] The Multilayer Perceptron (MLP) architecture is used to perform dimensionality mapping and unified encoding of features from different modalities, and is represented as:

[0177] ;

[0178] The multimodal attention mechanism module assigns dynamic weights to features of each modality. Weighted fusion is achieved, represented as:

[0179] ;

[0180] The multi-scale feature fusion module adopts a feature pyramid structure. Upsampling and fusion of features at different resolutions can improve the system's adaptability to defects at different scales (such as fine cracks and large-area water stains).

[0181] The modal confidence masking module generates a confidence tensor based on the noise, residuals, and consistency of each modal feature. And participate in the weighted control of the fusion process.

[0182] Through the above structure, the system can construct a unified representation tensor of image, point cloud, and structural response features in a high-dimensional fusion space. It possesses characteristics such as spatial alignment, anomaly sensitivity, and scale adaptability, providing a solid feature foundation for subsequent disease identification.

[0183] 2) Disease identification

[0184] This module is used to receive the fused multimodal feature representation, complete the identification of disease types and the calculation of structural parameters, specifically including:

[0185] Disease classification unit: set fusion characteristics After processing by a fully connected network (FC) and a Softmax function, the output is a disease probability vector. ,Right now:

[0186] ;

[0187] in Number of disease categories Indicates that the input belongs to the first... The probability of such defects (e.g., cracks, misalignment, etc.).

[0188] Geometric parameter extraction unit: For each identified defect, the system calls the matching quantization model and outputs a geometric parameter vector. For example, the length of the crack Average width wait:

[0189] ;

[0190] Among them, typical disease parameters may include:

[0191] Crack type: ,

[0192] Mismatched platforms: ,

[0193] Shedding type: ,

[0194] Water stains: ,in For the area of ​​water seepage, This is an estimated value for the seepage rate per unit time.

[0195] The identification and parameter results will be combined to form a structural diagnostic result data package. It is also bound to the structural spatial location for platform scheduling and visualization.

[0196] It is worth noting that this module supports a fusion-assisted diagnostic mechanism: if the image modal information is not obvious, but the sensor data shows an obvious abnormal trend (such as a sharp increase in strain), the system will trigger a modal guidance mechanism. Even if the visual evidence is weak, it can still output a "potential disease" label based on the sensor-driven fusion inference result, thereby improving the early warning capability and the overall detection rate.

[0197] Specifically, the defect geometry calculation module: In this invention system, to achieve quantitative expression of defect identification results and engineering assessability of structural damage, a defect geometry calculation module is further set up. This module, based on the identification output of a multimodal fusion model, combines image, point cloud, and structural sensor data to achieve accurate extraction of defect morphology and multidimensional parametric modeling. Its core functions include crack geometry quantification, misalignment height determination, detachment area volume estimation, water seepage range and dynamic analysis, forming a complete defect geometry diagnostic dataset.

[0198] 1) Crack size identification and three-dimensional geometric estimation

[0199] For surface cracks in tunnel lining, the system first acquires the crack pixel region based on image data, then maps the image coordinate system to physical space using camera calibration parameters, and estimates the two-dimensional length of the crack. and width The calculation formula is as follows:

[0200] ;

[0201] in, This represents the number of pixels along the main axis of the crack. The average horizontal width in pixels. Pixel resolution (obtained from calibration, unit: mm / pixel).

[0202] To further obtain the morphological information of the crack in three-dimensional space, the system registers the crack region extracted from the image with the coordinates of the laser point cloud, and extracts the spatial coordinate difference between the point clouds on both sides of the crack:

[0203] ;

[0204] in, This represents the displacement difference along the vertical crack depth direction. This represents the estimated width of the opening along the radial direction of the tunnel. By combining image and point cloud data, the system can establish a three-dimensional geometric profile model of the crack.

[0205] If the system is equipped with permeability detection sensors such as ultrasonic waves and ground-penetrating radar, it can also obtain the depth of the cracks. This is used to assess whether cracks penetrate the lining layer. The current embodiment primarily models based on apparent images and strain anomalies, but it has the capability to be expanded to include other sensor sources.

[0206] 2) Measurement of misalignment height and displacement verification

[0207] For misalignment defects appearing at the splicing joints of lining rings, the system automatically extracts the average elevation on both sides of the fracture surface based on the elevation abrupt change characteristics of the laser point cloud within the cross-section of the misalignment:

[0208] ;

[0209] in, and The first For the elevation values ​​of adjacent point cloud blocks on both sides of the misalignment, To calculate the logarithm.

[0210] To improve accuracy, the system synchronously incorporates readings from nearby structural displacement sensors. Calculate the residual error between the two:

[0211] ;

[0212] If the residual exceeds the set threshold, the system will activate the anomaly verification mechanism, prompting the user to manually review the error to ensure millimeter-level accuracy in the misalignment estimation.

[0213] 3) Estimation of the area and volume of the lining detachment zone

[0214] For surface defects such as concrete spalling, peeling, or honeycomb pitting in tunnel lining, this system achieves geometric quantification through point cloud concave area boundary extraction and three-dimensional reconstruction.

[0215] First, the surface normal deviation is used to extract the depression boundary. ;

[0216] Calculate the area of ​​the depression region It can be estimated from the projected area of ​​the point cloud density:

[0217] ;

[0218] in, For the first Area of ​​each triangular unit This represents the number of triangular mesh units.

[0219] Volume estimation is achieved by constructing the depression depth of the concave region. Integrating with respect to the reference plane yields:

[0220] ;

[0221] Image data can also be used to enhance boundary detection and help determine whether there are non-geometric peeling areas (such as coating peeling).

[0222] 4) Quantitative and dynamic assessment of water stain leakage

[0223] For water stains or signs of leakage on structural surfaces, the system uses image semantic segmentation technology to identify areas of water seepage. Combined with pixel resolution Its area is estimated as follows:

[0224] ;

[0225] At the same time, the system collects readings from the humidity sensor in this area. Calculate the time gradient:

[0226] ;

[0227] If the humidity gradient increases significantly, combined with the growth rate of the water stain expansion boundary in the image... This allows for a comprehensive assessment of the activity and persistence of the seepage point.

[0228] 5) Output structure and diagnostics are linked

[0229] The geometric parameters of the above-mentioned diseases are uniformly encoded into parameter vectors by the system:

[0230] ;

[0231] It is then bound to the spatial location information and structural number of the disease identification output to form a diagnostic result entry. ,in Disease category To identify confidence levels.

[0232] The calculation and diagnostic binding of all defect geometric parameters are automatically completed in the data processing module, and the results are simultaneously visualized and output to the front-end display interface or uploaded to the central platform for defect maintenance scheduling and risk classification assessment. Through the above geometric quantification mechanism, the system realizes an integrated tunnel defect detection process of "identification-quantification-location", which significantly improves diagnostic efficiency and the scientific nature of structural maintenance.

[0233] Preferably, the risk assessment module is connected to the front-end industrial camera / depth camera, laser scanner, and sensors such as displacement and humidity. It receives time-synchronized images, point clouds, and environmental time-series data. The structural damage characteristics are calculated by the crack geometry estimation submodule, the misalignment height analysis submodule, the detachment area measurement submodule, and the water stain area and humidity change analysis submodule, respectively. In this embodiment, the risk index is formed by comprehensive weighting according to the weight coefficients, and the risk level is output according to the set threshold range.

[0234] In this embodiment, the data acquisition and alignment process includes: performing camera calibration and distortion correction on the image, establishing a pixel-to-actual-scale conversion relationship; performing coordinate unification and noise reduction filtering on the point cloud; and aligning the sampling sequences of the displacement and humidity sensors using a sliding time window. After alignment is completed, the four sub-modules run in parallel and output the corresponding feature factors.

[0235] 1) Crack geometry estimation submodule

[0236] The crack geometry estimation submodule is used to calculate the crack length, width, and depth by combining image scale and point cloud depth, and to form crack feature factors. Specifically, it includes:

[0237] a) Perform crack segmentation and thinning on the calibrated image, and extract the crack centerline; take normal profiles at fixed arc length intervals along the centerline direction, measure the crack boundary spacing, and calculate the average width. The crack length is obtained by multiplying the centerline arc length by the pixel scale. .

[0238] b) Project the crack centerline onto the point cloud coordinate system. Based on a well-fitted surface reference plane in the crack neighborhood, calculate the average negative offset of the crack bottom point from this reference plane to obtain the crack depth. .

[0239] c) Based on the design-permitted reference limits Calculate the crack characteristic factors:

[0240] ;

[0241] 2) Misalignment Height Analysis Submodule

[0242] The misalignment height analysis submodule is used to extract the elevation difference between point clouds of adjacent sections and verify it by combining it with displacement sensor readings to form misalignment characteristic factors. Specifically, it includes: a) Extracting point clouds of adjacent sections on both sides of the structural joint and completing registration under the same reference system; obtaining elevation difference statistics through the normal projection of the corresponding point sets to obtain... b) Read the displacement sensor readings at the corresponding positions. Normalize and combine the two, then calculate:

[0243] ;

[0244] in For design allowances or historical calibration reference values.

[0245] 3) Detachment Area Measurement Submodule

[0246] The detachment area measurement submodule is used to calculate the area and volume based on the boundary of the point cloud-based concave region, forming a detachment volume factor. Specifically, it includes: a) performing local surface fitting on the point cloud of the target component surface and calculating the residual field, then thresholding to obtain candidate concave regions and their boundaries; b) within the concave region, integrating the depth field with the fitted reference surface as the zero point to obtain the detachment volume. Simultaneously calculate its projected area; c) Using the reference volume Normalization yields:

[0247] ;

[0248] 4) Submodule for analyzing water stain area and humidity changes

[0249] The water stain area and humidity change analysis submodule is used to fuse the water seepage area and humidity temporal gradient from image segmentation to estimate water seepage risk, forming a water stain humidity factor. Specifically, it includes:

[0250] a) Segment the water stain regions in the image and obtain the seepage area based on camera calibration. ;

[0251] b) Humidity time series The slope is fitted within a preset time window to obtain the humidity change rate. ;

[0252] c) Using reference area Compared with reference rate of change Normalization, calculation:

[0253] ;

[0254] 5) Risk index calculation and risk level classification

[0255] In this embodiment, the feature factors output by the above four sub-modules are combined and weighted to obtain the risk index:

[0256] ;

[0257] in, These are the weighting coefficients for each damage feature, which can be obtained based on field experience or from training data.

[0258] When the risk index Exceeding the set threshold When this occurs, the system automatically determines it to be in a warning state. This embodiment sets the risk level range as follows: Based on this, three risk levels are output:

[0259] when Determined as "stable"

[0260] when Classified as "developing"

[0261] when It was determined to be "deteriorating".

[0262] If a single threshold strategy is used, then when It enters a warning state at that time.

[0263] 6) Parameter and threshold setting methods

[0264] Reference Limits Weighting coefficients can be selected from design allowable values, relevant specifications, or historical statistical distributions of similar structures; It can be obtained through training on a labeled sample set or determined by expert weighting; the threshold range can be searched through the validation set to achieve the expected balance between false alarm rate and false alarm rate.

[0265] Specifically, the results output and visualization module,

[0266] In this embodiment of the invention, after identifying tunnel structural defects and quantifying geometric parameters, the system further includes a result output and visualization module. This module presents the identification results to maintenance personnel in a structured and intuitive manner, and provides real-time alarm and data retention functions. This module can be deployed on the visualization platform of the ground control center, the maintenance cloud terminal, or integrated into the embedded human-machine interface of rail transit inspection vehicles, making it suitable for engineering maintenance applications across multiple scenarios and platforms.

[0267] The results output and visualization module mainly includes the following functional components:

[0268] 1) 3D structural model annotation and visual interactive presentation

[0269] The system receives information on the type, spatial location, and corresponding geometric dimensions of the tunnel from the defect identification and quantification module, and overlays this information graphically onto a digital model of the tunnel structure. Preferably, the digital model can be a 3D point cloud model of the tunnel, a vectorized structural diagram, or an unfolded 2D linear schematic diagram, which maintenance personnel can switch between as needed.

[0270] For structural locations with defects, the system highlights them using methods such as highlighting, flashing borders, and thermal color coding, automatically generating label information. The label content includes the defect type, structural number, spatial location information, and key dimensional parameters. For example:

[0271] "A longitudinal crack appeared in the upper left part of the Xth ring, with a length of 5.3 meters and an average width of 0.2 millimeters."

[0272] "Misalignment occurred at the joint between the Y and Z ring pieces, with a height difference of 3.5 mm";

[0273] "The Nth section of concrete has detached, with an area of ​​approximately 0.12 square meters and an estimated volume of 1.6 liters."

[0274] "Water seepage was found near the M measuring point, and the humidity slope continued to rise, which is suspected to be a leak."

[0275] The labels can be displayed / hidden, repositioned, or linked to historical data details pages based on user actions. The interface supports 3D model rotation, scaling, and cross-sectional viewing, facilitating a comprehensive assessment of structural health.

[0276] 2) Sensor monitoring data curve display and dynamic alarm mechanism

[0277] For the various structural sensor data (including but not limited to strain, displacement, humidity, temperature, etc.) collected in the embodiments, the system displays and analyzes trends in real time using graphical methods. Preferably, multi-channel charts are set up, supporting single-channel / multi-channel switching display and filtering by structural location or sensor type.

[0278] The system monitors and judges sensor signals in real time based on preset parameter thresholds. If a signal in a certain channel exceeds the warning threshold or shows a sudden change (such as a sudden increase in strain), the system automatically triggers an alarm mechanism, including but not limited to:

[0279] The audible and visual alarm device is activated;

[0280] The inspection vehicle or central control platform will issue a voice prompt.

[0281] The maintenance personnel will receive push notifications on their mobile devices, including location information and preliminary diagnostic suggestions.

[0282] Meanwhile, the system can record the alarm trigger time and sensor status, and automatically save them as alarm event logs for easy subsequent querying and handling feedback.

[0283] 3) Data storage and trend analysis support

[0284] The system of this invention is further provided with a data archiving and analysis interface module, which is used to uniformly store all identified disease results (including metadata such as category, size, location, and time), raw and processed sensor data, and user operation records into the structural health database.

[0285] The system supports the following functions:

[0286] Search for historical disease distribution by tunnel section or time period;

[0287] Compare the trends of the evolution trajectories of the same disease location;

[0288] Generate structural diagnostic cycle statistical reports to assist in maintenance decision-making;

[0289] It interfaces with BIM models or asset management systems to provide a traceable, trackable, and updatable data chain.

[0290] Through the above-mentioned output and visualization mechanisms, this invention not only realizes the intelligent display of disease identification results, but also supports the digital management and maintenance of tunnel structures throughout their entire life cycle through dynamic alarms and data closed loops, significantly improving the efficiency of disease treatment and the level of operation and maintenance response.

[0291] Regarding system implementation and application methods, in order to adapt to the operation and maintenance needs of different types of tunnel structures, this invention provides a variety of system deployment and application implementation methods, which have high flexibility and scalability and are suitable for structural defect detection tasks in a variety of complex environments;

[0292] 1) Mobile Inspection Deployment Plan

[0293] In a preferred embodiment, the system of the present invention is deployed using a vehicle-mounted mobile inspection unit. Specifically, the system integrates image acquisition devices (such as high-definition industrial cameras), laser point cloud scanners (such as lidar), and various types of structural sensors (such as strain, displacement, and temperature and humidity sensors) onto a tunnel inspection vehicle, and coordinates their operation through a unified power supply and a synchronized clock system.

[0294] The inspection vehicle is equipped with a high-performance industrial control computer, which embeds the multimodal data processing and fusion recognition software module of this invention. This enables a closed-loop process of simultaneously collecting, processing, and outputting detection results for the tunnel structure while it is in motion. In actual use, the system can stably complete various functions such as high-frequency data acquisition, fusion feature extraction, defect type identification, risk assessment, result presentation, and alarm prompts while maintaining an appropriate operating speed for the inspection vehicle (e.g., 10–40 km / h). It exhibits excellent real-time performance and industrial-grade reliability.

[0295] This method is suitable for routine inspection tasks, especially for environments where vehicles can pass through, such as subway lines and highway tunnels, enabling rapid and comprehensive inspection of long-distance tunnel structures in a short period of time.

[0296] 2) Fixed Online Monitoring Deployment Solution

[0297] In another typical embodiment, the system of the present invention can be deployed in the tunnel using fixed monitoring units to achieve all-weather, long-term disease monitoring. In this method, the system installs image acquisition modules and multimodal sensor nodes at preset intervals (such as every 50 meters or every section) at the tunnel lining structure or segment interfaces to form a monitoring network covering the entire tunnel.

[0298] All acquisition units are connected to the central control server via wired Ethernet, industrial wireless networks (such as Wi-Fi, ZigBee, or 5G), etc. The server integrates and deploys the data processing and identification software system of this invention to achieve remote data aggregation and automatic analysis. The server can also perform unified configuration management, task scheduling, and data synchronization for edge devices, forming a hybrid architecture that combines centralized management with edge distributed sensing.

[0299] This fixed deployment scheme enables continuous health monitoring of tunnel structures, and is especially suitable for important passages (such as extra-long subway sections, hydraulic tunnels, etc.) or environments where frequent access is inconvenient. It has strong continuity, stability and timely alarm.

[0300] 3) Multi-scenario adaptation and system expansion capabilities

[0301] The system of this invention has good scalability at both the hardware and software levels, and can flexibly configure sensing modalities and diagnostic strategies according to specific application scenarios:

[0302] For urban rail transit tunnels, the system focuses on identifying typical defects such as lining cracks, segment misalignment, and water leakage, and supports association with shield tunnel segment numbering information to form a structural health record.

[0303] For highway tunnels, based on structural defect detection, the system can integrate vibration response data caused by traffic loads and compare it with synchronous information of vehicle-induced loads to help assess structural fatigue risk and the causal relationship of defect triggering.

[0304] For water conservancy tunnels or water diversion channels, the system is equipped with high-precision humidity, temperature and displacement sensors, combined with point cloud depression detection, which is suitable for identifying typical hydraulic structure defects such as lining plate detachment, water stain seepage, deformation and displacement.

[0305] 4) System performance verification

[0306] Field deployment and comparative experiments demonstrate that the system of this invention can operate stably in complex tunnel environments, continuously collect and fuse multi-source data, and accurately complete the tasks of defect identification and geometric quantification. Compared with traditional manual inspection results, this system exhibits good technical consistency and a low error range in terms of defect identification rate, measurement accuracy, and response speed. Especially in the identification of early defects, micro-cracks, or edge areas, due to the fusion of abnormal parameter signals from within the sensors, the system's detection performance is superior to single-image or manual methods, demonstrating significant value for engineering application.

[0307] Example 2:

[0308] refer to Figures 6-8 This invention provides a multimodal disease evolution monitoring and trend assessment system. The system possesses functions such as disease identification, long-term evolution tracking, trend analysis, and risk classification, and is used for intelligent operation and maintenance and early warning decision-making for tunnel structures. It includes a multi-source data acquisition module, a multi-time-series data archiving and alignment module, a disease evolution analysis module, a trend identification and risk classification module, and a results output module.

[0309] Specifically, the multi-source data acquisition module is suitable for long-term monitoring and disease evolution perception of tunnel structures. It consists of three types of sensing devices: image acquisition unit, laser point cloud acquisition unit, and structural sensor node, forming a multimodal sensing array covering the inner surface and internal structural state of the tunnel.

[0310] Furthermore, the image acquisition unit includes several high-definition industrial cameras installed between the tunnel lining ring segments. The spacing between the cameras along the longitudinal direction of the tunnel is preferably 30 to 60 meters, and in one specific embodiment, the spacing is 50 meters. This image acquisition unit is used to periodically acquire surface image information of the tunnel lining inner wall to capture visually identifiable defects such as cracks, water stains, and concrete spalling. The cameras have a resolution of no less than 1920×1080 pixels, support autofocus and low-light enhancement, and the acquisition frequency can be set according to a schedule (e.g., once daily) or triggered by events (e.g., immediate acquisition when a sensor malfunctions), to meet the dual needs of inspection and online diagnosis.

[0311] The laser point cloud acquisition unit is configured on the tunnel maintenance passage or inspection platform, preferably using a 360° rotating laser scanner with high-speed multi-line laser emission capability. It is used to acquire three-dimensional point clouds of the tunnel cross-section and internal space at a set cycle (e.g., once a week). The acquired data has a spatial resolution of no more than 5 mm, sufficient to reconstruct the three-dimensional structural contour features, including tunnel misalignment, erosion, spalling, and linear deformation. This point cloud data is acquired synchronously with image data and managed uniformly within the system.

[0312] The structural sensor nodes are installed at critical stress locations and vulnerable parts of the tunnel structure, including but not limited to circumferential joints, longitudinal joints, rebar anchorage zones, and areas of structural abrupt changes. The sensor nodes include:

[0313] Strain sensors are used to measure the strain response of lining structures;

[0314] Displacement sensors are used to monitor the cumulative displacement during localized faulting and cracking processes.

[0315] Temperature and humidity sensors are used to identify the risk of water damage caused by factors such as water seepage and changes in environmental humidity.

[0316] All of the aforementioned sensors have sampling periods as short as one hour or even minutes, and all support local caching and remote data transmission functions. They can be connected to the system data processing center via Ethernet or wireless communication protocols (such as Wi-Fi, LoRa, NB-IoT, etc.).

[0317] To ensure the accuracy of data fusion analysis, the image, point cloud, and sensor data are all bound with precise timestamps and location information tags during acquisition, supporting subsequent spatiotemporal alignment and sequence management. Preferably, the system is configured with a global clock synchronization module (such as a GPS time source or a high-precision Network Time Protocol (NTP) server) to ensure the consistency of multimodal heterogeneous data in the time domain.

[0318] Through the coordinated deployment and parallel operation of the aforementioned multi-source sensing modules, the system of this invention can achieve continuous sensing and multimodal data acquisition of tunnel structures in both time and space dimensions, providing high-quality input data support for subsequent analysis of disease evolution trends and risk level classification.

[0319] Specifically, the multi-temporal data archiving and alignment module is used to perform unified spatiotemporal alignment processing on tunnel structure image data, point cloud data and structural response data collected by multi-source sensing devices at different times, and to build a temporal fusion dataset with comparative analysis capabilities to support the subsequent extraction of disease evolution trends and risk assessment processes.

[0320] Furthermore, the multi-time-series data archiving and alignment module includes:

[0321] The image registration and archiving submodule is used to perform temporal archiving and geometric alignment of tunnel images acquired at different times from the same monitoring point. Preferably, image registration algorithms based on Scale Invariant Feature Transform (SIFT), Accelerated Robust Feature Transform (SURF), or feature point matching and optical flow analysis are employed to achieve accurate registration even when image size, angle, or brightness changes, ensuring the consistency of the location of the affected area in images from different times. Aligned images are stored in chronological order of acquisition, labeled with the acquisition timestamp, camera device ID, and physical location index.

[0322] The point cloud model alignment submodule is used to reconstruct and spatially register the 3D point cloud of the tunnel across multiple observation periods. Specifically, it involves filtering, denoising, and voxelizing the original point cloud data, followed by using an ICP (Iterative Closest Point) algorithm or FPFH+RANSAC feature matching method to unify the point cloud models from multiple time points to the same spatial reference frame, enabling accurate tracking and analysis of structural morphology changes on the same tunnel cross-section.

[0323] The structural sensing data synchronization submodule is used to perform unified timeline registration of the time series of structural response parameters such as strain, displacement, and humidity with image and point cloud data. The sensor data is indexed by sensor ID, installation location coordinates, and acquisition timestamp, and supports spatial association with image / point cloud records at corresponding times, thereby constructing a complete "multimodal + time series" structural response record.

[0324] To achieve integrated management of the aforementioned data, this module preferably employs a structured time-series data storage framework. It establishes a unified data indexing system using tunnel structural units (such as ring segment numbers and cross-sectional coordinates) as the organizational unit. Each structural unit's data record item contains multiple image sequences, multiple point cloud models, and structural response data within the corresponding time window, and provides interfaces for subsequent modules to call.

[0325] After processing by this module, the system can form a multimodal fusion dataset based on "tunnel structural units + time series", which has the ability to compare the spatiotemporal evolution of diseases. It provides temporal feature support for the disease evolution trend extraction module in the system of this invention, and effectively supports the dynamic tracking and development judgment of disease processes such as crack expansion, misalignment aggravation, and erosion deepening.

[0326] Specifically, the disease evolution analysis module is used to mine the temporal evolution characteristics of diseased areas in tunnel structures based on archived and aligned temporal multimodal data, extracting indicators such as morphological expansion trends, structural response change trends, and evolution rates to support subsequent risk trend judgment and classification. This module includes three functional units: an image evolution tracking submodule, a point cloud deformation comparison submodule, and a sensor trend fitting submodule.

[0327] Furthermore, the image evolution tracking submodule:

[0328] This submodule is used to perform defect boundary tracking and extended analysis on image sequences acquired at the same location and at different times within the target tunnel structure. Specifically, firstly, using the multi-temporal image registration algorithm described above, the aligned multi-period images are compared, and a deep learning-based semantic segmentation network, such as U-Net, DeepLab, or Transformer structural segmentation model, is applied to the pre-labeled defect areas (such as cracks, water stains, etc.) to extract the boundary contours of the defect areas.

[0329] By analyzing the spatial boundary changes in the segmentation results, the expansion trend of the disease over time can be calculated, such as the change sequence of crack length and width. In a specific embodiment, the crack in a tunnel lining was measured to be 2.1 meters long in the initial image and expanded to 3.4 meters in the third image. Combining the acquisition period, the average crack growth rate can be calculated to be 0.65 meters / month. The change curve generated in this process will serve as an important basis for subsequent trend judgment.

[0330] Further, the point cloud deformation comparison submodule:

[0331] This submodule is used to perform spatial quantitative analysis of the deformation evolution process of the structural surface based on a 3D point cloud model of the tunnel cross section or structural surface. By performing voxel reconstruction, spatial alignment and differential analysis on point cloud data acquired at multiple time points, the three-dimensional evolution process of geometric defects such as ring misalignment and concrete spalling can be identified.

[0332] Preferably, the system employs RANSAC registration and multi-scale difference algorithms based on FPFH features to automatically detect local elevation abrupt changes, volumetric depressions, or convex regions in the point cloud model. For example, in a certain monitoring period, a 6.3mm increase in the height of the point cloud model at the Z-segment joint was detected, which the system determined to be a structural misalignment phenomenon. Simultaneously, in another lining surface point cloud model, the volume of the depression region increased from 2.7 liters to 8.2 liters, indicating a continuous expansion of the concrete spalling area and volume. The geometric change parameters output by this submodule provide support for the quantification of structural defects.

[0333] Furthermore, the sensor trend fitting submodule:

[0334] This submodule is used for dynamic fitting and trend prediction of multi-time response data collected by structural sensors (strain gauges, displacement gauges, hygrometers, etc.). The system extracts time series segments of the target sensor based on a sliding window approach and uses algorithms such as one-dimensional convolutional neural networks (1D-CNN), multinomial fitting, exponential regression, or recurrent neural networks (RNN, LSTM) to model the changing trends of the sensor data.

[0335] In a typical embodiment, system analysis of strain gauge data over 48 hours revealed a sharp increase in strain value from 0.006 to 0.012, exhibiting a clear acceleration characteristic after curve fitting. The model automatically triggered an early warning mechanism for disease development. Similarly, if the humidity sensor curve shows a periodic increase, the system can preliminarily determine that the area has a long-term leakage trend and recommend switching to high-frequency monitoring.

[0336] The three sub-modules mentioned above can work together to integrate structural appearance image features, spatial geometric point cloud features, and structural response time-series signals to form a multi-dimensional disease evolution map, which significantly improves the accuracy and comprehensiveness of disease evolution modeling and constitutes a key pre-support link for the trend recognition module in the system of this invention.

[0337] Specifically, the trend identification and risk classification module is used to conduct multi-factor comprehensive analysis of the morphological expansion speed of the diseased area, the evolution trend of structural response parameters, and historical early warning events, establish a quantitative discrimination model of disease development trend, and realize the automatic determination and classification output of structural disease risk level based on preset rules.

[0338] Furthermore, the trend identification and risk classification module includes the following sub-functional units:

[0339] 1) Disease evolution rate calculation unit

[0340] This unit receives time-series morphological parameters (such as crack length, fault height, erosion volume, etc.) output from the disease evolution analysis module, and performs differential processing on them to calculate their evolution rate. A differential velocity model of the following form is preferred:

[0341] ;

[0342] in, Indicates the morphological parameters of the disease at time... The rate of evolution, It is a function of time for a certain quantitative index of a disease (such as crack length, mm; misalignment height difference, mm; detachment volume, L).

[0343] If a continuous positive growth trend exists across multiple cycles, the system can mark the disease as being in an "expanding state" and store it in a trend cache list for subsequent comprehensive evaluation.

[0344] 2) Structural response over-limit analysis unit

[0345] This unit performs threshold judgments on the strain, displacement, and other response indicators output by the sensor trend fitting submodule to determine whether there are persistent or intermittent exceedances of limits. If a certain indicator exceeds the limit during a specified monitoring period... Inner continuity The number of times exceeded the set threshold If the location is considered to have a potential risk of instability, then the following criteria can be met:

[0346] ;

[0347] in, For indicator functions, Indicates the first The sensor value sampled next time. Set a preset alarm count threshold (e.g., 3 times). This is a preset strain or displacement safety threshold.

[0348] In one specific embodiment, a circumferential gap sensor records three instances of strain values ​​exceeding 0.01 within a consecutive 72-hour period, and the system marks this point as "abnormal response".

[0349] 3) Historical Early Warning Statistics Unit

[0350] This unit maintains historical alarm event records for each monitoring unit and counts the number of alarms issued by the system for the affected area within a certain period (e.g., the last 3 months or the last 10 monitoring cycles). If the historical alarm frequency of a certain affected area exceeds an empirical threshold (e.g., ≥5 times), it is considered a "high-frequency abnormal area," and its weight is correspondingly increased in the comprehensive evaluation.

[0351] 4) Comprehensive risk level assessment and classification unit

[0352] This unit is based on the above three indicators: disease evolution rate. Structural response over-limit ratio Historical alarm frequency A multi-factor risk assessment model is constructed, and the risk level is determined by combining the following rules:

[0353] Assessment condition Risk level ， ， Stable and growing, , Developing mutation or acceleration, , Rapidly deteriorating

[0354] in, This indicates the percentage of the structural response index relative to the threshold. This is the historical alarm frequency warning value.

[0355] In one specific application of the present invention, the system identifies that the crack in the left longitudinal seam of the 34th ring of segment Y increases in length from 2.1m to 3.4m over three cycles, with an average propagation rate of... m / month; the number of times its structural sensor strain exceeded the set threshold of 0.01 was 2, and the historical alarm frequency was 3. The system comprehensively judges its risk level as "developing".

[0356] In the Z-segment misalignment area, the point cloud elevation difference showed a step-like increase. Within 48 hours, the displacement sensor data showed three sudden increases, exceeding the set threshold, and the alarm frequency reached 7 times. Based on this, the system determined it to be at the "rapid deterioration" level and triggered the platform alarm push mechanism.

[0357] Through the above multi-dimensional fusion judgment, this module effectively realizes the trend identification and hierarchical management of tunnel structural defects, providing a reliable basis for subsequent prediction modules and maintenance intervention decisions.

[0358] Specifically, the results output module is as follows: the system automatically displays the analysis results on the subway operation and maintenance platform interface, including:

[0359] Location map of defects in a 3D tunnel model;

[0360] Evolution trend curves for each part (such as crack length over time curves, strain trend curves, etc.);

[0361] Risk level labels (color-coded: green = stable, orange = developing, red = deteriorating);

[0362] The text alert reads: "The misalignment of the Z-ring joint is increasing too rapidly, rising by 3.5 mm within 48 hours. This is classified as rapid deterioration, and immediate repair is recommended."

[0363] In addition, the system supports extending the current trend to the next 7 days and 30 days to predict the size range of future defects, and pushes the relevant reports in PDF format to the email addresses of the operation and maintenance unit and the person in charge of maintenance.

[0364] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal fusion tunnel structure appearance defect identification system, characterized in that, include: Image acquisition module, used to acquire continuous images of the inner wall of the tunnel lining; The laser point cloud acquisition module is used to acquire three-dimensional point cloud data of the tunnel cross-section. The structural sensor acquisition module is used to acquire structural state parameters, including strain, displacement, temperature, and humidity. The data synchronization and preprocessing module performs time alignment, coordinate transformation, and noise filtering on the aforementioned multi-source data. The multimodal feature extraction module performs deep feature extraction on images, point clouds, and sensor data; The heterogeneous feature fusion and disease identification module fuses features from various modalities and outputs disease type identification results; The risk assessment module estimates the size and parametrically represents the identified defects, and performs a risk assessment.

2. The system according to claim 1, characterized in that, The multimodal feature extraction module includes: A two-dimensional convolutional neural network submodule is used to extract texture and edge features of cracks, peeling, and water stains in images; The point cloud feature extraction submodule uses a 3D convolutional network or PointNet structure to extract geometric structural features from the point cloud. The sensor sequence analysis submodule uses a one-dimensional convolutional network or a recurrent neural network to process time-series data, including strain and displacement.

3. The system according to claim 2, characterized in that, After extracting features, the multimodal feature extraction module encodes them into high-dimensional feature vectors through modal embedding for subsequent fusion.

4. The system according to any one of claims 1-3, characterized in that, The heterogeneous feature fusion and disease identification module uses an attention mechanism to weight features between modalities, preferably including a multi-head cross-attention layer or a modal self-attention fusion structure. The fusion module further integrates a feature pyramid structure to achieve a fusion representation of multi-scale disease features, thereby enhancing the detection capability for cracks and misalignments of different sizes.

5. The system according to claim 4, characterized in that, The risk assessment module includes: The crack geometry estimation submodule is used to calculate crack length, width, and depth by combining image scale and point cloud depth; The misalignment height analysis submodule is used to extract the elevation difference of point clouds of adjacent cross sections and verify it in combination with displacement sensor readings; The detachment area measurement submodule is used to calculate the area and volume of the point cloud concave region based on its boundary. The submodule for analyzing water stain area and humidity changes is used to fuse image segmentation results with humidity temporal gradients to estimate the risk of water seepage. The structural damage characteristics output from the crack geometry estimation submodule, the misalignment height analysis submodule, the detachment area measurement submodule, and the water stain area and humidity change analysis submodule are comprehensively weighted to form a risk index, and the risk level is classified according to the set threshold range. Among them, the risk index Calculate using the following formula: ； In the formula: The crack characteristic factor is represented by the crack length. ,width With depth Comprehensive calculations yielded the following results: ； in Reference limits for design permitting; The misalignment characteristic factor is defined as the difference in point cloud elevation between adjacent cross sections. With sensor displacement reading Normalized combination: ； This represents the volume factor of the detachment area, taken as the volume of the concave region. Compared with reference volume The ratio; The humidity factor of water stains is determined by the area of ​​water seepage. With humidity change rate Normalization yields: ； These are the weighting coefficients for each damage feature, determined based on field experience or training data; When the risk index Exceeding the set threshold When this occurs, the system automatically determines it to be in an early warning state, and classifies it according to the risk level range. The risk level is categorized into three levels: "stable," "developing," or "deteriorating." 6. The system according to claim 5, characterized in that, The visualization display module includes a monitoring terminal connected to the data processing and analysis module. The monitoring terminal marks the location, type, and size information of defects on the three-dimensional model or two-dimensional unfolded diagram of the tunnel structure, and issues an alarm signal when the size of the defect exceeds the limit.

7. The system according to claim 6, characterized in that, The result output and visualization module highlights the location of the disease based on the three-dimensional tunnel model or two-dimensional unfolded diagram, and displays the disease type, size and risk level in the form of labels. The system has a sensor threshold alarm function. When the real-time data exceeds the preset range or the size of the defect exceeds the limit, it will automatically trigger an audible and visual alarm or remotely push alarm information.

8. A multimodal disease evolution monitoring and trend assessment system, comprising a multi-source data acquisition module, a multi-time-series data archiving and alignment module, a disease evolution analysis module, a trend identification and risk classification module, and a result output module. Its features are: The multi-source data acquisition module is used to collect multimodal monitoring data of the tunnel structure, including image data of the inner surface of the tunnel, three-dimensional point cloud data of the tunnel cross section, and structural response data of key parts of the tunnel structure. The multi-temporal data archiving and alignment module is used to perform temporal and spatial alignment and fusion of the image data, point cloud data and structural response data collected at different times to form a temporal fusion dataset that can be used for comparative analysis of disease evolution. The disease evolution analysis module is used to extract evolutionary characteristic parameters of tunnel disease areas based on the time-series fusion dataset, including disease morphology expansion trend, structural response change trend, and evolution rate index. The trend identification and risk classification module is used to identify the development trend of the disease based on the evolutionary feature parameters, and to determine the risk level of the disease based on preset rules; The results output module is used to output the results of tunnel defect monitoring and trend assessment.

9. The system according to claim 8, characterized in that, The multi-source data acquisition module includes: several high-definition industrial cameras installed between tunnel lining ring segments as image acquisition units, with each camera spaced 30-60 meters apart along the longitudinal direction of the tunnel; a 360° rotating laser scanner configured on the tunnel maintenance passage or inspection carrier as a laser point cloud acquisition unit to acquire three-dimensional point cloud data of the tunnel cross-section and internal cavity space; and several structural sensor nodes deployed at vulnerable locations to acquire structural response data. The sensor nodes include strain sensors, displacement sensors, and temperature and humidity sensors; all images, point clouds, and sensor data acquired by the multi-source data acquisition module are bound with precise timestamps and location information tags, and the consistency of multimodal data in time is ensured through a global clock synchronization module.

10. The system according to claim 8, characterized in that, The multi-time-series data archiving and alignment module includes: The image registration and archiving submodule is used to perform temporal archiving and geometric alignment processing on tunnel inner wall images acquired at different times from the same monitoring location; The point cloud model alignment submodule is used to preprocess and spatially align the tunnel 3D point cloud data acquired in multiple monitoring cycles. After filtering, noise reduction and voxelization resampling of the original point cloud, the iterative nearest point ICP algorithm or the FPFH feature matching combined with the RANSAC algorithm is used to unify the point cloud models of each period into the same coordinate system. The structural sensing data synchronization submodule is used to perform unified timeline alignment and association indexing of time-series data from structural sensors such as strain, displacement, temperature and humidity with corresponding image and point cloud data. It associates sensing data with image / point cloud records at the corresponding time based on sensor ID, installation location coordinates and acquisition timestamp.

Images