A method and system for building damage diagnosis that integrates multimodal sensing and deep learning

By integrating multimodal sensing and deep learning, this method addresses the issues of single data modality and insufficient generalization ability across working conditions in existing building damage diagnosis methods. It achieves efficient, accurate, and robust diagnosis of building damage and is applicable to damage detection in building structures.

CN123116015APending Publication Date: 2026-11-17CHINA JK INST OF ENG INVESTIGATION & DESIGN +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611377946.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing deep learning-based building damage diagnosis methods suffer from several problems, including insufficient diagnostic information due to a single data modality, a severe lack of robustness to missing scenarios in actual engineering projects, and insufficient generalization ability across structural types and working conditions.

Method used

By employing a method that integrates multimodal sensing and deep learning, vibration response signals, appearance images, and on-site description text of building structures are collected simultaneously. A deep feature extraction network is constructed using multimodal data, a bidirectional cross-attention mechanism is introduced to achieve feature fusion, and end-to-end optimization is performed through a multi-task joint loss function. Combined with modal dropout training strategy, knowledge distillation mechanism, and domain adaptation technology, cross-modal and cross-condition diagnostic capabilities are achieved.

Benefits of technology

It significantly improves the accuracy of damage diagnosis, ensuring reliable diagnostic performance even in modal incomplete or cross-domain scenarios. It achieves comprehensive and accurate diagnosis of damage location, type, and extent, and reduces the data acquisition and annotation costs for model deployment in new scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116015A_ABST
    Figure CN123116015A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for diagnosing building damage by integrating multimodal sensing and deep learning, belonging to the fields of building engineering and artificial intelligence technology. The method includes: simultaneously acquiring and preprocessing multimodal data of building structures; constructing a deep feature extraction network based on the multimodal data to extract deep features, and introducing a bidirectional cross-attention mechanism to obtain deeply fused features; constructing three task-specific prediction heads in parallel for damage location identification, damage type classification, and damage degree regression; introducing a modality discarding training strategy and a knowledge distillation mechanism to guide single-modal prediction under conditions of missing modalities with predictions under complete modalities; and introducing domain adaptation technology to achieve generalization and transfer of diagnostic knowledge from known structures to unknown structures. Using the above method and system, this invention achieves comprehensive and accurate diagnosis of the location, type, and degree of damage to building structures, while ensuring the robustness and generalization ability of the model under data loss or cross-scenario conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of building engineering and artificial intelligence technology, and in particular to a method and system for diagnosing building damage that integrates multimodal sensing and deep learning. Background Technology

[0002] Throughout their entire life cycle, building structures inevitably suffer from multiple impacts, including environmental erosion, long-term loads, material performance degradation, and extreme natural disasters such as earthquakes and typhoons. Various types of damage—from microscopic material fatigue cracks to macroscopic component cracking, spalling, corrosion, and even overall deformation and instability—gradually emerge, accumulate, and evolve within the structure. If such damage cannot be detected, accurately assessed, and effectively intervened upon in a timely manner, the structure's load-bearing capacity and safety margin will continue to deteriorate, potentially leading to sudden failure or even complete collapse, causing severe casualties and incalculable economic losses. In recent years, numerous building safety accidents caused by the failure to diagnose structural damage in a timely manner have occurred both domestically and internationally. This makes the development of efficient, accurate, and automated structural damage diagnosis technologies a crucial issue that urgently needs to be addressed in the field of civil engineering.

[0003] Traditionally, structural damage detection in buildings relies primarily on manual visual inspections by personnel, supplemented by spot checks using single instruments such as crack detectors, rebound hammers, and ultrasonic testing equipment. This approach has several inherent limitations: First, the results are highly dependent on the professional competence and subjective experience of the inspectors; different personnel often provide significantly different assessments of the same damage condition, making it difficult to guarantee objectivity, consistency, and repeatability. Second, for complex structures such as long-span bridges, super high-rise buildings, and large public venues, manual inspection is not only time-consuming, labor-intensive, and inefficient, but also presents numerous blind spots and safety hazards associated with working at heights. Third, the traditional periodic spot check model is a discrete and discontinuous inspection method, unable to achieve real-time tracking and continuous monitoring of structural conditions, and severely lacks early warning capabilities for sudden or progressively cumulative damage.

[0004] With the rapid development of sensing, communication, and computer technologies, structural health monitoring (SHM) systems have been increasingly widely used in critical buildings and infrastructure. By deploying various types of sensing devices, such as accelerometers, strain sensors, displacement sensors, and piezoelectric sensors, at key structural locations, SHM systems can collect real-time mechanical response signals of structures under various loads and environmental excitations, providing an unprecedented massive data foundation for continuous monitoring and quantitative assessment of structural condition. However, with the continuous increase in monitoring scale and sampling frequency, the amount of data collected by SHM systems is growing exponentially. Traditional data processing paradigms based on manual experience interpretation or shallow statistical analysis are no longer sufficient to effectively extract subtle damage characteristics from massive monitoring data. There is an urgent need to introduce new data processing technologies with stronger nonlinear mapping capabilities and automatic feature extraction capabilities.

[0005] In recent years, artificial intelligence technologies centered on deep learning have shown broad application prospects in the field of structural damage identification due to their superior automatic feature extraction and high-dimensional nonlinear mapping capabilities. On the one hand, deep learning-based vibration signal analysis methods can directly abstract deep feature representations closely related to the physical state of the structure from raw sensor data layer by layer, effectively avoiding the cumbersome manual feature engineering design that relies on prior knowledge in traditional methods. On the other hand, deep learning methods based on computer vision can perform end-to-end automatic analysis of structural surface images, achieving accurate detection and location of surface damage such as cracks, spalling, corrosion, and biological growth. In addition, deep learning methods based on natural language processing can transform on-site inspection text records into computable semantic feature vectors, mining damage description information and professional experience knowledge contained in unstructured text. These technical approaches have broken through several bottlenecks of traditional methods from different dimensions, laying an important foundation for the intelligent development of structural damage diagnosis.

[0006] However, existing deep learning-based structural damage diagnosis methods still suffer from problems such as insufficient diagnostic information due to the single data modality, a severe lack of robustness to missing scenarios in actual engineering, and a severe lack of generalization diagnostic ability across structural types and working conditions. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for diagnosing building damage that integrates multimodal sensing and deep learning, thereby solving the problems existing in the prior art.

[0008] To achieve the above objectives, this invention provides a building damage diagnosis method integrating multimodal sensing and deep learning, comprising the following steps: S1. Synchronously collect multimodal data of the building structure and perform preprocessing; S2. Based on multimodal data, a deep feature extraction network adapted to the characteristics of each modality is constructed to extract its deep features respectively, and a bidirectional cross-attention mechanism is introduced to achieve deep fusion of multimodal features to obtain deep fused features; S3. Using deep fusion features as shared input, three task-specific prediction heads are constructed in parallel: damage location identification, damage type classification, and damage degree regression. End-to-end optimization is achieved through a multi-task joint loss function. S4. Introduce a modality dropout training strategy and a knowledge distillation mechanism to guide single-modality prediction under modality missing conditions with prediction under complete modality. S5. Introducing domain adaptation technology, by minimizing the distribution difference between the deep fusion features of the source domain and the target domain and introducing domain adversarial training, the diagnostic knowledge can be generalized and transferred from known structures to unknown structures.

[0009] Preferably, step S1 specifically includes: S11. Deploy sensors at key stress-bearing and vulnerable parts of the building structure to collect three types of heterogeneous modal data: vibration response signals (acceleration / displacement time series data), appearance images (visual information of visible damage on the structural surface), and on-site descriptive text (semantic records of the inspectors); synchronize and calibrate the acquisition clocks of all sensors to ensure that the acquisition time references of all types of data are consistent.

[0010] S12. Regarding vibration response signals, due to the complex operating conditions of building structures and significant environmental noise interference, the original vibration signal contains a large amount of noise components unrelated to damage. Directly using it for damage diagnosis will severely affect the identification accuracy. Empirical mode decomposition is used to adaptively decompose the original vibration signal into several intrinsic mode function components and a residual term: ; in, This is the original vibration response signal. For the first One IMF component, It is a residual signal. The total number of IMF components obtained from the decomposition; Each IMF component represents a vibrational component at different frequency scales in the original signal, with the components containing damage characteristic information mainly concentrated in the first few high-frequency IMFs. To obtain information from... From the IMF components, the components carrying the main damage characteristics were selected, and the variance contribution rate of each IMF component was calculated using principal component analysis. ; ; ; in, For the first The variance of each IMF component The number of signal sampling points. For the first The first IMF component Each sample value, For the first The mean of each IMF component, For the first The variance contribution rate of each IMF component for The sum of the variances of the IMF components. For the front The cumulative contribution rate of each IMF component; when the cumulative contribution rate reaches a preset threshold (usually above 85%), the number of valid IMF components is determined. Before selection The IMF components serve as an effective characterization of the vibration modes, denoted as the vibration characteristic tensor. ;the remaining IMF components and residuals The information contained in it is very limited and should be discarded.

[0011] S13. For structural appearance images, in order to extract multi-scale visual features related to damage in the images, continuous wavelet transform is used to perform time-frequency analysis on the appearance images, decomposing the image signals into time-frequency domain representations at different scales: ; in, These are wavelet coefficients, representing the image signal. In scale Translation position The following features The coordinate variables of the image signal. The conjugate of the mother wavelet function is used; through analysis at different scales, high-frequency details (such as the edges of micro-cracks) and low-frequency trends (such as the overall deformation area) in the image are captured simultaneously to achieve multi-resolution analysis; the time-frequency image after wavelet transform is converted into a grayscale image to obtain the visual feature tensor. Its dimensions are unified as ,in and These are the height and width of the image in pixels, respectively. S14. For the scene description text, the DeBERTa pre-trained language model is used to perform semantic encoding on the scene description text. DeBERTa separates the content representation and position representation of words in the scene description text through a decoupled attention mechanism, and calculates the attention weights between word units based on semantic and positional information respectively: ; in, For position and Attention weights between lexical units and Representing the content and position vectors respectively, after semantic encoding, we obtain the text semantic feature tensor. Its dimensions are ,in The length of the text sequence. This is the semantic encoding dimension.

[0012] Step S1 completed the acquisition and normalization preprocessing of three types of heterogeneous modal data, obtaining vibration feature tensors, visual feature tensors, and textual semantic feature tensors, respectively. However, the three types of features reside in different feature spaces—vibration features reflect mechanical response, visual features reflect apparent damage, and semantic features reflect manually interpreted information—direct splicing or simple fusion cannot effectively mine complementary information between modalities. This step addresses this challenge by designing deep feature extraction networks adapted to the characteristics of each modality and introducing a bidirectional cross-attention mechanism to achieve deep interaction and fusion of cross-modal information.

[0013] Preferably, in step S2, a deep feature extraction network adapted to the characteristics of each modality is constructed based on the multimodal data to extract its deep features, specifically as follows: The vibration characteristic tensor obtained in S12 of Each effective IMF component is respectively used as We construct a multi-channel one-dimensional convolutional neural network using inputs from independent channels. Each channel undergoes independent convolution and pooling operations. The convolution operation for each channel is represented as follows: ; in, For the first The first channel The output feature map of each convolutional layer For the first The first channel The convolutional kernel weights of each convolutional layer It is a one-dimensional convolution operator. For bias terms, The ReLU activation function is used; after each channel independently completes three layers of convolution-pooling operations, the output features of all channels are concatenated and fused along the channel dimension. ; in, For the deep mechanical integration characteristics of vibration modes, For the first The output feature map of the third convolutional layer in each channel; this multi-channel parallel structure enables each IMF component (representing vibration components at different frequency scales) to extract features independently in its own channel, avoiding mutual interference between different frequency components, and finally integrating the multi-scale vibration features through splicing and fusion.

[0014] The visual feature tensor obtained in S13 The input is processed by the Swing Transformer network for visual feature extraction. The Swing Transformer extracts deep visual features of the image through a hierarchical feature representation mechanism and a shifted window multi-head self-attention mechanism. Standard window multi-head self-attention and sliding window multi-head self-attention are calculated alternately. ; ; ; ; in, This represents a standard window multi-head self-attention layer. This indicates a sliding window multi-head self-attention layer. Representation layer normalization, This represents a multilayer perceptron. For the first The output characteristics of each module For the first Intermediate features output by the MSA layer of each module For the first Intermediate features output by the MSA layer of each module For the first The output characteristics of each module For the first The output features of each module; the hierarchical representation mechanism starts from small-sized image patches and gradually merges adjacent image patches in the deep network to form multi-scale hierarchical feature maps, enabling the model to focus on both subtle local damage features such as cracks and peeling, as well as the overall deformation and damage of the component; after encoding by SwinTransformer, the visual deep features are obtained. .

[0015] The text semantic feature tensor obtained in S14 Inputting into the DeBERTa network for deep semantic feature extraction, DeBERTa's decoupled attention mechanism decomposes attention weights into three parts: content-to-content, content-to-location, and location-to-content. ; softmax ; in, These are content mapping vectors for queries, keys, and values, respectively. These are the relative position mapping vectors for the query and the key, respectively. as a word element and The relative distance between them Let be the dimension of the mapping matrix. The output of the attention layer is used; after DeBERTa encoding, the deep semantic features of the text are obtained. .

[0016] Preferably, in step S2, the original features are... , , The features enhanced by other modalities are concatenated and fused along the channel dimension to obtain a three-modal deep fused feature: ; in, Visual features enhanced by vibration information Visual features enhanced with textual information, Vibration characteristics enhanced by visual information Vibration characteristics enhanced by text information For text features enhanced with visual information, This is a text feature enhanced with vibration information. The deep fusion feature includes the original information from three modalities as well as pairwise cross-enhancement information. When the data quality of a certain modality is poor or partially missing, the features of that modality can be compensated for through cross-attention from other modalities, thereby improving the model's robustness under incomplete data conditions.

[0017] Taking the cross-attention fusion of visual and vibrational features as an example, the visual features are projected into a query matrix. The vibrational features are projected as a bond matrix. Sum matrix : ; Calculate the attention weight between the visual query and the vibrating key to obtain the visual features enhanced by vibration information: ; in, This represents the dimension of the key vector. Similarly, using vibration features as the query and visual features as the key, we obtain vibration features enhanced with visual information. Similarly, calculate the visual-text cross-attention separately. and Vibration-Text Cross-Attention and .

[0018] Preferably, in step S3, three task-specific prediction heads are constructed in parallel using deep fusion features as shared input: damage location identification, damage type classification, and damage severity regression. Specifically: Deep fusion features The input branch identifies the damage location and consists of a three-layer fully connected network. The number of neurons in the output layer is equal to the total number of units in the building structure. The Softmax activation function is used to output the probability distribution of damage located in each unit: ; in, Let be the probability distribution vector of the damage location. and The first The weight matrix and bias terms of a fully connected layer. The ReLU activation function is used, and the loss function for the location recognition branch is cross-entropy loss. ; in, For the first The actual damage label of each unit (1 indicates that the unit has damage). The model predicts the first The probability that a unit is damaged; Deep fusion features The input damage type classification branch uses a network structure similar to the location recognition branch, and the number of neurons in the output layer equals the total number of damage types. (Including types such as cracks, spalling, corrosion, biological growth, and water seepage), the probability distribution of each damage type is output using the Softmax activation function. : ; The loss function for the type classification branch also uses cross-entropy loss: ; in, For the first True labels for each type of damage The model predicts the first The probability of each damage type; Deep fusion features The input is a damage degree regression branch, and the output layer uses a Linear activation function to output the predicted damage degree of each element (such as quantitative indicators like crack depth and cross-sectional loss rate). ; The loss function for the degree regression branch uses the mean squared error loss: ; in, For the first The actual damage level of each unit (e.g., crack depth, in mm). The model predicts the first The damage level value of each unit.

[0019] Preferably, the joint loss function for multiple tasks in step S3 is: ; in, , and These are the weight coefficients for the location recognition, type classification, and degree regression tasks, respectively. The value is 1.0. The value should be between 1.0 and 1.5, preferably 1.2. The value is set to 1.5-2.0, with 1.5 being the preferred value. This value is used to balance the importance of each task in joint training. Through end-to-end backpropagation optimization, the three tasks share the underlying deep fusion features. At the same time, each task-specific network parameter is optimized to achieve multi-task collaborative learning.

[0020] Preferably, in step S4, the modality dropout training strategy uses a preset dropout probability during model training. Randomly mask the input data of a certain modality, specifically: For each training sample, three binary masks are generated independently. These correspond to vibration, visual, and textual modalities, respectively, with each mask representing a probability. The value is 1 (reserved), with probability. The value is set to 0 (discarded), and the input feature tensor of the discarded mode is set to a zero tensor. ; The masked feature tensor is input into subsequent S2, allowing the model to simultaneously encounter multiple input scenarios during training, including complete modalities, single modalities, and dual modalities, thus learning robust feature representations independent of any specific modality. The dropout probabilities for the three modalities can be set independently based on the historical statistics of their respective data quality—if the missing rate of a certain modality's data is high in actual collection, its dropout probability is increased accordingly, reducing the model's dependence on that modality.

[0021] Preferably, in step S4, the knowledge distillation mechanism uses the prediction results of the multi-task joint learning model in step S3 under trimodal input as the teacher output, and uses the prediction results of the corresponding prediction head under single-modal or bimodal input as the student output. For each diagnostic task... Calculate the distillation loss between teacher and student outputs. For classification tasks (location recognition and type classification), use KL divergence with a temperature coefficient: ; ; in, and Teacher networks and student networks in the task The logical output above, The temperature coefficient has a value of 2-5, preferably 3. The KL divergence is used to measure the difference between two probability distributions; for regression tasks, the distillation loss uses the mean squared error. ; in, and These represent the impairment values ​​predicted by the teacher network and the student network, respectively. The joint optimization of the multi-task learning objective and the knowledge distillation objective combines the multi-task joint loss and the cross-modal knowledge distillation loss to obtain the complete total loss function for model training. ; in, For the task The distillation loss weighting coefficient is set to 0.1-0.5. Through end-to-end optimization of this total loss function, the model learns accurate damage diagnosis capabilities on complete modal data (by...). ), while maintaining reliable diagnostic performance even under modal loss conditions (through Distill the "integrated judgment experience" of the complete modality into the single-modality branch.

[0022] Preferably, step S5 specifically includes: S51. Define the structure type or operating condition with sufficient existing labeled data as the source domain. Define the new structural type or new operating condition to be diagnosed as the target domain. , The number of samples in the source domain. The number of samples in the target domain; the source and target domains share the same feature space and label space, but the data distributions are different—that is... .

[0023] To measure and reduce the distributional difference between two domains, the maximum mean difference is used as a measure of distributional distance. In the regenerating kernel Hilbert space, the maximum mean difference distance between the deep fused features of the source and target domains is: ; in, For the feature mapping induced by the kernel function, Let Hilbert be the norm in the regenerated kernel space. and The source domain is respectively The nth sample and the target domain The deep fusion features of each sample; the smaller the maximum mean difference, the closer the feature distributions of the two domains are, and the better the generalization performance of the model in the target domain.

[0024] S52, End-of-line access area discriminator in step S2 This is used to determine whether input features come from the source domain or the target domain. The domain discriminator engages in an adversarial training process with the feature extraction network: the feature extraction network attempts to extract "domain-invariant features" that prevent the domain discriminator from distinguishing the source, while the domain discriminator attempts to accurately determine the source of the features. This adversarial training is achieved through a Gradient Reversal Layer (GRL). ; in, For the first Domain labels for each sample (0 represents the source domain, 1 represents the target domain). The predicted probability of the domain discriminator is denoted as ...

[0025] S53. Combining the multi-task joint loss, knowledge distillation loss, and domain adversarial loss yields the total loss function for cross-domain transfer learning: ; in, The weighting coefficient for the domain adversarial loss is set to 0.1-0.3, with 0.2 being preferred. Through end-to-end optimization of this total loss function, the model learns accurate damage diagnosis capabilities on the source domain labeled data (through...). On the other hand, domain adversarial training forces the feature extraction network to learn cross-domain invariant feature representations (through...). This enables the effective transfer of diagnostic knowledge in target domains with limited or no labeled data.

[0026] This invention also provides a building damage diagnosis system integrating multimodal sensing and deep learning, comprising: The multimodal data acquisition and preprocessing module simultaneously acquires vibration response signals, appearance images, and on-site description text of the building structure, and preprocesses various types of data. The multimodal feature extraction and interactive fusion module extracts deep features from multimodal data and achieves deep fusion of modal features through a bidirectional cross-attention mechanism. The multi-task joint learning and modality adaptive diagnosis module completes three diagnostic tasks—damage location identification, damage type classification, and damage severity regression—based on deep fusion features. The modality adaptation and knowledge distillation enhancement module enables the model to maintain its diagnostic capabilities under modality loss conditions through modality dropout training and cross-modality knowledge distillation. The cross-condition transfer learning module transfers diagnostic knowledge learned from known structures to unknown structures through domain adaptation technology.

[0027] Therefore, the building damage diagnosis method and system of the present invention, which integrates multimodal sensing and deep learning, has the following beneficial effects: (1) By integrating three types of heterogeneous modal data—vibration response signals, apparent images, and on-site description text—deep interaction between modalities is achieved through a bidirectional cross-attention mechanism, overcoming the limitation of incomplete information in single modal data and significantly improving the accuracy of damage diagnosis. (2) By using the modal dropout training strategy and cross-modal knowledge distillation mechanism, the model can still maintain reliable diagnostic performance under modal incomplete conditions such as image blurring, sensor damage or text missing; (3) Through a multi-task joint learning architecture, three types of diagnostic information, namely damage location, damage type and damage degree, are output simultaneously; (4) Through domain adaptation and adversarial training techniques, diagnostic knowledge transfer from known structure types to unknown structure types was achieved, which greatly reduced the data collection and annotation costs for model deployment in new scenarios.

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0029] Figure 1 This is a flowchart of the building damage diagnosis method that integrates multimodal sensing and deep learning according to the present invention; Figure 2 This is a structural block diagram of the building damage diagnosis system that integrates multimodal sensing and deep learning, as described in this invention. Detailed Implementation

[0030] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0031] To verify, as Figure 1-2 To demonstrate the effectiveness of the building damage diagnosis method and system integrating multimodal sensing and deep learning, this embodiment selects a five-story, two-span reinforced concrete frame structure as the experimental object. The structure has a first-floor height of 4.5m, a standard-floor height of 3.6m, column cross-sections of 600mm × 600mm, beam cross-sections of 300mm × 600mm, floor slab thickness of 120mm, concrete strength grade of C30, and HRB400 grade steel reinforcement. Damage detection devices (such as accelerometers, steel corrosion testers, etc.), high-definition industrial cameras, and on-site recording terminals are simultaneously deployed at key beam-column joints and mid-span locations on the first, third, and fifth floors to collect three types of heterogeneous modal data: Vibration response signal: White noise excitation was applied to the bottom of the structure, and the acceleration response of each floor was collected. The sampling frequency was 1000Hz, and the sampling time was 60 seconds each time.

[0032] Appearance images: Full coverage of the structural surface is captured, with a focus on the nodal areas and the bottom of the mid-span. The image resolution is uniformly 224×224 pixels.

[0033] On-site description text: The inspection personnel write a text description based on the structural appearance and monitoring records, using a standardized template. It includes component numbers, signs of damage, environmental information, etc.

[0034] Six typical damage conditions were set up (cracks, spalling, corrosion, biofouling, water seepage, and combined damage). Each damage condition was tested at three different locations and three severity levels (mild, moderate, and severe). A dataset containing 12,000 complete trimodal samples was constructed and divided into training, validation, and test sets in a 6:2:2 ratio. Simultaneously, a similar frame structure with different floor heights and spans was used as the target domain, and 2,000 unlabeled samples (sensor data only, without damage labels) were collected for cross-domain generalization validation.

[0035] This embodiment comprehensively evaluates the method of the present invention on a test set. The results are as follows: Table 1 Overall performance of the three diagnostic tasks

[0036] As shown in Table 1, in the damage location identification task, the Top-1 accuracy reached 92.3%, and the Top-3 accuracy reached 97.1%; in the damage type classification task, the macro-average F1-Score was 0.87, and the weighted average F1-Score was 0.89; in the damage severity regression task, the normalized MAE was 0.052, the RMSE was 0.078, and the R-squared value was [missing data]. 2 The coefficient of determination is 0.91.

[0037] To verify the effectiveness of the bidirectional cross-attention mechanism, a systematic ablation experiment was conducted in this embodiment. In the modal fusion strategy comparison, the bidirectional cross-attention fusion (trimodal) adopted in this invention was compared with trimodal simple channel splicing, bimodal fusion, and single-modal schemes. The results are shown in Table 2: Table 2 Performance Comparison of Different Modal Configurations and Fusion Strategies

[0038] The results show that the present invention achieves optimal performance in all three tasks: compared with simple trimodal stitching, the location recognition accuracy is improved by 6.3 percentage points (from 86.0% to 92.3%), the type classification F1-Score is improved by 0.07 (from 0.82 to 0.89), and the degree regression RMSE is reduced by 17.0% (from 0.094 to 0.078), which fully demonstrates the effectiveness of the bidirectional cross-attention mechanism in mining cross-modal complementary information. Among the three single-modal schemes, the image single-modal scheme performs best (78.1% for location recognition and 0.75 for type classification F1), but both are far lower than the multimodal fusion scheme of the present invention.

[0039] Robustness experiments were conducted for scenarios with missing modes. The performance degradation rate of each method was calculated based on the performance of the complete modes. The results are shown in Table 3. Table 3 Diagnostic performance degradation rate under different modality loss scenarios

[0040] Under complete modal conditions, the performance of this invention is comparable to that of the baseline model without robust training methods. However, when modalities are missing during the testing phase, the performance of the baseline model degrades significantly. In contrast, this invention, through the combined effect of modal dropout training and knowledge distillation, demonstrates a significant advantage in all missing modal scenarios. In the scenarios of missing text (vibration + image), missing image (vibration + text), and missing vibration (image + text), the attenuation rates of this invention are 13.8%, 16.2%, and 11.5%, respectively, all significantly lower than the baseline model's attenuation levels of 44.2%, 51.6%, and 39.8%, demonstrating the effectiveness of the robust training strategy.

[0041] The source domain model was directly applied to the target domain, and MMD alignment, adversarial training, and the complete solution of this invention were gradually introduced. The target domain diagnostic accuracy and feature MMD distance are shown in Table 4. Table 4 Target Domain Accuracy and MMD Distance Before and After Cross-Domain Migration

[0042] Cross-domain generalization experiments show that when the source domain model is directly used for target domain testing, the location recognition accuracy is only 41.3%, and the MMD distance between the source and target domain features is as high as 0.82, indicating a significant difference in feature distribution between the two domains. After introducing MMD distribution alignment, the accuracy improves to 63.7%, and the MMD distance decreases to 0.45; after further introducing domain adversarial training, the accuracy improves to 72.1%, and the MMD distance decreases to 0.28. Using the complete scheme of this invention (including knowledge distillation and domain adaptation), the target domain accuracy reaches 78.6%, an improvement of 37.3 percentage points compared to the baseline without transfer, while the MMD distance further decreases to 0.21, proving that the learned features have good domain invariance. In addition, multi-task joint learning improves the accuracy by an average of 3.8 percentage points across the three tasks compared to single-task independent training; the marginal contribution analysis of each module shows that multimodal fusion and cross-attention mechanisms contribute the most, the robustness module has limited improvement on source domain accuracy but is crucial for modality-deficient scenarios, and the domain adaptation module, although it does not significantly improve source domain accuracy, brings a qualitative leap to cross-domain transfer.

[0043] Therefore, this invention adopts the above-mentioned building damage diagnosis method and system that integrates multimodal sensing and deep learning. By fusing vibration, visual and textual three-modal data and establishing a cross-modal interaction mechanism, it achieves a comprehensive and accurate diagnosis of the location, type and degree of damage to building structures. At the same time, by combining modal dropout training, knowledge distillation and domain adaptation techniques, it ensures the robustness and generalization ability of the model in data-deficient or cross-domain scenarios, providing intuitive and reliable technical support for engineering decision-making.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A building damage diagnosis method integrating multimodal sensing and deep learning, characterized in that, Includes the following steps: S1. Synchronously collect multimodal data of the building structure and perform preprocessing; S2. Based on multimodal data, a deep feature extraction network adapted to the characteristics of each modality is constructed to extract its deep features respectively, and a bidirectional cross-attention mechanism is introduced to achieve deep fusion of multimodal features to obtain deep fused features; S3. Using deep fusion features as shared input, three task-specific prediction heads are constructed in parallel: damage location identification, damage type classification, and damage degree regression. End-to-end optimization is achieved through a multi-task joint loss function. S4. Introduce a modality dropout training strategy and a knowledge distillation mechanism to guide single-modality prediction under modality missing conditions with prediction under complete modality. S5. Introducing domain adaptation technology, by minimizing the distribution difference between the deep fusion features of the source domain and the target domain and introducing domain adversarial training, the diagnostic knowledge can be generalized and transferred from known structures to unknown structures.

2. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 1, characterized in that, Step S1 is as follows: S11. Deploy sensors in key stress-bearing and vulnerable parts of the building structure to collect three types of heterogeneous modal data: vibration response signals, apparent images, and on-site description text; S12. The original vibration signal is adaptively decomposed using empirical mode decomposition, which decomposes it into several intrinsic mode function components and a residual term: ; in, This is the original vibration response signal. For the first One IMF component, It is a residual signal. This represents the total number of IMF components obtained from the decomposition. Principal component analysis was used to calculate the variance contribution rate of each IMF component: ; ; ; in, For the first The variance of each IMF component The number of signal sampling points. For the first The first IMF component Each sample value, For the first The mean of each IMF component, For the first The variance contribution rate of each IMF component for The sum of the variances of each IMF component. For the front The cumulative contribution rate of each IMF component; when the cumulative contribution rate reaches a preset threshold, the number of valid IMF components is determined. Before selection The IMF components serve as an effective characterization of the vibration modes, denoted as the vibration characteristic tensor. ; S13. Continuous wavelet transform is used to perform time-frequency analysis on the apparent image, decomposing the image signal into time-frequency domain representations at different scales: ; in, These are wavelet coefficients, representing the image signal. In scale Translation position The following features The coordinate variables of the image signal. The conjugate of the mother wavelet function is used; the time-frequency image after wavelet transform is converted into a grayscale image to obtain the visual feature tensor. Its dimensions are unified as ,in and These are the height and width of the image in pixels, respectively. S14. The DeBERTa pre-trained language model is used to perform semantic encoding on the scene description text. DeBERTa separates the content representation and position representation of words in the scene description text through a decoupled attention mechanism, and calculates the attention weights between word units based on semantic and positional information respectively. ; in, For position and Attention weights between lexical units and Let the content and position vectors be represented respectively. After semantic encoding, the text semantic feature tensor is obtained. Its dimensions are ,in The length of the text sequence. This is the semantic encoding dimension.

3. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 2, characterized in that, In step S2, a deep feature extraction network adapted to the characteristics of each modality is constructed based on the multimodal data to extract its deep features, specifically as follows: The vibration characteristic tensor obtained in S12 of Each effective IMF component is respectively used as We construct a multi-channel one-dimensional convolutional neural network using inputs from independent channels. Each channel undergoes independent convolution and pooling operations. The convolution operation for each channel is represented as follows: ; in, For the first The first channel The output feature map of each convolutional layer For the first The first channel The convolutional kernel weights of each convolutional layer It is a one-dimensional convolution operator. For bias terms, The ReLU activation function is used; after each channel independently completes three layers of convolution-pooling operations, the output features of all channels are concatenated and fused along the channel dimension. ; in, For the deep mechanical integration characteristics of vibration modes, For the first The output feature map of the third convolutional layer for each channel; The visual feature tensor obtained in S13 The input is processed by the Swin Transformer network for visual feature extraction. The Swin Transformer extracts deep visual features of the image through a hierarchical feature representation mechanism and a shifted window multi-head self-attention mechanism. Standard window multi-head self-attention and sliding window multi-head self-attention are calculated alternately. ; ; ; ; in, This represents a standard window multi-head self-attention layer. This indicates a sliding window multi-head self-attention layer. Representation layer normalization, This represents a multilayer perceptron. For the first The output characteristics of each module For the first Intermediate features output by the MSA layer of each module For the first Intermediate features output by the MSA layer of each module For the first The output characteristics of each module For the first The output features of each module are encoded by the Swing Transformer to obtain the visual deep features. ; The text semantic feature tensor obtained in S14 Inputting into the DeBERTa network for deep semantic feature extraction, DeBERTa's decoupled attention mechanism decomposes attention weights into three parts: content-to-content, content-to-location, and location-to-content. ; softmax ; in, These are content mapping vectors for queries, keys, and values, respectively. These are the relative position mapping vectors for the query and the key, respectively. as a word element and The relative distance between them Let be the dimension of the mapping matrix. The output of the attention layer is used; after DeBERTa encoding, the deep semantic features of the text are obtained. .

4. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 3, characterized in that, In step S2, the original features , , The features, after being enhanced by other modalities, are concatenated and fused along the channel dimension to obtain a three-modal deep fused feature: ; in, Visual features enhanced by vibration information Visual features enhanced with textual information, Vibration characteristics enhanced by visual information Vibration characteristics enhanced by text information For text features enhanced with visual information, Text features enhanced with vibration information.

5. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 4, characterized in that, In step S3, using deep fusion features as shared input, three task-specific prediction heads are constructed in parallel: damage location identification, damage type classification, and damage severity regression. Specifically: Deep fusion features The input branch identifies the damage location and consists of a three-layer fully connected network. The number of neurons in the output layer is equal to the total number of units in the building structure. The Softmax activation function is used to output the probability distribution of damage located in each unit: ; in, Let be the probability distribution vector of the damage location. and The first The weight matrix and bias terms of a fully connected layer. The ReLU activation function is used, and the loss function for the location recognition branch is cross-entropy loss. ; in, For the first The actual damage label of each unit, The model predicts the first The probability that a unit is damaged; Deep fusion features The input damage type classification branch uses the same network structure as the location recognition branch, and the number of neurons in the output layer equals the total number of damage types. The probability distribution of each damage type is output using the Softmax activation function. : ; The loss function for the type classification branch also uses cross-entropy loss: ; in, For the first True labels for each type of damage The model predicts the first The probability of each damage type; Deep fusion features The input damage severity regression branch, with the output layer employing a Linear activation function, outputs the predicted damage severity values ​​for each unit: ; The loss function for the degree regression branch uses the mean squared error loss: ; in, For the first The actual damage level of each unit, The model predicts the first The damage level value of each unit.

6. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 5, characterized in that, The joint loss function for multiple tasks in step S3 is: ; in, , and These are the weight coefficients for the location recognition, type classification, and degree regression tasks, used to balance the importance of each task in joint training. Through end-to-end backpropagation optimization, the three tasks share the underlying deep fusion features. At the same time, each task-specific network parameter is optimized to achieve multi-task collaborative learning.

7. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 6, characterized in that, In step S4, the modal dropout training strategy, during model training, uses a preset dropout probability. Randomly mask the input data of a certain modality, specifically: For each training sample, three binary masks are generated independently. These correspond to vibration, visual, and textual modalities, respectively, with each mask representing a probability. The value is 1, with probability A value of 0 sets the input feature tensor of the discarded mode to a zero tensor. ; The masked feature tensor is input into the subsequent S2, allowing the model to face multiple input scenarios simultaneously during training and learn robust feature representations that do not depend on any specific modality.

8. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 7, characterized in that, In step S4, the knowledge distillation mechanism uses the prediction results of the multi-task joint learning model in step S3 under trimodal input as the teacher output, and the prediction results of the corresponding prediction head under single-modal or bimodal input as the student output. For each diagnostic task... Calculate the distillation loss between teacher and student outputs. For classification tasks, use KL divergence with a temperature coefficient. ; ; in, and Teacher networks and student networks in the task The logical output above, For temperature coefficient, The KL divergence is used to measure the difference between two probability distributions; for regression tasks, the distillation loss uses the mean squared error. ; in, and These represent the impairment values ​​predicted by the teacher network and the student network, respectively. The joint optimization of the multi-task learning objective and the knowledge distillation objective combines the multi-task joint loss and the cross-modal knowledge distillation loss to obtain the complete total loss function for model training. ; in, For the task Distillation loss weighting coefficient.

9. The building damage diagnosis method integrating multimodal sensing and deep learning according to claim 8, characterized in that, Step S5 is as follows: S51. Define the structure type or operating condition with sufficient existing labeled data as the source domain. Define the new structural type or new operating condition to be diagnosed as the target domain. , The number of samples in the source domain. The number of samples in the target domain; The maximum mean difference is used as a measure of distribution distance. In the regenerating kernel Hilbert space, the maximum mean difference distance between the deep fused features of the source and target domains is: ; in, For the feature mapping induced by the kernel function, Let Hilbert be the norm in the regenerated kernel space. and The source domain is respectively The nth sample and the target domain Deep fusion features of individual samples; S52, End-of-line access area discriminator in step S2 It is used to determine whether the input features come from the source domain or the target domain, and adversarial training is achieved through a gradient reversal layer: ; in, For the first Domain labels for each sample, The predicted probability for the neighborhood discriminator; S53. Combining the multi-task joint loss, knowledge distillation loss, and domain adversarial loss yields the total loss function for cross-domain transfer learning: ; in, The weighting coefficients for domain adversarial loss.

10. A building damage diagnosis system integrating multimodal sensing and deep learning, applied to the building damage diagnosis method integrating multimodal sensing and deep learning as described in any one of claims 1-9, characterized in that, include: The multimodal data acquisition and preprocessing module simultaneously acquires vibration response signals, appearance images, and on-site description text of the building structure, and preprocesses various types of data. The multimodal feature extraction and interactive fusion module extracts deep features from multimodal data and achieves deep fusion of modal features through a bidirectional cross-attention mechanism. The multi-task joint learning and modality adaptive diagnosis module completes three diagnostic tasks—damage location identification, damage type classification, and damage severity regression—based on deep fusion features. The modality adaptation and knowledge distillation enhancement module enables the model to maintain its diagnostic capabilities under modality loss conditions through modality dropout training and cross-modality knowledge distillation. The cross-condition transfer learning module transfers diagnostic knowledge learned from known structures to unknown structures through domain adaptation technology.