Bridge structure disease evolution prediction method and system based on double-flow neural network

Abstract

The invention relates to the technical field of bridge structure disease prediction, in particular to a bridge structure disease evolution prediction method and system based on a double-flow neural network, and the method comprises the steps: obtaining: obtaining multi-modal data of a bridge structure, the multi-modal data comprising image data, point cloud data and sensor data; a processing step: constructing a double-flow neural network model based on the multi-modal data; according to the double-flow neural network model, generating a bridge structure disease evolution prediction result; and an output step, including outputting the bridge structure disease evolution prediction result, realizing deep fusion of image data and point cloud data through a double-flow neural network structure, and making full use of complementary advantages of different data modals. Image data provides visual information of bridge surface diseases, and point cloud data can accurately capture geometric deformation of the structure. The multi-modal fusion greatly improves the comprehensiveness and accuracy of disease detection and evolution prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge structure disease prediction, and more specifically, to a method and system for predicting the evolution of bridge structure diseases based on a dual-stream neural network. Background Art

[0002] With the rapid development of infrastructure construction in China, the importance of bridges as key transportation hubs has become increasingly prominent. However, the problems of bridge structure diseases have become increasingly serious due to long-term use and the influence of harsh environments, threatening public safety and economic development. Therefore, accurately predicting the evolution trend of bridge structure diseases has become a hot research topic in the field of bridge health monitoring.

[0003] Traditional methods for predicting bridge structure diseases mainly rely on regular manual inspections and simple statistical models. Although these methods are simple and intuitive, they have disadvantages such as strong subjectivity, low efficiency, and difficulty in capturing complex disease evolution patterns. With the progress of technology, some more advanced methods have been introduced into this field. For example, methods based on finite element analysis can simulate the mechanical behavior of bridge structures, but their accuracy highly depends on the accuracy of the model and the setting of boundary conditions, and it is difficult to adapt to complex and changing actual environments.

[0004] In recent years, with the rapid development of artificial intelligence technology, machine learning methods have begun to stand out in the prediction of bridge structure diseases. Among them, deep learning methods have received extensive attention due to their powerful feature extraction and pattern recognition capabilities. However, existing deep learning-based methods still have some obvious deficiencies. First, most methods only focus on a single data modality, such as only using image data or sensor data, and it is difficult to comprehensively capture the health status of bridge structures. Second, these methods are often purely data-driven and lack consideration of the mechanical principles of bridge structures, resulting in prediction results that may violate physical laws. Moreover, existing methods generally lack quantification of the uncertainty of prediction results, which may lead to biases in risk assessment in actual decision-making.

[0005] The closest prior art attempts to improve prediction accuracy by fusing multi-source data, but its data fusion method is often simple feature splicing or weighted averaging, and fails to fully utilize the complementarity of different data modalities. In addition, these methods have poor adaptability when facing newly emerging disease types or changes in environmental conditions, and it is difficult to perform effective continuous learning and updating. Summary of the Invention

[0006] In view of the above problems, the present invention proposes a method and system for predicting the evolution of bridge structure diseases based on a dual-stream neural network. The method aims to achieve more accurate, reliable, and continuously learning-capable prediction of the evolution of bridge structure diseases through innovative multi-modal data fusion, introduction of physical constraints, uncertainty quantification, and dynamic adaptation mechanisms.

[0007] The present invention provides a method for predicting the disease evolution of a bridge structure based on a dual-stream neural network, including:

[0008] An acquisition step, including:

[0009] Acquire multimodal data of the bridge structure, where the multimodal data includes image data, point cloud data, and sensor data;

[0010] A processing step, including:

[0011] Based on the multimodal data, construct a dual-stream neural network model;

[0012] According to the dual-stream neural network model, generate a prediction result for the disease evolution of the bridge structure;

[0013] An output step, including:

[0014] Output the prediction result for the disease evolution of the bridge structure.

[0015] Preferably, the acquisition step specifically includes:

[0016] Obtain high-resolution image data of the bridge structure through fixed cameras and mobile drones;

[0017] Use vehicle-mounted lidar and fixed laser scanners to obtain high-precision point cloud data of the bridge structure;

[0018] Use various types of sensors arranged on the bridge structure to obtain real-time dynamic response data.

[0019] Preferably, before the processing step, there is also a data preprocessing step:

[0020] Based on GPS and inertial measurement units, perform spatio-temporal synchronization on the multimodal data;

[0021] Use feature point matching and geometric constraints to perform adaptive registration on the multimodal data;

[0022] Based on wavelet transform and autoencoders, perform multi-scale noise suppression on the multimodal data.

[0023] Preferably, the construction of the dual-stream neural network model includes:

[0024] Construct an image stream branch, where the image stream branch includes a multi-scale feature extraction network and a temporal attention module;

[0025] Construct a point cloud stream branch, where the point cloud stream branch includes a dynamic graph convolutional network and a voxelization-inverse voxelization structure;

[0026] Construct a multimodal information interaction module, where the multimodal information interaction module includes a cross-attention mechanism and an adaptive feature fusion strategy.

[0027] Preferably, the multi-scale feature extraction network includes dilated convolution and a pyramid pooling structure; the temporal attention module is implemented based on a transformer; the dynamic graph convolutional network adapts to the irregular structure of point cloud data.

[0028] Preferably, in the processing step, it further includes an introduction of physical constraints step:

[0029] Based on the laws of material mechanics, add a stress-strain relationship constraint term to the loss function;

[0030] Based on the structural dynamics equation, add a structural dynamic response consistency constraint;

[0031] Establish a coupling model between environmental factors and disease evolution, and introduce it as a soft constraint into the network.

[0032] Preferably, it further includes a knowledge embedding step:

[0033] Encode engineering experience and expert knowledge into a rule set, and embed it into the feature extraction process through an attention mechanism;

[0034] Construct a bridge disease knowledge graph, and integrate prior knowledge into the prediction model through a graph neural network.

[0035] Preferably, it further includes an uncertainty quantification step:

[0036] Introduce Bayesian inference at the key layers of the neural network to quantify the uncertainty of model prediction;

[0037] Based on the bootstrap method, provide a confidence interval for disease evolution prediction.

[0038] Preferably, it further includes a dynamic adaptation and continuous learning step:

[0039] Based on importance sampling, achieve parameter-level incremental updates;

[0040] Use a neural architecture search algorithm to dynamically adjust the network structure according to new data;

[0041] Based on the uncertainty of model prediction, implement an active learning strategy to select the most informative samples for annotation and learning.

[0042] A bridge structure disease evolution prediction system based on a two-stream neural network adopting the above method, includes:

[0043] A data acquisition module, used to obtain multimodal data of the bridge structure, where the multimodal data includes image data, point cloud data, and sensor data;

[0044] A data preprocessing module for spatio-temporal synchronization, adaptive registration, and multi-scale noise suppression of the multi-modal data;

[0045] A two-stream neural network model construction module for constructing a two-stream neural network model including an image stream branch, a point cloud stream branch, and a multi-modal information interaction module;

[0046] A physical constraint module for introducing physical constraints based on material mechanics, structural dynamics, and environmental factors into the neural network;

[0047] A knowledge embedding module for integrating engineering experience, expert knowledge, and a bridge disease knowledge graph into the prediction model;

[0048] An uncertainty quantification module for quantifying the uncertainty of the prediction results through Bayesian inference and the bootstrap method;

[0049] A dynamic adaptation and continuous learning module for realizing parameter-level incremental updates, structural dynamic adjustments, and active learning of the model;

[0050] A prediction result output module for generating and outputting the prediction results of the evolution of bridge structure diseases.

[0051] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0052] First, through the two-stream neural network structure, the method realizes the deep fusion of image data and point cloud data, making full use of the complementary advantages of different data modalities. Image data provides intuitive information about bridge surface diseases, while point cloud data can accurately capture the geometric deformation of the structure. This multi-modal fusion greatly improves the comprehensiveness and accuracy of disease detection and evolution prediction.

[0053] Second, the present invention introduces a physical constraint mechanism, organically combining traditional mechanical models with deep learning methods. This not only ensures that the prediction results conform to physical laws but also improves the interpretability and credibility of the model. For example, by introducing material mechanics and structural dynamics constraints, the model can better understand and predict complex disease evolution processes such as crack propagation and deformation accumulation.

[0054] Furthermore, the uncertainty quantification mechanism of the present invention provides a reliable confidence interval for the prediction results. This is of great significance for actual bridge maintenance decisions, as it allows engineers to adjust maintenance strategies according to the uncertainty of the prediction, thereby more effectively allocating resources and managing risks.

[0055] In addition, the dynamic adaptation and continuous learning mechanism of the present invention enables the model to continuously self-optimize and adapt to new data distributions and disease patterns. This characteristic greatly improves the long-term effectiveness and practicality of the model, enabling it to handle various complex situations that the bridge structure may encounter throughout its life cycle.

[0056] Finally, through the synergistic effect of multiple innovative points, the method of the present invention achieves a comprehensive improvement in prediction accuracy, reliability, adaptability, and practicality. For example, the combination of multi-modal data fusion and physical constraints not only improves the prediction accuracy but also enhances the generalization ability of the model. The combination of uncertainty quantification and dynamic adaptation mechanism provides the model with the ability of self-assessment and continuous improvement. This systematic innovation gives the present invention significant advantages at both the theoretical and practical levels, providing a comprehensive, efficient, and reliable technical solution for bridge structure health monitoring and maintenance management. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a logic block diagram of the method of the present invention.

[0058] Figure 2 It is a flow chart of constructing a dual-stream neural network model of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0059] Please refer to Figure 1-2 , the present invention provides a method and system for predicting the disease evolution of bridge structures based on a dual-stream neural network. This method realizes high-precision prediction of the disease evolution of bridge structures by innovatively integrating multi-modal data and deep learning techniques.

[0060] First, this method includes an acquisition step for acquiring multi-modal data of the bridge structure. These multi-modal data include image data, point cloud data, and sensor data. Image data can intuitively reflect the disease conditions on the surface of the bridge, point cloud data can accurately capture the three-dimensional geometric features of the bridge structure, and sensor data provides real-time dynamic response information of the bridge structure. By comprehensively using these different types of data, this method can comprehensively depict the health status of the bridge structure.

[0061] Next, the processing step of this method includes constructing a dual-stream neural network model based on the acquired multi-modal data. This dual-stream neural network model is one of the core innovations of the present invention. It includes two main processing branches: one for processing image data and the other for processing point cloud data. This design fully considers the characteristics of different types of data and can more effectively extract their respective features.

[0062] After constructing the dual-stream neural network model, this method generates the prediction results of the bridge structure disease evolution according to the model. This prediction result not only includes the current disease state but also the possible development trend of the disease in a period of time in the future. This has important guiding significance for the preventive maintenance and management of bridges.

[0063] Finally, this method outputs the prediction results of the bridge structure disease evolution. This output step can adopt various forms, such as visualization charts, numerical reports or warning information, etc., so as to facilitate engineers to intuitively understand and use the prediction results.

[0064] Preferably, in one embodiment of the present invention, the acquisition step can be described more specifically as: obtaining high-resolution image data of the bridge structure through fixed cameras and mobile drones; obtaining high-precision point cloud data of the bridge structure by using vehicle-mounted lidar and fixed laser scanners; obtaining real-time dynamic response data by using multi-type sensors arranged on the bridge structure.

[0065] This multi-source data acquisition method has significant advantages. Fixed cameras can continuously monitor the key parts of the bridge, while mobile drones can flexibly obtain images of areas that are difficult to access. Vehicle-mounted lidar is suitable for quickly scanning the entire bridge structure, while fixed laser scanners can perform high-precision scanning on specific areas. The use of multi-type sensors ensures that the dynamic response characteristics of the bridge structure can be comprehensively captured.

[0066] In another embodiment of the present invention, before the processing step, there is also a data preprocessing step. This step specifically includes: performing spatio-temporal synchronization on the multi-modal data based on GPS and inertial measurement units; performing adaptive registration on the multi-modal data by using feature point matching and geometric constraints; performing multi-scale noise suppression on the multi-modal data based on wavelet transform and autoencoders.

[0067] The introduction of the data preprocessing step greatly improves the accuracy and efficiency of subsequent processing. Spatio-temporal synchronization ensures the consistency of data from different sources in time and space, which is crucial for accurately analyzing the dynamic characteristics of the bridge structure. Adaptive registration solves the alignment problem between different modal data, enabling images, point clouds and sensor data to accurately correspond. Multi-scale noise suppression effectively improves the quality of data, especially for data collected in complex environments.

[0068] For example, when performing spatio-temporal synchronization, a high-precision GPS receiver and an inertial measurement unit (IMU) can be used. The accuracy of the GPS receiver can reach the centimeter level, and the sampling rate of the IMU can be as high as 1000Hz. Through the Kalman filtering algorithm, the data of GPS and IMU can be fused to obtain more accurate spatio-temporal information.

[0069] During the adaptive registration process, the SIFT (Scale-Invariant Feature Transform) algorithm can be used to extract feature points from image and point cloud data. The advantage of the SIFT algorithm lies in its good robustness to scale changes, rotation, and illumination changes. For the extracted feature points, the RANSAC (Random Sample Consensus) algorithm can be used for matching to remove incorrect matching points. The threshold of the RANSAC algorithm can be set to 0.5, which means that at least 50% of the matching points need to conform to geometric constraints to be considered correct matches.

[0070] When performing multi-scale noise suppression, a method combining wavelet transform and autoencoder can be adopted. First, the Daubechies wavelet is used to perform multi-scale decomposition on the data, and the decomposition level is usually selected to be 3 - 5 layers. Then, the coefficients of each scale are denoised using an autoencoder. The number of hidden layer nodes of the autoencoder can be set to 50% - 75% of the input dimension, so that noise can be effectively removed while retaining the main information.

[0071] Through these detailed data preprocessing steps, the present invention can effectively improve the training effect and prediction accuracy of the subsequent dual-stream neural network model. The combined use of these preprocessing techniques not only solves the problem of multi-modal data fusion, but also provides high-quality input data for the prediction of bridge structure disease evolution, thus laying a solid foundation for achieving high-precision prediction results. In the core processing steps of the present invention, the construction of the dual-stream neural network model is a crucial link. The design of this model fully considers the different characteristics of image data and point cloud data, and realizes the effective processing and fusion of multi-modal data through an innovative network structure.

[0072] Specifically, the dual-stream neural network model of the present invention includes three main parts: an image stream branch, a point cloud stream branch, and a multi-modal information interaction module. The image stream branch is mainly responsible for processing high-resolution image data obtained from fixed cameras and mobile drones. This branch includes a multi-scale feature extraction network and a temporal attention module. The point cloud stream branch is specifically used to process high-precision point cloud data from vehicle-mounted lidar and fixed laser scanners, and it includes a dynamic graph convolutional network and a voxelization - inverse voxelization structure. The multi-modal information interaction module serves as a bridge to achieve the deep fusion of image features and point cloud features.

[0073] In the image stream branch, the design of the multi-scale feature extraction network is particularly crucial. This network adopts dilated convolution and pyramid pooling structures, enabling the model to capture the disease characteristics of bridge structures at different scales. For example, for subtle diseases such as cracks, a smaller receptive field may be required for precise localization; while for larger-scale diseases such as concrete spalling, a larger receptive field is needed to comprehensively grasp the situation. By adjusting the dilation rate of dilated convolution, this method can flexibly adjust the size of the receptive field. Preferably, three dilated convolution layers with different dilation rates can be set, with the dilation rates being 1, 2, and 4 respectively, which can effectively expand the receptive field range while maintaining computational efficiency.

[0074] The introduction of the temporal attention module further enhances the model's ability to capture the temporal characteristics of disease evolution. This module is implemented based on the transformer architecture and can effectively model long-term dependencies. In practical applications, 8 attention heads can be set, and the hidden layer dimension can be 512, which can fully capture the disease evolution patterns at different time scales.

[0075] In the point cloud stream branch, the use of the dynamic graph convolutional network is a major innovation point. Traditional convolutional neural networks are difficult to directly process irregular point cloud data, while the dynamic graph convolutional network well solves this problem by dynamically constructing the connection relationships between points. In specific implementation, the K-nearest neighbor algorithm can be used to construct the graph structure, and the value of K can be set to 20, which can ensure sufficient local information without causing too high computational complexity.

[0076] The introduction of the voxelization - inverse voxelization structure balances computational efficiency and accuracy. By converting point cloud data into regular voxel grids, the processing speed can be greatly improved. Preferably, the voxel size can be set to 0.1m×0.1m×0.1m, which can significantly reduce computational complexity while retaining sufficient details. In the inverse voxelization process, trilinear interpolation can be used to effectively restore the fine structure of the point cloud.

[0077] The multi-modal information interaction module is the key to achieving deep fusion of image and point cloud features. This module includes a cross-attention mechanism and an adaptive feature fusion strategy. The cross-attention mechanism allows the features of the two modalities to pay attention to and enhance each other. In implementation, scaled dot-product attention can be used, and the scaling factor can be set to the square root of 8, which can stabilize the training process. The adaptive feature fusion strategy realizes more intelligent feature fusion by learning the importance weights of different modality features.

[0078] In a preferred embodiment of the present invention, in order to further improve the reliability and physical rationality of prediction, physical constraints are introduced in the processing steps. These physical constraints are based on the laws of material mechanics, structural dynamics equations, and the coupling relationship between environmental factors and disease evolution, effectively integrating domain knowledge into the deep learning model.

[0079] Specifically, based on the laws of material mechanics, a stress-strain relationship constraint term is added to the loss function. This can be expressed as:

[0080]

[0081] where σ i and ∈ i respectively represent the predicted values of stress and strain at the i-th point, E is the elastic modulus, and α is the weight coefficient. Through this constraint term, the prediction results of the model will be more in line with the mechanical properties of the material. Preferably, α can be set to 0.1, which can ensure the physical rationality of the prediction results while not overly affecting the flexibility of the model.

[0082] Based on the structural dynamics equations, a structural dynamic response consistency constraint is added. This can be expressed as:

[0083]

[0084] where M, C, and K are the mass matrix, damping matrix, and stiffness matrix respectively, u is the displacement vector, F is the external force vector, and β is the weight coefficient. This constraint ensures that the predicted structural response conforms to the dynamics equations. In practical applications, β can be set to 0.01, which can ensure the prediction accuracy without overly constraining the learning ability of the model.

[0085] In addition, the method of the present invention also establishes a coupling model between environmental factors and disease evolution and introduces it into the network as a soft constraint. This can be expressed as:

[0086] L environment = γ||f(T, H, L - ΔD|| 2

[0087] where f(T, H, L represents the influence function of temperature T, humidity H, and load L on the disease increment ΔD, and γ is the weight coefficient. This constraint takes into account the influence of environmental factors on disease evolution, making the prediction results closer to the actual situation. Preferably, γ can be set to 0.05, which can appropriately consider the influence of environmental factors without overly relying on the environmental model.

[0088] By introducing these physical constraints, the method of the present invention not only improves the prediction accuracy but also enhances the interpretability and credibility of the model, which is particularly important for the management of critical infrastructure such as bridge structures.

[0089] In another embodiment of the present invention, to further improve the performance and adaptability of the model, a knowledge embedding step is introduced. This step mainly includes two aspects: encoding engineering experience and expert knowledge into a rule set, and constructing a knowledge graph of bridge diseases.

[0090] First, encode engineering experience and expert knowledge into a rule set and embed it into the feature extraction process through an attention mechanism. This can be expressed as:

[0091]

[0092] where Q, K, and V represent the query, key, and value matrices respectively, and d k is the dimension of the key. In this way, the model can dynamically adjust the attention to different features according to expert rules. For example, for concrete bridges, rules can be set to pay more attention to crack and spalling features; while for steel structure bridges, more attention can be paid to corrosion and deformation features.

[0093] Secondly, construct a knowledge graph of bridge diseases and integrate prior knowledge into the prediction model through a graph neural network. The knowledge graph can be represented as a set of triples:

[0094]

[0095] where h and t represent the head entity and tail entity respectively, and r represents the relationship between them. Through the Graph Attention Network (GAT), the knowledge graph information can be integrated into the feature representation:

[0096]

[0097] where h i ′ represents the updated feature of node i, α ij represents the attention coefficient, W is the weight matrix, represents the set of neighbors of node i.

[0098] Through these knowledge embedding techniques, the method of the present invention can effectively integrate the experience of domain experts and systematic prior knowledge into the deep learning model, thereby improving the interpretability and generalization ability of the model. This is particularly important for dealing with complex bridge structure disease evolution prediction problems, because it can help the model better understand and predict the development trends of various disease types and perform well even in the case of limited training data. In a further embodiment of the present invention, to more comprehensively evaluate the reliability of the prediction results, an uncertainty quantification step is introduced. This step mainly includes two aspects: introducing Bayesian inference in the key layers of the neural network, and providing a confidence interval for disease evolution prediction based on the bootstrap method.

[0099] First, introducing Bayesian inference into the key layers of the neural network can quantify the uncertainty of model predictions. Specifically, the weights of the neural network can be regarded as random variables and assumed to follow a Gaussian distribution. In this way, the output of the network will also be a distribution rather than a single value. Mathematically, this can be expressed as:

[0100] p(y|x,D)=∫p(y|x,w)p(w|D)dw

[0101] where y is the predicted output, x is the input, w is the network weight, and D is the training data. p(w|D) represents the posterior distribution of the weights and can be approximated by methods such as variational inference. In this way, the model can not only give a point estimate but also provide an estimate of the uncertainty of the prediction.

[0102] Preferably, Bayesian inference can be introduced into the last few fully connected layers of the network. This can effectively capture the uncertainty of the model while maintaining computational efficiency. For the prior distribution of the weights, a Gaussian distribution with a mean of 0 and a standard deviation of 0.1 can be selected, which can avoid overfitting while providing sufficient flexibility.

[0103] Secondly, the bootstrap method is used to provide a confidence interval for the prediction of disease evolution. The bootstrap method generates multiple sub-datasets by sampling from the original dataset with replacement, and then trains multiple models on these sub-datasets. The final prediction result is the integration of the predictions of these models. The confidence interval can be obtained by calculating the quantiles of these predictions. Mathematically, this can be expressed as:

[0104]

[0105] where and represent the α / 2 and 1 - α / 2 quantiles of the prediction result respectively. α is usually set to 0.05, corresponding to a 95% confidence interval.

[0106] In practical applications, 100 bootstrap samples can be generated, which can control the computational cost while ensuring statistical reliability. For each sample, a complete two-stream neural network model can be trained, and the final prediction result and uncertainty estimate will be based on the outputs of these 100 models.

[0107] By introducing these uncertainty quantification techniques, the method of the present invention can not only provide accurate predictions of disease evolution but also evaluate the reliability of the prediction results. This is particularly important for bridge maintenance decisions, as it can help engineers better understand the credibility of the prediction results and thus formulate more reasonable maintenance strategies.

[0108] In another embodiment of the present invention, to enable the model to continuously adapt to new data and new disease patterns, a dynamic adaptation and continuous learning step is introduced. This step mainly includes three aspects: parameter-level incremental update, dynamic adjustment of network structure, and active learning strategy.

[0109] First, parameter-level incremental update is achieved based on importance sampling. When new data arrives, instead of retraining the entire model, the model can adapt to the new data by updating some important parameters. Specifically, the Fisher information matrix can be used to measure the importance of parameters:

[0110]

[0111] where F i represents the Fisher information of the i-th parameter, and θ represents the model parameters. During the update, more important parameters will be adjusted more, while less important parameters will remain basically unchanged. This method can quickly adapt to new data while maintaining the model's performance on old tasks.

[0112] Preferably, a threshold can be set. For example, the top 20% of the parameters with respect to Fisher information are regarded as important parameters and updated. This can achieve a good balance between adaptability and computational efficiency.

[0113] Secondly, using the neural architecture search (NAS) algorithm, the network structure is dynamically adjusted according to new data. NAS can automatically design the network structure that is most suitable for the current data distribution. In the present invention, an NAS method based on reinforcement learning can be adopted, and its objective function can be expressed as:

[0114] J(θ) = E p(a；θ) [R]

[0115] where a represents the network structure, θ represents the controller parameters, and R represents the reward signal of the network performance. By optimizing this objective function, the optimal network structure can be found.

[0116] In practical applications, network structure search can be performed every certain period (such as every 3 months) to adapt to possible changes in data distribution. The search space can include hyperparameters such as the number of layers, kernel size, activation function, etc.

[0117] Finally, based on the uncertainty of model prediction, an active learning strategy is implemented to select the most informative samples for annotation and learning. This can be achieved by maximizing the expected information gain:

[0118]

[0119] Among them, H represents entropy, and x* is the most informative sample selected. In this way, the model can actively select the samples that are most helpful for improving performance for learning, thereby making more effective use of the annotation resources.

[0120] In practice, the top-k samples with the highest prediction uncertainty can be selected each time for manual annotation and model update. The value of k can be dynamically adjusted according to the available annotation resources. For example, it can be set to 1% of the current dataset size.

[0121] By introducing these dynamic adaptation and continuous learning techniques, the method of the present invention can continuously self-optimize and evolve to adapt to new patterns and new challenges that may appear during the evolution of bridge structure diseases. This is of great significance for long-term bridge health monitoring and maintenance management, because it can ensure that the prediction model always maintains high performance and high reliability.

[0122] Finally, the present invention also provides a bridge structure disease evolution prediction system based on a two-stream neural network. The system includes multiple functional modules, and each module is optimized and designed for specific tasks in the prediction process.

[0123] The data acquisition module 1 is responsible for acquiring multi-modal data of the bridge structure, including image data, point cloud data, and sensor data. This module can integrate various advanced data acquisition devices, such as high-resolution cameras, laser scanners, and various sensors. Preferably, this module can also include functions of data transmission and preliminary processing, such as data compression and encryption, to ensure the safe and efficient transmission of data.

[0124] The data preprocessing module 2 performs spatio-temporal synchronization, adaptive registration, and multi-scale noise suppression on the acquired multi-modal data. The design of this module fully considers the characteristics of different types of data and adopts a series of advanced signal processing algorithms. For example, spatio-temporal synchronization can use an extended Kalman filter, adaptive registration can combine the RANSAC algorithm and the ICP (Iterative Closest Point) algorithm, and multi-scale noise suppression can adopt a method combining wavelet transform and deep learning.

[0125] The two-stream neural network model construction module 3 is the core of the system and is responsible for constructing a two-stream neural network model including an image stream branch, a point cloud stream branch, and a multi-modal information interaction module. The design of this module has high flexibility and can be adjusted according to the specific bridge type and monitoring requirements. For example, for long-span bridges, more attention layers can be added to capture long-distance dependencies; while for multi-span continuous bridges, a recurrent neural network can be introduced to model the cross-span mutual influence.

[0126] The physical constraint module 4 introduces physical constraints based on material mechanics, structural dynamics, and environmental factors into the neural network. The design of this module embodies the innovative idea of combining deep learning with traditional mechanical models in this system. It not only contains the mathematical expressions of various physical constraints but also designs the fusion mechanism of these constraints with the neural network, such as achieving end-to-end training through a differentiable physical simulation layer.

[0127] The knowledge embedding module 5 is responsible for integrating engineering experience, expert knowledge, and the bridge disease knowledge graph into the prediction model. This module can include a knowledge base management system for storing and updating various domain knowledge, as well as a knowledge-model converter for converting symbolic knowledge into a form directly usable by the neural network.

[0128] The uncertainty quantification module 6 quantifies the uncertainty of the prediction results through Bayesian inference and the bootstrap method. This module not only provides the calculation method of uncertainty but also includes the visualization function of the results, enabling engineers to intuitively understand the reliability of the prediction.

[0129] The dynamic adaptation and continuous learning module 7 is used to achieve parameter-level incremental updates, structural dynamic adjustments, and active learning of the model. This module can be designed as an autonomous learning system that can automatically detect the arrival of new data, evaluate the performance of the current model, and decide when and how to update the model.

[0130] The prediction result output module 8 is responsible for generating and outputting the prediction results of the bridge structure disease evolution. In addition to providing numerical prediction results, this module can also integrate various visualization tools, such as heat maps, time series graphs, etc., as well as decision support functions, such as a maintenance recommendation generator, etc.

[0131] Through these carefully designed modules, the system of the present invention can comprehensively, efficiently, and accurately achieve the prediction of the bridge structure disease evolution, providing strong technical support for the health monitoring and maintenance management of bridges. The modular design of this system also makes it have good scalability and adaptability, and can be flexibly configured and optimized according to different application scenarios.

[0132] It should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A bridge structure defect evolution prediction method based on a dual-stream neural network is characterized by: include: The acquisition steps include: Acquire multimodal data of the bridge structure, wherein the multimodal data includes image data, point cloud data, and sensor data; Processing steps include: Based on the multimodal data, construct a two-stream neural network model; Generate bridge structure disease evolution prediction results according to the dual-stream neural network model; Output steps include: Output the prediction result of the bridge structure disease evolution.

2. The method according to claim 1, characterized in that The acquisition step specifically includes: Obtain high-resolution image data of bridge structures through fixed cameras and mobile drones; Use vehicle-mounted laser radar and fixed laser scanner to obtain high-precision point cloud data of bridge structures; Real-time dynamic response data are obtained using multiple types of sensors arranged on the bridge structure.

3. The method according to claim 1, characterized in that Prior to the processing steps, a data preprocessing step is also included: Based on GPS and an inertial measurement unit, the multimodal data is synchronized in time and space; Adaptively registering the multimodal data using feature point matching and geometric constraints; Based on wavelet transform and autoencoder, multi-scale noise suppression is performed on the multimodal data.

4. The method according to claim 1, characterized in that The construction of the two-stream neural network model includes: Constructing an image stream branch, wherein the image stream branch includes a multi-scale feature extraction network and a temporal attention module; Constructing a point cloud stream branch, wherein the point cloud stream branch includes a dynamic graph convolutional network and a voxelization-devoxelization structure; A multimodal information interaction module is constructed, wherein the multimodal information interaction module includes a cross-attention mechanism and an adaptive feature fusion strategy.

5. The method according to claim 4, characterized in that The multi-scale feature extraction network includes a dilated convolution and a pyramid pooling structure; the temporal attention module is implemented based on a transformer; and the dynamic graph convolution network adapts to the irregular structure of point cloud data.

6. The method according to claim 1, characterized in that The processing steps also include introducing a physical constraint step: Based on the laws of material mechanics, a stress-strain relationship constraint term is added to the loss function; Based on the structural dynamics equation, the structural dynamic response consistency constraint is added; A coupling model of environmental factors and disease evolution was established and introduced into the network as soft constraints.

7. The method according to claim 1, characterized in that It also includes the knowledge embedding step: Encode engineering experience and expert knowledge into a set of rules and embed them into the feature extraction process through an attention mechanism; Construct a knowledge graph of bridge defects and integrate prior knowledge into the prediction model through graph neural network.

8. The method according to claim 1, characterized in that: It also includes the uncertainty quantification step: Introducing Bayesian inference at key layers of neural networks to quantify the uncertainty of model predictions; Based on the bootstrap method, confidence intervals are provided for disease evolution prediction.

9. The method according to claim 1, characterized in that: It also includes dynamic adaptation and continuous learning steps: Based on importance sampling, parameter-level incremental update is achieved; Use neural architecture search algorithms to dynamically adjust network structures based on new data; Based on the uncertainty of model predictions, an active learning strategy is implemented to select the most informative samples for annotation and learning.

10. A bridge structure disease evolution prediction system based on a dual-stream neural network using the method according to any one of claims 1 to 9, characterized in that: include: A data acquisition module, used to acquire multimodal data of the bridge structure, wherein the multimodal data includes image data, point cloud data and sensor data; A data preprocessing module, used for performing spatiotemporal synchronization, adaptive registration and multi-scale noise suppression on the multimodal data; A two-stream neural network model building module, used to build a two-stream neural network model including an image stream branch, a point cloud stream branch and a multimodal information interaction module; Physical constraint module, which is used to introduce physical constraints based on material mechanics, structural dynamics and environmental factors into neural networks; Knowledge embedding module, which is used to integrate engineering experience, expert knowledge and bridge disease knowledge graph into the prediction model; Uncertainty quantification module, used to quantify the uncertainty of prediction results through Bayesian inference and bootstrap methods; Dynamic adaptation and continuous learning module, used to achieve parameter-level incremental update, dynamic structural adjustment and active learning of the model; The prediction result output module is used to generate and output the prediction results of bridge structure disease evolution.

Images