Multi-modal analysis system for monitoring data of historical building structure
By designing a multimodal analysis system, integrating information acquisition, abnormal diagnosis, inversion and prediction modules, the complexity of multimodal data processing in historical building structure monitoring data processing is solved, and the comprehensiveness and simplification of data processing is achieved. Improved overall capability of monitoring data processing.
Patent Information
- Application Number
- PCT/CN2024/089947
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-21
AI Technical Summary
The existing technology cannot fully process historical building structure monitoring data, and lacks a unified multimodal data processing method, which leads to the complex and difficult process of monitoring data processing, and cannot solve multiple problems at the same time, such as exception repair and data modeling.
A multimodal analysis system for monitoring data of historical building structures was designed, including information acquisition and preprocessing modules, abnormal diagnosis and repair modules, response inversion modules, response prediction modules and multimodal integration modules, and these are integrated using multimodal networks. module to realize systematic processing of data.
The system integrates various processes of monitoring data processing into a unified system, simplifies the data processing process, improves the overall ability of monitoring data processing of historical building structures, and can comprehensively solve the analysis of multiple processes such as abnormal diagnosis, data inversion and response prediction. need.
Smart Images

Figure CN2024089947_21112024_PF_FP_ABST
Abstract
Description
Multimodal Analysis System for Historical Building Structural Monitoring Data Technical Field
[0001] The invention relates to a multimodal analysis system for historical building structure monitoring data. Background Art
[0002] As time goes by, the structure and materials of historical buildings may be affected by various factors such as nature, environment, and human factors, resulting in damage such as cracks, deformation, corrosion, and falling off. Monitoring the displacement, stress, and strain of historical building structures and analyzing the data are of great guiding significance for the protection of historical buildings.
[0003] Traditional monitoring data processing methods can only solve the local tasks they are responsible for, and can only proceed from a single aspect such as anomaly repair or data modeling, but cannot address the overall problem and solve multiple problems at the same time. For example: for tasks such as anomaly repair or data modeling, there are corresponding models, such as response prediction models, data anomaly repair models, etc., but there is no model that can solve multiple problems in a unified way.
[0004] Currently, there is a lack of a complete set of multimodal data processing methods that can simultaneously input information such as temperature and various structural responses and realize full-process analysis. This means that in the process of structural monitoring data processing, models must be selected for different problems, which increases the difficulty and workload of processing.
[0005] Summary of the Invention
[0006] The purpose of the present invention is to provide a multimodal analysis system for historical building structure monitoring data.
[0007] To solve the above problems, the present invention provides a multimodal analysis system for historical building structure monitoring data, comprising:
[0008] An information acquisition and preprocessing module, which is used to obtain monitoring data of historical building structures through structural health monitoring sensors installed on the historical building structures, preprocess the monitoring data, and input weather forecast information, strain information, and temperature information into different modules of the multimodal network;
[0009] The abnormality diagnosis and repair module is used to establish a structural temperature-based response prediction model for abnormality diagnosis. It uses the abnormality factor and its probability distribution that characterize the difference between the predicted value and the monitored value to diagnose and repair abnormal monitoring data.
[0010] a response inversion module, configured to establish a response inversion model by utilizing response correlations between different structural health monitoring sensors, and invert the monitoring data of the remaining sensors based on the response inversion model and the monitoring data of some sensors;
[0011] The response prediction module is used to predict the structural temperature based on the atmospheric temperature information from the weather forecast and to predict the future structural response by combining the existing historical structural response;
[0012] The multimodal integration module is used to integrate the abnormality diagnosis and repair module, the response inversion module, and the response prediction module into a multimodal network to form a multimodal network and systematically process the monitoring data.
[0013] Furthermore, in the above system, the information acquisition and preprocessing module is used to determine the type of monitoring data, the type and model of sensors, the number and distribution of sensors, and the frequency and signal transmission method of monitoring data; standardize the monitoring data to make the data a series of data with a mean of 0 and a standard deviation of 1; according to the requirements of the model establishment and training module, the weather forecast information, strain information and temperature information are input into different input positions of the multimodal network, and processed into a data structure form that conforms to the model input, so as to facilitate the multimodal network's call, extraction and analysis process of data.
[0014] Furthermore, in the above system, the monitoring data includes: structural temperature, stress and strain data and displacement data at various locations of the structure.
[0015] Furthermore, in the above system, the abnormality diagnosis and repair module is used to establish a neural network model that depicts the relationship between the structural temperature and the structural response based on the data of the structural temperature and the structural response, taking into account the sequence characteristics of the temperature-structure relationship in the monitoring data, wherein the structural response includes stress-strain data and displacement data at various locations of the structure;
[0016] The neural network model is trained using data to accurately depict the relationship between structural temperature and structural response. The input in the training process is the sequence of structural temperatures, the target output is the structural response, and the target loss function is the mean square error between the response monitoring value and the network output. The adaptive moment estimation method is used for the optimization method of neural network model training. The training of the neural network model is stopped when the target loss function reaches a preset expected value.
[0017] Different anomaly factors are established based on the network output after inputting temperature and the actual monitoring data; after the calculation of the anomaly factor is completed, the probability is used to describe the outlier degree of the anomaly factor, and a Gaussian probability distribution model is introduced; the maximum likelihood estimate of the probability distribution mean and the probability distribution variance of the anomaly factor are calculated respectively to obtain the probabilistic normal distribution model of the anomaly factor and the probability threshold of the anomaly factor corresponding to various states; based on the probability threshold of the anomaly factor, the monitoring data in the time period corresponding to the anomaly factor is diagnosed to be anomaly, and the anomaly monitoring data segment is replaced with the predicted data of the neural network model.
[0018] Furthermore, in the above system, the abnormal diagnosis and repair module is used to select the first abnormal factor as Among them, u i is the monitoring value of the structural response at each moment in a day, v i is the predicted value of the structural response model at each moment in a day, n is the number of discrete values of the structural response in a day, and the first abnormal factor represents the mean square error of the difference between the predicted structural response data and the monitored data in a day, which describes the overall difference between the predicted response and the monitored data;
[0019] Another abnormal factor is selected as in, is the average value of monitoring data response at each time of the day, It is the average value of the predicted structural response at each moment in a day. The abnormal factor is the ratio of the standard deviation of the difference between the predicted structural response data and the monitored data in a day, which describes the distribution characteristics of the predicted response and the monitored data.
[0020] Furthermore, in the above system, the abnormality diagnosis and repair module is used to describe the degree of outlier of abnormal factors by probability, and introduce the Gaussian probability distribution model Among them, μ is the mean of the probability distribution, σ 2 is the probability distribution variance;
[0021] The maximum likelihood estimates of the mean and variance of the probability distribution are The maximum likelihood estimates of the probability distribution mean and probability distribution variance of the abnormal factors α and β are calculated respectively to obtain the probability normal distribution model of the abnormal factors and the probability thresholds of the abnormal factors corresponding to various states.
[0022] Furthermore, in the above system, the response inversion module is used to train the response inversion model using monitoring data. The input in the response inversion model training process is the monitoring data of some sensors, the target output is the structural response of the remaining part, and the target loss function is the mean square error between the response monitoring value and the network output; the training optimization method uses the adaptive moment estimation method, and the training of the response inversion model is stopped until the target loss function reaches a preset expected value;
[0023] The structural responses of the remaining parts of the structure are inverted based on the structural responses of multiple sensors through the response inversion model.
[0024] Furthermore, in the above system, the response prediction module is used to establish and train a structural temperature model for the future predicted by the weather forecast, and the structural temperature model is trained using monitoring data; the input in the training process of the structural temperature model is the atmospheric temperature field data of the existing weather forecast, the target output is the corresponding structural temperature monitoring result, and the loss function is the mean square error between the actual structural temperature monitoring result and the network output; the training optimization method uses the adaptive moment estimation method, and the training of the structural temperature model is stopped until the target loss function reaches a preset expected value;
[0025] A prediction model based on structural temperature and a prediction model based on historical response are established and trained respectively. The input of the prediction model based on structural temperature is the structural temperature sequence, and the input of the prediction model based on historical response is the structural historical response sequence. The target outputs of the prediction model based on structural temperature and the prediction model based on historical response are both structural monitoring responses, and the loss function is the mean square error between the actual structural response monitoring results and the network output. The optimization method for training the prediction model based on structural temperature and the prediction model based on historical response uses adaptive moment estimation. When the target loss function reaches a preset expected value, the training of the prediction model based on structural temperature and the prediction model based on historical response is stopped.
[0026] The two response prediction models are fused; the output of the structural temperature model is used as the network input of the structural temperature-based prediction model to form the final deep learning network of the response prediction module.
[0027] Furthermore, in the above system, the response prediction module is used to take the average of the results of the prediction model based on structural temperature and the prediction model based on historical response to obtain the final prediction result of the structural response; or, the results of the prediction model based on structural temperature and the prediction model based on historical response are used as input values to establish a new prediction fusion model.
[0028] Furthermore, in the above system, the multimodal integration module is used to successively run the self-network corresponding to the abnormal diagnosis and repair module, the response inversion module and the response prediction module. After the operation is completed, the network terminates and outputs the results; in the integration process of the data output process, the abnormality occurrence time, the comparison of the abnormal repair results with the original data, the response inversion results at each location and the response prediction within the next day are successively output, and finally the integration process of the multimodal network is completed.
[0029] Compared with the current methods of processing historical building structure monitoring data based only on single aspects such as anomaly repair or data modeling, the beneficial effects of the present invention are mainly as follows:
[0030] (1) Comprehensive data processing process. This system integrates various processes in monitoring data processing into a unified system, integrating the analysis of multiple processes such as abnormal diagnosis and repair, data inversion, and response prediction, improving the overall capability of the historical building structure monitoring data processing system, and comprehensively solving various needs in the historical building monitoring data processing process.
[0031] (2) Simplify the monitoring data processing process. The multimodal network proposed in this system effectively transforms "multiple data inputs and multiple network selections" into "one-time data input and automatic network operation", simplifying the historical building monitoring data processing process. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] FIG1 is a flow chart of a multimodal analysis system for historical building structure monitoring data according to an embodiment of the present invention;
[0033] FIG2 is a schematic diagram of information acquisition and preprocessing according to an embodiment of the present invention;
[0034] FIG3 is a schematic diagram of abnormality diagnosis and repair according to an embodiment of the present invention;
[0035] FIG4 is a schematic diagram of monitoring data response inversion according to an embodiment of the present invention;
[0036] FIG5 is a schematic diagram of future response prediction according to an embodiment of the present invention;
[0037] FIG6 is a schematic diagram of multimodal integration according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] As shown in FIG1 , the present invention provides a multimodal analysis system for historical building structure monitoring data, comprising:
[0040] An information acquisition and preprocessing module is used to obtain monitoring data of historical building structures through structural health monitoring sensors installed on the historical building structures, preprocess the monitoring data, and input weather forecast information, strain information, and temperature information into different modules of the multimodal network;
[0041] The abnormality diagnosis and repair module is used to establish a structural temperature-based response prediction model for abnormality diagnosis. It uses the abnormality factor and its probability distribution that characterize the difference between the predicted value and the monitored value to diagnose and repair abnormal monitoring data.
[0042] a response inversion module, configured to establish a response inversion model by utilizing response correlations between different structural health monitoring sensors, and invert the monitoring data of the remaining sensors based on the response inversion model and the monitoring data of some sensors;
[0043] The response prediction module is used to predict the structural temperature based on the atmospheric temperature information from the weather forecast and to predict the future structural response by combining the existing historical structural response;
[0044] The multimodal integration module is used to integrate the abnormality diagnosis and repair module, the response inversion module, and the response prediction module into a multimodal network to form a multimodal network and systematically process the monitoring data.
[0045] In summary, the multimodal analysis system for historical building structure monitoring data of the present invention designs and establishes a unified, multimodal neural network method from a holistic perspective, realizes the full process processing of historical building structure health monitoring data, solves the current problem of lacking a complete set of data processing methods, realizes the integration of anomaly repair, inversion reconstruction and response prediction, and improves the overall capability of the monitoring data processing system.
[0046] Traditional monitoring data processing methods can only solve the local work they are responsible for, but cannot solve multiple problems at the same time, and have weak overall processing capabilities for monitoring data. In response to the above-mentioned problems, the present invention proposes a new multimodal analysis system for historical building structure monitoring data. Based on the multimodal neural network method of artificial intelligence, an integrated model is established to realize the analysis of multiple processes such as abnormal diagnosis and repair, data inversion, and response prediction, thereby improving the overall capabilities of the historical building structure monitoring data processing system.
[0047] Traditional monitoring data processing methods can only address a small portion of their respective tasks, focusing solely on a single area like anomaly remediation or data modeling. They cannot address multiple issues simultaneously. For example, existing methods have separate models for anomaly remediation or data modeling, such as response prediction models and data anomaly remediation models, but no single model can address multiple issues simultaneously.
[0048] In order to solve the above-mentioned problem of the lack of a complete set of multimodal data processing methods that can simultaneously input information such as temperature and various structural responses and realize full-process analysis, the present invention proposes a multimodal analysis system for historical building structure monitoring data, whose technical components are: information acquisition and preprocessing module, anomaly diagnosis and repair module, response inversion module, response prediction module, and multimodal integration module.
[0049] The overall architecture of the technical solution of the present invention is shown in FIG1 .
[0050] First, based on the finite element model and the specific needs of engineering monitoring, structural health monitoring sensors were installed on the structure to obtain historical building structure monitoring data. This monitoring data was preprocessed and weather forecast information, strain information, and temperature information were input into different locations of the multimodal network to facilitate the multimodal network's data extraction and analysis process.
[0051] Afterwards, a structural temperature-based response prediction model for anomaly diagnosis was established. Using the anomaly factor and its probability distribution that characterize the difference between the predicted and monitored values, abnormal monitoring data was diagnosed and repaired.
[0052] Furthermore, an inversion model is established by utilizing the response correlation between different structural health monitoring sensors, and based on the inversion model and the monitoring data of some sensors, the long-term missing monitoring data of the remaining sensors are inverted;
[0053] Then, the structural temperature is predicted based on the atmospheric temperature information from the weather forecast, and the future structural response is predicted by combining the existing historical structural response;
[0054] Furthermore, based on the network prediction results, the final required multimodal network output is output;
[0055] Finally, by integrating the previous modules and configuring each to automatically extract monitoring data, a multimodal monitoring data processing network model is formed. By integrating the content of each module in terms of data input, model operation sequence, and data output, a complete multimodal data processing method for historical building structure monitoring can be formed. This method integrates the analysis of multiple processes, including anomaly diagnosis and repair, data inversion, and response prediction, thereby improving the overall capabilities of the historical building structure monitoring data processing system.
[0056] As shown in FIG2 , in one embodiment of the multimodal analysis system for historical building structure monitoring data of the present invention, the information acquisition and preprocessing module is used to determine the type of monitoring data, sensor type and model, sensor quantity and distribution, and monitoring data frequency and signal transmission method; standardize the monitoring data to a series of data with a mean of 0 and a standard deviation of 1; and, in accordance with the requirements of the model building and training module, input weather forecast information, strain information, and temperature information into different input positions of the multimodal network and process them into a data structure that conforms to the model input, facilitating the multimodal network's call, extraction, and analysis process.
[0057] Here, the information acquisition and preprocessing module is mainly responsible for the collection and preprocessing of monitoring data.
[0058] Regarding the placement of monitoring sensors, the project owner or monitoring system designer will work together to determine the type of steady-state loads on the historical building to be monitored. Combined with specific project information such as the finite element model, the types of monitoring data (such as structural temperature, stress and strain information at various locations, and displacement information), sensor types and models, sensor quantity and distribution, and monitoring data frequency and signal transmission will be determined to maximize structural information acquisition at a low cost. Subsequently, the structure is monitored and pre-processed to ensure that the monitoring information meets the requirements of model training and testing before entering the subsequent modules. This primarily involves standardizing the monitoring data to a series with a mean of 0 and a standard deviation of 1, minimizing the impact of factors such as the units and fluctuation range of the monitored physical quantities on subsequent model training and prediction. Finally, according to the requirements of the model establishment and training module, weather forecast information, strain information, and temperature information are input into different input locations of the multimodal network. This data is processed into a data structure that meets the model input requirements, facilitating the multimodal network's call, extraction, and analysis processes. This completes the information acquisition and pre-processing module.
[0059] In one embodiment of the multimodal analysis system for historical building structure monitoring data of the present invention, the monitoring data includes: structural temperature, stress and strain data and displacement data at various locations of the structure.
[0060] As shown in FIG3 , in one embodiment of the multimodal analysis system for historical building structure monitoring data of the present invention, the abnormality diagnosis and repair module is used to establish a neural network model depicting the relationship between structural temperature and structural response based on data on structural temperature and structural response, taking into account the sequence characteristics of the relationship between temperature and structure in the monitoring data. The structural response includes stress, strain and displacement data at various locations on the structure.
[0061] The neural network model is trained using data to accurately depict the relationship between structural temperature and structural response. The input during training is the sequence of structural temperatures, the target output is the structural response, and the target loss function is the mean square error between the response monitoring value and the network output. The Adam (Adaptive Moment Estimation) optimization method is used for training the neural network model. Training the neural network model is stopped when the target loss function reaches a preset expected value.
[0062] Different anomaly factors are established based on the network output after inputting temperature and the actual monitoring data; after the calculation of the anomaly factor is completed, the probability can be used to describe the outlier degree of the anomaly factor, and a Gaussian probability distribution model is introduced; the maximum likelihood estimate of the probability distribution mean and the probability distribution variance of the anomaly factor are calculated respectively to obtain the probabilistic normal distribution model of the anomaly factor and the probability threshold of the anomaly factor corresponding to various states; based on the probability threshold of the anomaly factor, the monitoring data in the time period corresponding to the anomaly factor is diagnosed to be anomaly, and the anomaly monitoring data segment is replaced with the predicted data of the neural network model.
[0063] Here, the anomaly diagnosis and repair module is mainly responsible for diagnosing anomalies in monitoring data and repairing them.
[0064] This module first establishes a neural network model that depicts the relationship between structural temperature and response based on the data of structural temperature and structural response, considering the sequence characteristics of the relationship between temperature and structure in the monitoring data. The neural network model adopts LSTM (Long Short Term Memory, long short-term memory recursive neural network).
[0065] By using data for training, the neural network model can accurately depict the relationship between structural temperature and response. The input in the training process is a sequence of structural temperatures, the target output is the structural response (stress, strain and displacement data at various locations on the structure), and the target loss function is the mean square error between the response monitoring value and the network output. To speed up the training, the optimization method for the neural network model training is Adam (Adaptive Moment Estimation), and training can be stopped until the target loss function reaches a preset expected value.
[0066] After training is completed, in order to determine whether the sensor monitoring data is abnormal, different abnormal factors are established based on the network output after inputting the temperature and the actual monitoring data, such as the standard deviation, mean, and extreme value of the difference between the inversion information and the sensor data within a day.
[0067] The first abnormal factor is selected as Among them, u i is the monitoring value of the structural response at each moment in a day, v i is the predicted value of the structural response model at each moment in a day, n is the number of discrete values of the structural response in a day (n = 24 when the monitoring frequency is once an hour), and the first abnormal factor represents the mean square error of the difference between the predicted structural response data and the monitored data in a day, which describes the overall difference between the predicted response and the monitored data;
[0068] Another abnormal factor is selected as in, is the average value of monitoring data response at each time of the day, It is the average value of the predicted structural response at each moment in a day. The abnormal factor is the ratio of the standard deviation of the difference between the predicted structural response data and the monitored data in a day, which describes the distribution characteristics of the predicted response and the monitored data.
[0069] After calculating the abnormal factors, we can use probability to describe the degree of outliers of the abnormal factors and introduce the Gaussian probability distribution model. Among them, μ is the mean of the probability distribution, σ 2 is the probability distribution variance.
[0070] According to the laws of statistics, the maximum likelihood estimate of the mean and variance of the probability distribution is By calculating the maximum likelihood estimates of the probability distribution mean and probability distribution variance of the anomaly factors α and β, respectively, we can obtain a probabilistic normal distribution model for the anomaly factors and the probability of occurrence of the anomaly factors corresponding to various states. A threshold for the probability of occurrence of an anomaly factor can be determined based on actual conditions. For example, when the probability of occurrence of an anomaly factor is less than 5%, the structural monitoring data corresponding to the anomaly factor is diagnosed as abnormal. By replacing the monitoring data segments with the predicted data from the neural network model, anomalies in the historical building structural monitoring data can be repaired and the final repaired data can be output.
[0071] At this point, the abnormal diagnosis and repair module is completed.
[0072] As shown in FIG4 , in one embodiment of the multimodal analysis system for historical building structure monitoring data of the present invention, the response inversion module is used to train the response inversion model using the monitoring data. The input in the response inversion model training process is the monitoring data of some sensors, the target output is the structural response of the remaining part, and the target loss function is the mean square error between the response monitoring value and the network output.
[0073] The structural responses of the remaining parts of the structure are inverted based on the structural responses of multiple sensors through the response inversion model.
[0074] Here, the response inversion module is mainly responsible for inverting the response of the remaining parts of the structure based on the responses of multiple sensors.
[0075] First, an inversion model must be established, using a DNN (Deep Neural Network) model. Monitoring data is used to train the inversion model, enabling it to accurately invert the structural response. The input during inversion model training is the monitoring data from a subset of sensors, the target output is the structural response of the remaining components, and the target loss function is the mean squared error between the response monitoring values and the network output. The Adam method is used for training optimization, and training of the response inversion model ceases when the target loss function reaches a pre-set expected value.
[0076] After training is complete, a response inversion model for the historical building structural response monitoring data is obtained. This response inversion model inverts the structural responses of the remaining parts of the structure based on the structural responses of multiple sensors. This model can be used to invert the structural response monitoring data of historical buildings. In particular, when sensor data exhibits anomalies or sensor damage over a period of time, the model can be used to input monitoring data from other sensors during that period. The model outputs the inversion results of the sensor monitoring data during that period, serving as a supplement to the data collected when the sensor was damaged.
[0077] From then on, the response inversion module work is completed.
[0078] As shown in FIG5 , in one embodiment of the multimodal analysis system for historical building structure monitoring data of the present invention, the response prediction module is used to establish and train a structural temperature model for the future predicted by weather forecasts, and the structural temperature model is trained using monitoring data. The input during the training process of the structural temperature model is the atmospheric temperature field data of the existing weather forecast, the target output is the corresponding structural temperature monitoring result, and the loss function is the mean square error between the actual structural temperature monitoring result and the network output. The Adam method is selected as the training optimization method, and the training of the structural temperature model is stopped until the target loss function reaches a preset expected value.
[0079] A prediction model based on structural temperature and a prediction model based on historical response are established and trained respectively. The input of the prediction model based on structural temperature is the structural temperature sequence, and the input of the prediction model based on historical response is the structural historical response sequence. The target outputs of the prediction model based on structural temperature and the prediction model based on historical response are both structural monitoring responses, and the loss function is the mean square error between the actual structural response monitoring results and the network output. Adam is selected as the optimization method for training the prediction model based on structural temperature and the prediction model based on historical response. When the target loss function reaches the preset expected value, the training of the prediction model based on structural temperature and the prediction model based on historical response is stopped.
[0080] The two response prediction models are fused; the output of the structural temperature model is used as the network input of the structural temperature-based prediction model to form the final deep learning network of the response prediction module.
[0081] Here, the response prediction module is mainly responsible for predicting the future structural response based on weather forecast information and historical structural responses.
[0082] First, a weather forecast model for future structural temperature predictions is established and trained. Similar to the response inversion phase, a structural temperature model for atmospheric temperature field prediction is established, employing a DNN model. The structural temperature model is trained using monitoring data to accurately predict structural temperature. The input for training the structural temperature model is the atmospheric temperature field data from existing weather forecasts, the target output is the corresponding structural temperature monitoring results, and the loss function is the mean squared error between the actual structural temperature monitoring results and the network output. The Adam method is used as the training optimization method, and training is terminated when the target loss function reaches a preset expected value. Upon completion of training, a structural temperature prediction model based on weather forecast information is obtained.
[0083] Next, two response prediction models were established and trained: a structural temperature-based prediction model and a historical response-based prediction model. Both models utilize the LSTM deep learning model. The input for the structural temperature-based prediction model is the structural temperature sequence, while the input for the historical response-based prediction model is the structural response sequence. Their target outputs are both structural monitoring responses, and the loss function is the mean squared error between the actual structural response monitoring results and the network output. Adam optimization was used for training, and training was terminated when the target loss function reached a preset expected value. After training, the structural temperature-based prediction model and the historical response-based prediction model were obtained. These two models predict future structural responses based on temperature and historical responses, respectively.
[0084] The two response prediction models are then fused. Here, the results of the structural temperature-based prediction model and the historical response-based prediction model are averaged to obtain the final structural response prediction. Furthermore, another approach uses the results of the structural temperature-based prediction model and the historical response-based prediction model as input to establish a new prediction fusion model, whose target output is the structural monitoring response. The loss function and optimization method are consistent with the previous two response prediction models. After training is complete, the prediction fusion model is obtained, which can output the final future structural response prediction.
[0085] Finally, the output of the structural temperature model predicted by the weather forecast is used as the network input of the structural temperature-based prediction model, and the above networks can be combined into a future response prediction model to form the final deep learning network of the response prediction module.
[0086] At this point, the response prediction module is completed.
[0087] In one embodiment of the multimodal analysis system for historical building structure monitoring data of the present invention, the multimodal integration module is used to successively run the self-network corresponding to the abnormality diagnosis and repair module, the response inversion module, and the response prediction module. After the operation is completed, the network terminates and outputs the results; during the integration process of the data output process, the abnormality occurrence time, the comparison of the abnormality repair result with the original data, the response inversion results at each location, and the response prediction within the next day are successively output, and the multimodal network integration process is finally completed.
[0088] The multimodal integration module is responsible for integrating the aforementioned modules into a cohesive multimodal network. Considering that data anomaly repair and prediction processes should be completed within the same system in practice, the anomaly diagnosis and repair module, the response inversion module, and the response prediction module are integrated into a unified system to form a multimodal network, systematically processing data from the structural monitoring system.
[0089] Model integration mainly involves data input, model operation sequence and data output.
[0090] During the data input integration process, weather forecast information, strain information, and temperature information are input into different input locations of the multimodal network according to the requirements of the model establishment and training module. They are processed into a data structure that conforms to the model input, facilitating the multimodal network's call, extraction, and analysis of the data. During the integration process of the model operation sequence, the corresponding self-networks of the anomaly diagnosis and repair module, the response inversion module, and the response prediction module are successively run. After the operation is completed, the network terminates and outputs the results. During the integration process of the data output process, the anomaly occurrence time, the comparison of the anomaly repair results with the original data, the response inversion results at each location, and the response prediction for the next day are successively output, finally completing the multimodal integration process.
[0091] After the multimodal network is integrated, the data can be input once and a series of analysis processes can be automatically completed, and the multimodal network can be used to realize intelligent data processing. The structural diagram of the multimodal network is shown in Figure 6.
[0092] Compared with the current methods of processing historical building structure monitoring data based only on single aspects such as anomaly repair or data modeling, the beneficial effects of the present invention are mainly as follows:
[0093] (1) Comprehensive data processing process. This method integrates the various processes in monitoring data processing into a unified system, and realizes the analysis of multiple processes such as abnormal diagnosis and repair, data inversion, and response prediction in an integrated manner, thereby improving the overall capability of the historical building structure monitoring data processing system and comprehensively solving various needs in the historical building monitoring data processing process.
[0094] (2) Simplify the monitoring data processing process. The multimodal network proposed in this method effectively transforms "multiple data inputs and multiple network selections" into "one-time data input and automatic network operation", simplifying the historical building monitoring data processing process.
[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0096] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.
[0097] Obviously, those skilled in the art may make various changes and modifications to the invention without departing from the spirit and scope of the invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.
Claims
1. A multimodal analysis system for historical building structure monitoring data, It is characterized in that include: An information acquisition and preprocessing module is used to obtain monitoring data of the historical building structure through sensors for structural health monitoring installed on the historical building structure, preprocess the monitoring data, and input weather forecast information, strain information, and temperature information into different modules of the multimodal network; The abnormal diagnosis and repair module is used to establish a response prediction model based on structural temperature for abnormal diagnosis, and to diagnose and repair abnormal monitoring data by using the abnormal factors and their probability distribution that characterize the difference between the predicted value and the monitored value; A response inversion module, used to establish a response inversion model by using the response correlation between different structural health monitoring sensors, and to invert the monitoring data of the remaining sensors based on the response inversion model and the monitoring data of some sensors; The response prediction module is used to predict the structural temperature based on the atmospheric temperature information from the weather forecast and to predict the future structural response in combination with the existing historical structural response; The multimodal integration module is used to integrate the abnormal diagnosis and repair module, the response inversion module, and the response prediction module into a multimodal network to form a multimodal network and systematically process the monitoring data.
2. The multimodal analysis system for historical building structure monitoring data as claimed in claim 1, It is characterized in that The information acquisition and preprocessing module is used to determine the type of monitoring data, the type and model of sensors, the number and distribution of sensors, and the frequency of monitoring data and signal transmission method; standardize the monitoring data to make the data a series of data with a mean of 0 and a standard deviation of 1; according to the requirements of the model establishment and training module, the weather forecast information, strain information and temperature information are input into different input positions of the multimodal network and processed into a data structure form that conforms to the model input, so as to facilitate the multimodal network to call, extract and analyze the data.
3. The multimodal analysis system for historical building structure monitoring data as claimed in claim 2, It is characterized in that The monitoring data include: structural temperature, stress strain data and displacement data at various locations of the structure.
4. The multimodal analysis system for historical building structure monitoring data as claimed in claim 3, It is characterized in that The abnormal diagnosis and repair module is used to establish a neural network model describing the relationship between the structural temperature and the structural response based on the data of the structural temperature and the structural response, taking into account the sequence characteristics of the relationship between the temperature and the structure in the monitoring data, wherein the structural response includes: stress strain data and displacement data at various positions of the structure; The neural network model is trained by using data to accurately describe the relationship between the structural temperature and the structural response. The input in the training process is the sequence of structural temperatures, the target output is the structural response, and the target loss function is the mean square error between the response monitoring value and the network output. The optimization method for training the neural network model uses the adaptive moment estimation method until the target loss function reaches the preset expected value, and then the training of the neural network model is stopped. Different abnormal factors are established based on the network output after inputting temperature and the actual monitoring data; after calculating the abnormal factors, the probability is used to describe the degree of outlier of the abnormal factors, and a Gaussian probability distribution model is introduced; the maximum likelihood estimate of the probability distribution mean and the probability distribution variance of the abnormal factors are calculated respectively to obtain the probability normal distribution model of the abnormal factors and the probability threshold of the abnormal factors corresponding to various states; based on the probability threshold of the abnormal factors, the monitoring data in the time period corresponding to the abnormal factors is diagnosed to be abnormal, and the abnormal monitoring data segment is replaced with the predicted data of the neural network model.
5. The multimodal analysis system for historical building structure monitoring data as claimed in claim 4, It is characterized in that The abnormal diagnosis and repair module is used to select the first abnormal factor as Among them, u i is the monitoring value of the structural response at each time of the day, v i is the predicted value of the structural response model at each moment in a day, n is the number of discrete values of the structural response in a day, and the first abnormal factor represents the mean square error of the difference between the predicted data and the monitored data of the structural response in a day, which describes the overall difference between the predicted response and the monitored data; Another abnormal factor is selected as in, is the average value of monitoring data response at each time of the day, It is the average value of the predicted structural response at each moment in a day. The abnormal factor is the ratio of the standard deviation of the difference between the predicted structural response data and the monitored data in a day, which describes the distribution characteristics of the predicted response and the monitored data.
6. The multimodal analysis system for historical building structure monitoring data as claimed in claim 1, It is characterized in that The abnormal diagnosis and repair module is used to describe the degree of outlier of abnormal factors by probability, and introduces the Gaussian probability distribution model Among them, μ is the mean of the probability distribution, σ 2 is the probability distribution variance; The maximum likelihood estimates of the mean and variance of the probability distribution are The maximum likelihood estimates of the probability distribution mean and probability distribution variance of the abnormal factors α and β are calculated respectively to obtain the probability normal distribution model of the abnormal factors and the occurrence probability thresholds of the abnormal factors corresponding to various states.
7. The multimodal analysis system for historical building structure monitoring data as claimed in claim 1, It is characterized in that The response inversion module is used to train the response inversion model using the monitoring data, wherein the input in the response inversion model training process is the monitoring data of some sensors, the target output is the structural response of the remaining part, and the target loss function is the mean square error between the response monitoring value and the network output; The optimization method for training uses the adaptive moment estimation method, and the training of the response inversion model is stopped when the target loss function reaches the preset expected value; The structural responses of the remaining parts of the structure are inverted based on the structural responses of multiple sensors through the response inversion model.
8. The multimodal analysis system for historical building structure monitoring data as claimed in claim 1, It is characterized in that The response prediction module is used to establish and train a structural temperature model for the future predicted by the weather forecast, and to train the structural temperature model using monitoring data; the input in the training process of the structural temperature model is the atmospheric temperature field data of the existing weather forecast, the target output is the corresponding structural temperature monitoring result, and the loss function is the mean square error between the actual structural temperature monitoring result and the network output; The optimization method for training uses the adaptive moment estimation method, and the training of the structural temperature model is stopped when the target loss function reaches the preset expected value; The prediction model based on structural temperature and the prediction model based on historical response are established and trained respectively. The input of the prediction model based on structural temperature is the structural temperature sequence, and the input of the prediction model based on historical response is the structural historical response sequence. The target outputs are all structural monitoring responses, and the loss function is the mean square error between the actual structural response monitoring results and the network output; The optimization method for training the prediction model based on structural temperature and the prediction model based on historical response uses adaptive moment estimation, and stops training the prediction model based on structural temperature and the prediction model based on historical response until the target loss function reaches a preset expected value; The two response prediction models are fused; the output of the structural temperature model is used as the network input of the structural temperature-based prediction model to form a final deep learning network of the response prediction module.
9. The multimodal analysis system for historical building structure monitoring data as claimed in claim 1, It is characterized in that The response prediction module is used to take the average value of the results of the prediction model based on the structural temperature and the prediction model based on the historical response to obtain the final prediction result of the structural response; or, use the results of the prediction model based on the structural temperature and the prediction model based on the historical response as input values to establish a new prediction fusion model.
10. The multimodal analysis system for historical building structure monitoring data according to claim 1, It is characterized in that The multimodal integration module is used to successively run the self-network corresponding to the abnormal diagnosis and repair module, the response inversion module and the response prediction module. After the operation is completed, the network is terminated and the results are output; in the integration process of the data output process, the abnormal occurrence time, the comparison between the abnormal repair result and the original data, the response inversion result at each position and the response prediction within the next day are output successively, and finally the integration process of the multimodal network is completed.
Citation Information
Patent Citations
Monitoring data repairing method and system for bridge health monitoring system
CN112906793A
Historic building health monitoring system, method and equipment
CN115146230A
Historical building structure response prediction method and system in atmospheric temperature field
CN115906672A
Historical building structure monitoring data abnormity diagnosis and restoration system
CN116049958A
Historical building structure monitoring data dimension reduction inversion system
CN116050271A
Cited By
Remote monitoring method and device for intelligent water meter integrated with NB-IoT
CN119618332A
Optical system temperature field acquisition method and system
CN119647287A
Infrared thermal imaging abnormal scene monitoring method based on multi-modal large model and related device
CN119723464A
House structure health detection method based on multi-modal data
CN119849143A
A building structure health detection method based on multimodal data
CN119849143B
