Bridge state evaluation method and system based on large model

Abstract

The invention relates to the technical field of bridge state evaluation, in particular to a bridge state evaluation method and system based on a large model, and the method comprises the following steps: collecting historical multi-modal data of different regions on a bridge, and then generating a standardized multi-modal data set; extracting multi-modal features from the standardized data set by using a pre-training field large model, and generating a data state tag as a weak supervision signal; constructing a first-level bridge state evaluation model based on a Transform architecture according to the multi-modal features and the weak supervision signals, wherein the first-level bridge state evaluation model is used for evaluating the regional state of the bridge and generating a regional state vector; and constructing a Transform-based secondary bridge state evaluation model, and further obtaining an overall state evaluation result of the bridge according to the regional state vector and the space coordinate code. According to the method, through pre-training the large model and the cross-modal attention mechanism, the data annotation cost can be reduced, and precise assessment from the local state to the overall state of the bridge is realized in combination with a two-stage Transform architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bridge state evaluation, and in particular to a bridge state evaluation method and system based on a large model. BACKGROUND

[0002] With the rapid development of sensor technology, artificial intelligence and the Internet of Things, bridge state evaluation has gradually shifted from traditional manual inspection to intelligent monitoring. Current mainstream methods can be divided into four categories: first, rule-driven experience threshold judgment, which triggers an alarm by setting fixed thresholds for physical quantities such as strain and vibration; second, data-driven models based on machine learning, which use algorithms such as SVM and LSTM to detect anomalies or classify states in monitoring data; third, multi-modal data fusion technology, which integrates image, vibration, strain and other multi-source information to improve evaluation accuracy; fourth, simulation analysis based on physical models, which simulates bridge structure behavior by establishing finite element models. These methods have achieved remarkable results in specific scenarios. However, as bridge sizes expand and monitoring needs escalate, existing technologies are gradually revealing their limitations.

[0003] The main problems of existing technologies are as follows: first, high data annotation cost, machine learning models rely on a large amount of labeled data, but in actual engineering, abnormal samples are scarce, resulting in low model training efficiency; second, insufficient model generalization ability, traditional models (such as SVM and RF) are sensitive to data distribution, and their performance significantly decreases when crossing bridges or environments; third, multi-modal data fusion is shallow, traditional methods (such as weighted average) fail to fully exploit the relevance between heterogeneous data such as images and time series, limiting evaluation accuracy; fourth, lack of local or global state evaluation, existing technologies usually directly evaluate the overall state of the bridge, failing to implement a bridge evaluation mechanism from local to global, resulting in imperfect bridge state evaluation results. These problems collectively constrain the intelligent level and accuracy of bridge state evaluation. SUMMARY

[0004] In view of the defects in the prior art, the present application provides a bridge state evaluation method and system based on a large model.

[0005] In order to achieve the above object, the application provides a bridge state evaluation method based on a large model, which comprises the following steps: collecting historical multi-modal data of different regions on a bridge, and generating a standardized multi-modal data set using the historical multi-modal data; using a pre-trained domain large model to extract multi-modal features from the standardized data set, while generating data state labels as weak supervision signals; constructing a first-level bridge state evaluation model based on a Transformer architecture according to the multi-modal features and the weak supervision signals, for evaluating the regional state of the bridge and generating a regional state vector; constructing a second-level bridge state evaluation model based on a Transformer architecture, and then obtaining the overall state evaluation result of the bridge according to the regional state vector and spatial coordinate coding. Through the pre-training of the large model and the cross-modal attention mechanism, the data labeling cost can be reduced, and the precise evaluation of the local state to the overall state of the bridge can be realized by combining the two-level Transformer architecture.

[0006] Optionally, the historical multi-modal data comprises historical image data and historical discrete time series data of different regions of the bridge, and the historical discrete time series data comprises historical strain data, historical acceleration data, historical displacement data, historical temperature data and historical humidity data. By collecting multi-modal data to evaluate the state of the bridge, the problem of low accuracy of evaluating the state of the bridge by single modal data can be solved, and the accuracy of evaluating the state of the bridge can be improved.

[0007] Optionally, collecting historical multi-modal data of different regions on a bridge, and generating a standardized multi-modal data set using the historical multi-modal data comprises the following steps: collecting the historical image data and the historical discrete time series data through a sensor network deployed on the bridge; respectively pre-processing the historical image data and the historical discrete time series data to obtain standardized multi-modal data; and constructing a standardized multi-modal data set using the standardized multi-modal data.

[0008] Optionally, the standardized multi-modal data comprises standardized historical image data and standardized historical discrete time series data; and the respective pre-processing of the historical image data and the historical discrete time series data to obtain standardized multi-modal data comprises the following steps: denoising and scaling the historical image data to obtain the standardized historical image data; and performing outlier rejection, missing value supplementation and normalization processing on the historical discrete time series data to obtain the standardized historical discrete time series data. By the standardization processing procedure, the original data quality can be improved, the negative influence of noise interference and missing values on model training can be reduced, the dimensional difference between different modal data can be eliminated, and the key information can be more accurately captured in the feature extraction stage.

[0009] Optionally, the step of extracting multi-modal features from the standardized dataset using a pre-trained domain large model while generating data state labels as weak supervision signals comprises the following steps: extracting image feature vectors of the standardized historical image data using an image large model, and generating image state labels based on a clustering algorithm; extracting time series feature vectors of the standardized historical discrete time series data through a time series large model, and generating time series state labels based on reconstruction error; and fusing the image state labels and the time series state labels into weak supervision signals by weighting. This method extracts multi-modal features through a pre-trained domain large model, and generates weak supervision signals based on dynamic clustering and reconstruction error, which not only utilizes the professional knowledge of the domain model to improve the feature representation capability and provides reliable data input for bridge region state evaluation, but also reduces the dependence on manual annotation through weak supervision, thereby to a certain extent, alleviating the problem of high data annotation cost.

[0010] Optionally, the step of extracting image feature vectors of the standardized historical image data using an image large model, and generating image state labels based on a clustering algorithm comprises the following steps: using a VisionTransformer model fine-tuned on a bridge image dataset to extract the image feature vectors of the standardized historical image data; and performing dynamic clustering on the image feature vectors through a K-means++ algorithm to generate the image state labels containing three states of health, slight damage and severe damage.

[0011] Optionally, the step of extracting time series feature vectors of the standardized historical discrete time series data through a time series large model, and generating time series state labels based on reconstruction error comprises the following steps: using an Informer model to extract the time series feature vectors of the standardized historical discrete time series data; calculating the Euclidean distance between the original data and the reconstruction data of the Informer model, dynamically setting the 95% quantile value and the 99% quantile value as threshold values, and generating the time series state labels of normal, slight abnormality and severe abnormality. This method models long sequence dependency through the ProbSparse attention of the Informer, dynamically sets the 95% quantile value threshold based on reconstruction error to generate abnormality labels, and can improve the accuracy of detecting implicit damage.

[0012] Optionally, the first-level bridge state evaluation model based on the Transformer architecture is constructed according to the multi-modal features and the weak supervision signal, used for evaluating the regional state of the bridge and generating a regional state vector, comprising the following steps: generating a joint feature vector through a cross-modal attention mechanism by the image feature vector and the time series feature vector; associating the joint feature vector with the weak supervision signal, and then constructing a pre-training data set; obtaining the real regional health state, real damage type and real abnormal source labeled by experts, and constructing a model fine-tuning data set in combination with the joint feature vector; constructing a first-level bridge state evaluation model based on the Transformer architecture according to the pre-training data set and the model fine-tuning data set, the input of which is the joint feature vector, and the output of which is the predicted regional health state, the predicted damage type and the predicted abnormal source; and splicing the predicted regional health state, the predicted damage type and the predicted abnormal source output by the first-level bridge state evaluation model into a regional state vector. The first-level model of the method realizes the deep interaction of image and time series features through a cross-modal attention mechanism, can fully mine the correlation between image and time series heterogeneous data, break through the information loss bottleneck of traditional weighted average fusion, and improve the accuracy of bridge state evaluation; and the finally output regional state vector can provide an input with both semantic information and quantitative indicators for the second-level evaluation.

[0013] Optionally, the second-level bridge state evaluation model based on the Transformer architecture is constructed, and then the overall state evaluation result of the bridge is obtained according to the regional state vector and the spatial coordinate code, including the following steps: determining the three-dimensional spatial coordinate position and the connection relationship of different regions of the bridge, and then obtaining the spatial coordinate code of the different regions of the bridge; splicing the regional state vector and the spatial coordinate code into a bridge state evaluation sub-vector, and then obtaining a bridge state evaluation matrix using the bridge state evaluation sub-vector of different regions; obtaining the evaluation result of the overall health state and the remaining life of the bridge by experts, and combining the bridge state evaluation matrix to construct a second-level model data set; constructing a second-level bridge state evaluation model based on the Transformer architecture according to the second-level model data set, the input of which is the bridge state evaluation matrix, and the output of which is the predicted overall health state and the remaining life of the bridge; using the second-level bridge state evaluation model to obtain the current overall health state and the current remaining life of the bridge, and taking them as the overall state evaluation result of the bridge. The method constructs a second-level evaluation model, introduces the spatial coordinate code to associate the regional state vector with the spatial position, generates a bridge state evaluation matrix, and trains the model based on the overall state information labeled by experts, realizes the systematic evaluation from the local region to the overall bridge, makes up for the lack of local or global evaluation in the prior art, finally cooperates the two-level models to reduce the dependence on large-scale labeled data, improve the model generalization ability, deepen the multi-modal data correlation mining and improve the local-to-global evaluation mechanism, alleviate the problems of high data labeling cost, insufficient generalization ability, shallow fusion and imperfect evaluation mechanism, and improve the intelligent level and accuracy of bridge state evaluation.

[0014] In a second aspect, the present application further provides a bridge state evaluation system based on a large model, which comprises a data acquisition device, a data output device, a processor and a storage, the storage comprises a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions enable the processor to implement the bridge state evaluation method based on a large model provided by the present application when executed by the processor. The system can improve the practicability of the method and facilitate the popularization of the method. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0016] Figure 1 A flowchart of a bridge state evaluation method based on a large model according to an embodiment of the present application is shown in FIG. 1. Figure 2 A flowchart of obtaining a bridge region state evaluation result according to an embodiment of the present application is shown in FIG. 2. Figure 3 A flowchart of obtaining a bridge overall state evaluation result according to an embodiment of the present application is shown in FIG. 3. Figure 4 A framework diagram of a bridge state evaluation system based on a large model according to an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0017] Specific embodiments of the present application will be described in detail below, it should be noted that the embodiments described herein are only used for illustration and do not limit the present application. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, it is obvious to those skilled in the art that the present application does not have to be implemented with these specific details. In other instances, well-known circuits, software or methods have not been specifically described in order to avoid obscuring the present application.

[0018] Throughout the specification, the mention of "one embodiment", "an embodiment", "one example" or "an example" means that a particular feature, structure or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present application. Therefore, the phrases "in one embodiment", "in an embodiment", "one example" or "an example" appearing throughout the specification are not necessarily all referring to the same embodiment or example. In addition, specific features, structures or characteristics can be combined in any appropriate combination and / or subcombination in one or more embodiments or examples. In addition, those skilled in the art should understand that the diagrams provided herein are for illustrative purposes only and the diagrams are not necessarily drawn to scale.

[0019] It should be noted in advance that in an alternative embodiment, in addition to making independent descriptions, the same symbols or letters appearing in all formulas have the same meaning and value.

[0020] In an alternative embodiment, please refer to Figure 1 The present application provides a bridge state evaluation method based on a large model, the method comprising the following steps: S1, collecting historical multi-modal data of different regions of the bridge, and using the historical multi-modal data to generate a standardized multi-modal data set.

[0021] The historical multi-modal data includes historical image data of different regions of the bridge and historical discrete time series data, and the historical discrete time series data includes historical strain data, historical acceleration data, historical displacement data, historical temperature data and historical humidity data, and the lengths of the five data are all 1024. Step S1 specifically includes the following steps. S11, collecting the historical image data and the historical discrete time series data by a sensor network deployed on the bridge.

[0022] Specifically, in the embodiment, a camera, a strain sensor, an acceleration sensor, a displacement sensor, a temperature sensor and a humidity sensor are installed on the bridge to form a sensor network, and then image data, strain data, acceleration data, displacement data, temperature data and humidity data of the bridge are collected at a set time interval. The image data, strain data, acceleration data, displacement data, temperature data and humidity data collected before the current time are historical image data, historical strain data, historical acceleration data, historical displacement data, historical temperature data and historical humidity data.

[0023] Further, in order to realize the regional state evaluation of the bridge, sensors need to be deployed in zones on the bridge, and at least one of each type of sensor is deployed in each zone. Redundant sensors can be added in important areas such as support areas and 1 / 4 span areas to improve reliability. The division of the bridge region needs to consider the evaluation accuracy and calculation efficiency, and is dynamically adjusted according to the type of the bridge, the complexity of the structure and the monitoring requirements, which can be determined by relevant technical personnel and experts. The embodiment is aimed at a two-span continuous bridge with a span of 24 meters and a beam height h of about 1.6 meters. Considering the three support areas and the 1 / 4 span area, the 3 / 4 span area and the mid-span area of each span, the bridge is divided into 9 regions.

[0024] S12, respectively pre-processing the historical image data and the historical discrete time series data to obtain standardized multi-modal data.

[0025] The standardized multi-modal data includes standardized historical image data and standardized historical discrete time series data. Step S12 specifically includes the following steps: S121, denoising and scaling the historical image data to obtain the standardized historical image data.

[0026] Specifically, in the embodiment, the historical image data is first subjected to mean filtering, and then scaled to 256x256 pixels by bicubic interpolation to obtain the standardized historical image data.

[0027] In other optional embodiments, a target detection algorithm can also be used to detect the damage area in each image, and then the background and other irrelevant parts in the image are cropped. After completing the noise reduction and cropping of the historical image data, the pixel values of the image can be further normalized to the range of [0, 1], so that the historical image data has a uniform format and range.

[0028] S122, outlier elimination, missing value filling and normalization processing are performed on the historical discrete time series data to obtain the standardized historical discrete time series data.

[0029] Specifically, in the present embodiment, for any kind of historical discrete time series data, first, the outliers are identified and deleted by the quartile method, then the missing values are filled using the linear interpolation method, and finally the historical discrete time series data is scaled to the range of [0, 1] using Min-Max normalization, thereby obtaining the standardized historical discrete time series data.

[0030] S13, using the standardized multi-modal data to construct a standardized multi-modal data set.

[0031] Specifically, in the present embodiment, the standardized multi-modal data of the same region on the bridge is taken as a bridge region sample data, and all bridge region samples collected in the same time period are taken as a bridge overall sample data, and finally 1000 bridge overall sample data in different time periods are used to construct the standardized multi-modal data set.

[0032] The present embodiment collects multi-modal data and improves data quality through the standardization processing procedure, which can reduce the negative impact of noise interference and missing values on subsequent model training, eliminate the dimensional difference between different modal data, and enable more accurate capture of key information in the subsequent feature extraction stage. In addition, using multi-modal data to construct a standardized multi-modal data set to evaluate the bridge state can solve the problem of low accuracy of evaluating the bridge state with single modal data, and improve the accuracy of evaluating the bridge state.

[0033] S2, using a pre-trained domain large model to extract multi-modal features from the standardized data set, and generating data state labels as weak supervision signals.

[0034] Wherein, the multi-modal features include image feature vectors and time series feature vectors. The present embodiment extracts multi-modal features through a pre-trained domain large model, and generates weak supervision signals based on dynamic clustering and reconstruction error, which not only utilizes the professional knowledge of the domain model to improve the feature representation capability and provides reliable data input for bridge region state evaluation, but also reduces the dependence on manual annotation through weak supervision, to a certain extent, alleviating the problem of high data annotation cost. Step S2 specifically includes the following steps: S21, extracting an image feature vector of the standardized historical image data using an image large model, and generating an image state label based on a clustering algorithm.

[0035] In the step S21, the following steps are further included. S211, extracting the image feature vector of the standardized historical image data using a Vision Transformer model fine-tuned on a bridge image dataset.

[0036] Specifically, in the embodiment, a Vision Transformer-Base model pre-trained on an ImageNet-21K dataset is loaded, which has 12 layers of Transformer encoder and a hidden layer dimension of 768. Bridge region images under damage conditions such as cracks, rust and bridge surface protrusions are collected, and after denoising and scaling, the bridge region images are expanded to 8000 under the condition of ensuring the balance of the number of bridge region images of different damage types through data enhancement. Then, whether the corresponding region has damage is labeled on the image, and finally, the Vision Transformer-Base model is fine-tuned using the labeled image to construct a bridge image dataset.

[0037] Further, the Vision Transformer-Base model is fine-tuned using the bridge image dataset. During fine-tuning, the parameters of the first 6 layers of the encoder of the Vision Transformer-Base model are frozen, only the last 6 layers of the encoder and the classification head are trained, the learning rate is set to 0.00005, and the iteration is 20 epochs.

[0038] After fine-tuning the Vision Transformer-Base model, the standardized historical image data can be used as the input of the model, and the 768-dimensional feature vector output by the 12-layer encoder is used as the compact representation of the input image, i.e., the image feature vector.

[0039] S212, dynamically clustering the image feature vector through a K-means++ algorithm to generate the image state label containing three states of health, slight damage and severe damage.

[0040] Specifically, in the present embodiment, first, the elbow rule is used to determine the optimal number of clusters, which is specifically 3, corresponding to the three states of health, minor damage and serious damage. Then the K-means++ algorithm is used to initialize the cluster centers to avoid falling into local optimum; Next, the Euclidean distance of each image feature vector to each cluster center is calculated, and it is classified into the category corresponding to the nearest center, and after the classification is completed, the mean vector of each category is recalculated as the new cluster center, and this operation is repeated until the position of all cluster centers no longer changes significantly or the number of iterations reaches 300 times; Finally, the final cluster label is assigned to each image feature vector, and is mapped to the semantic label. Among them, the semantic label 0 is the health state, indicating that the image feature vector is close to the health category center; the semantic label 1 is the minor damage, indicating that the image feature vector is between the health and the serious damage; the semantic label 2 is the serious damage, indicating that the image feature vector is far away from the health category center.

[0041] Further, manual sampling is performed on the clustering results to confirm the semantic consistency of the labels with the actual bridge state, and the clustering threshold is adjusted if necessary.

[0042] In other optional embodiments, the 768-dimensional feature vector can also be reduced in dimension by PCA before clustering to reduce the computational complexity of the K-means++ algorithm while retaining the main features.

[0043] In other optional embodiments, the image state label can also not be limited to the three categories of health, minor damage and serious damage.

[0044] S22, extracting the time sequence feature vector of the standardized historical discrete time sequence data by the time sequence large model, and generating a time sequence state label based on the reconstruction error.

[0045] Among them, step S22 specifically includes the following steps: S221, extracting the time sequence feature vector of the standardized historical discrete time sequence data by the Informer model.

[0046] Specifically, in the present embodiment, first, a pre-trained Informer model is loaded, which is based on the ProbSparse self-attention mechanism and designed for long sequence time series data, and can efficiently process time series with an encoding length of ≥1024 points. The input data of the Informer model is any kind of standardized historical discrete time sequence data, which captures long-range dependencies through multiple layers of stacked ProbSparse self-attention layers, while reducing computational complexity through distillation operation. In the encoding stage, the input standardized historical discrete time sequence data is mapped to a high-dimensional hidden space representation, and finally the 512-dimensional feature vector output by the last layer of the Informer model is taken as the compact representation of the time sequence feature, that is, the time sequence feature vector.

[0047] Further, the Informer model can be used to extract time sequence feature vectors of different standardized historical discrete time series data respectively.

[0048] S222, calculate the Euclidean distance between the original data and the reconstructed data of the Informer model, dynamically set the 95% quantile value and the 99% quantile value as the threshold, and generate the time sequence state label of normal, slight abnormality and serious abnormality.

[0049] Specifically, in the present embodiment, the Informer model is not only used for feature extraction, but also can realize data reconstruction through the self-encoder structure. Specifically, after the input data is compressed into latent features through the encoder, the original data is reconstructed through the decoder. The Euclidean distance between the original data and the reconstructed data is calculated to obtain the reconstruction error of each time point. Since the time sequence pattern of the bridge in the normal state is relatively stable, while abnormal events such as structural damage and external impact will cause the reconstruction error to increase significantly, therefore, a dynamic threshold can be set through statistical method: statistical analysis is performed on the reconstruction error sequence of historical data, and the 95% quantile value and the 99% quantile value are calculated as the judgment boundary of “slight abnormality” and “serious abnormality” respectively. For example, if the 95% quantile error is 0.3 and the 99% quantile error is 0.8, then the points with error below 0.3 are marked as “normal”, the points between 0.3 and 0.8 are marked as “slight abnormality”, and the points exceeding 0.8 are marked as “serious abnormality”.

[0050] The dynamic threshold is applied to the current data, and the time sequence state label is assigned to different standardized historical discrete time series data, and the time sequence state label can be mapped to a semantic label similar to the image state label. In addition, in order to ensure that the time sequence state label is consistent with the actual working condition, sampling verification needs to be performed on the clustering result, and if necessary, the label assignment strategy can be optimized by adjusting the quantile value.

[0051] The present embodiment can improve the accuracy of detecting implicit damage by modeling long sequence dependence through the ProbSparse attention of Informer, and generating abnormal labels based on the 95% quantile threshold of reconstruction error.

[0052] In other optional embodiments, the time sequence state label can not be limited to three categories of normal, slight abnormality and serious abnormality.

[0053] S23, the image state label and the time sequence state label are weighted and fused into a weak supervision signal.

[0054] Specifically, in the present embodiment, the process of weighting and fusing the image state label and the time sequence state label into a weak supervision signal can be represented by the following relationship: wherein, is a weak supervision signal; is an image state label; is a time series state label of the h-th historical discrete time series data; denotes a semantic label, which is an independent variable in the relationship; and is an adjustable label weight, and and the initial values of and are respectively taken as 0.6 and 0.4; is an indicator function, when the condition condition is true, , otherwise ; denotes finding the independent variable that maximizes the function value f.

[0055] S3, constructing a first-level bridge state evaluation model based on a Transformer architecture according to the multi-modal features and the weak supervision signal, for evaluating the regional state of the bridge and generating a regional state vector.

[0056] wherein, please refer to Figure 2 ( the training process of the first-level bridge state evaluation model is not shown in Figure 2 ), step S3 specifically includes the following steps: S31, generating a joint feature vector through a cross-modal attention mechanism by using the image feature vector and the time series feature vector.

[0057] Specifically, in the embodiment, a cross-modal attention fusion module is constructed to realize the deep interaction of the image features and the time series features of the same region on the bridge. First, various time series feature vectors are spliced to obtain a time series feature splicing vector, and then the image feature vector and the time series feature splicing vector are projected to 512 dimensions through a fully connected layer using a GeLU activation function, to ensure the dimension matching of subsequent matrix operations. Then, a cross-modal attention matrix is generated, wherein the image feature vector is taken as the query matrix, the time series feature vector is taken as the key matrix and the value matrix, the product of the query matrix and the transpose of the key matrix is calculated and divided by a scaling factor to obtain an attention score matrix, and after Softmax normalization, the attention weight is obtained. Finally, the attention weight is multiplied by the value matrix to obtain the weighted sum of the time series features, i.e. the time series weighted features. Similarly, taking the time series feature vector as the query matrix and the image feature vector as the key and value matrices, the same process is used to calculate the weighted sum of the image features, i.e. the image weighted features.

[0058] To fully capture bidirectional interaction information, image feature vectors, temporal feature vectors, temporal weighted features, and image weighted features are concatenated to form a high-dimensional vector containing multimodal interaction information. Then, a nonlinear transformation is performed through a multilayer perceptron (MLP) to further integrate key information from different modal features, ultimately generating a joint feature vector that retains the original modal details and contains cross-modal correlations.

[0059] In this embodiment, the scaling factor is the square root of the 512 dimension of the key matrix. The MLP adopts a two-layer hidden layer structure, with each layer containing 512 neurons, and uses the ReLU activation function to enhance nonlinear expression capabilities.

[0060] S32. Associate the joint feature vector with the weak supervision signal to construct a pre-training dataset.

[0061] Specifically, in this embodiment, the joint feature vectors of the same area and time period on the bridge are associated with the weak supervision signal and used as a pre-training sample. Then, all the pre-training samples are used to construct a pre-training dataset.

[0062] S33. Obtain the real regional health status, real damage type and real abnormality source labeled by experts, and construct the model fine-tuning dataset by combining the joint feature vector.

[0063] Specifically, in this embodiment, 15% of the bridge area sample data is extracted from the standardized multimodal dataset. The extracted bridge area sample data should cover all areas of the bridge and include as many damage types as possible. Professional personnel are organized to manually annotate the extracted bridge area sample data. The annotation content includes three types of realistic labels: actual area health status, actual damage type, and actual anomaly source. Among them, the actual area health status includes healthy, minor damage, and severe damage; the actual damage type includes concrete cracks, steel corrosion, support displacement, etc.; and the actual anomaly source includes possible causes of bridge damage such as temperature stress, vehicle overloading, and material aging.

[0064] Furthermore, to ensure the quality of annotation, the expert annotation results are cross-validated. When different experts have different annotations for the same sample, the labels are unified through negotiation. Finally, a high-quality model fine-tuning dataset is constructed to train the subsequent first-level bridge condition assessment model.

[0065] Finally, the expert annotation results are associated with the joint feature vector generated in step S31 and used as a model fine-tuning sample. Then, all model fine-tuning samples are used to construct the model fine-tuning dataset.

[0066] In other alternative embodiments, if the only requirement is to assess the state of the bridge area, then only the real health status of the area labeled by experts can be obtained, and a model fine-tuning dataset can be constructed by combining the joint feature vectors.

[0067] S34, constructing a primary bridge state evaluation model based on a Transformer architecture according to the pre-training data set and the model fine-tuning data set, the input of which is the joint feature vector and the output of which is a predicted regional health state, a predicted damage type and a predicted abnormal source.

[0068] Specifically, in the embodiment, a multi-task classification model based on a Transformer architecture is constructed. The model takes the joint feature vector generated in step S31 as input, and performs deep feature extraction through a shared Transformer encoder layer. The Transformer encoder layer is composed of multiple layers of stacked Transformer modules, each layer containing a multi-head self-attention mechanism and a feedforward neural network, and the training process is stabilized through residual connection and layer normalization. The high-dimensional features output by the encoder are sent to three independent fully connected classification heads, including: a state classification head for outputting a predicted regional health state, i.e., a probability distribution of three categories of health, minor damage and severe damage; a damage type classification head for outputting a predicted damage type, i.e., a probability distribution of multiple damage types such as concrete cracks, steel corrosion and support displacement; and an abnormal source classification head for outputting a predicted abnormal source, i.e., a probability distribution of multiple abnormal sources such as temperature stress, vehicle overload and material aging.

[0069] The training of the primary bridge state evaluation model is divided into two stages: model pre-training and model fine-tuning.

[0070] In the pre-training stage, the pre-training data set is used, and only the state classification head is activated, and the parameters of the damage type and abnormal source classification heads are frozen. The cross-entropy loss of the state prediction value and the weakly supervised label is calculated, and the parameters of the Transformer encoder and the state classification head are updated through back propagation. The pre-training stage enables the model to preliminarily master the regional state evaluation capability.

[0071] In the fine-tuning stage, the model fine-tuning data set is used, and all three classification heads are activated during training, and a multi-task joint loss is calculated, including state classification loss, damage classification loss and abnormal source classification loss. The state classification loss, damage classification loss and abnormal source classification loss are the cross-entropy of the predicted regional health state and the true regional health state, the cross-entropy of the predicted damage type and the true damage type, and the cross-entropy of the predicted abnormal source and the true abnormal source, respectively. After weighted summation of the three parts of the loss, back propagation is performed, and all network parameters are updated, wherein the weights of the state classification loss, the damage classification loss and the abnormal source classification loss are 0.6, 0.2 and 0.2, respectively. In the fine-tuning stage, on the basis of pre-training, the model is fine-tuned by a small number of expert samples to give it the ability to identify damage types and abnormal sources.

[0072] In the process of training the primary bridge state evaluation model, the Adam optimization algorithm is used to update and adjust the parameters of the model to minimize the loss function.

[0073] More specifically, to verify the evaluation effect of the primary bridge state evaluation model, the model fine-tuning data set is obtained in the manner of step S33, a plurality of model fine-tuning data sets are constructed by extracting bridge region sample data from the standardized multi-modal data set according to proportions of 10%, 20%, 25% and 30%, and a primary bridge state evaluation model verification set is constructed by extracting 10% of the bridge region sample data from the standardized multi-modal data set, and the primary bridge state evaluation model verification set does not have data overlap with all the model fine-tuning data sets. The prediction ability of the primary bridge state evaluation model is evaluated using the weighted F1 score and the generalized Matthews correlation coefficient on the basis of the primary bridge state evaluation model verification set, and the results are shown in Table 1: Table 1 Evaluation ability verification results of the primary bridge state evaluation model As can be seen from Table 1, the prediction ability of the model for the region state is always better than that for the damage type and the abnormal source, because only the state classification head is activated in the pre-training stage, and the parameters of the damage type and abnormal source classification heads are frozen. With the increase of the data amount in the model fine-tuning data set, the model performance is getting better and better, among which, when the proportion reaches 20%, the model performance has reached a good level, and when the proportion reaches 25%, the prediction of the model still has room for further improvement. Considering comprehensively, 25% of the bridge region sample data can be extracted from the standardized multi-modal data set to construct the model fine-tuning data set. Therefore, the performance of the primary bridge state evaluation model of the embodiment is relatively reliable.

[0074] In other optional embodiments, if only the bridge region state is evaluated, the primary bridge state evaluation model can only retain the state classification head.

[0075] S35, splicing the predicted region health state, the predicted damage type and the predicted abnormal source output by the primary bridge state evaluation model into a region state vector.

[0076] Specifically, in the embodiment, the output of the state classification head of the primary bridge state evaluation model is a region state probability distribution , is the health probability, is the slight damage probability, is the severe damage probability; the output of the damage type classification head is a damage type probability distribution , is the probability of the i-th damage type, n is the number of damage types; the output of the anomaly source classification head is an anomaly source probability distribution , P j is the probability of the jth anomaly source, m is the number of anomaly sources.

[0077] Therefore, the predicted regional health state, the predicted damage type and the predicted anomaly source output by the primary bridge state evaluation model can be vector spliced to obtain a 3+n+m-dimensional vector , i.e., a regional state vector. If the bridge is divided into 9 regions, 9 3+n+m-dimensional vectors can be obtained at the same time period. Similarly, based on the current image data and the discrete time series data of the bridge, the current regional state, the current damage type and the current anomaly source of the bridge can be obtained, and then 9 3+n+m-dimensional vectors, i.e., 9 current regional state vectors, can be obtained.

[0078] The primary model of the embodiment realizes deep interaction between images and time series features through a cross-modal attention mechanism, can fully mine the correlation between heterogeneous data such as images and time series, break through the information loss bottleneck of traditional weighted average fusion, and improve the accuracy of bridge state evaluation; the final output regional state vector can provide an input that combines semantic information and quantitative indicators for secondary evaluation.

[0079] S4, constructing a secondary bridge state evaluation model based on a Transformer architecture, and then obtaining a whole bridge state evaluation result according to the regional state vector and the spatial coordinate code.

[0080] Wherein, please refer to Figure 3 ( the training process of the secondary bridge state evaluation model is not shown in Figure 3 ), step S4 specifically includes the following steps: S41, determining the three-dimensional spatial coordinate positions and connection relationships of different regions of the bridge, and then obtaining the spatial coordinate codes of the different regions of the bridge.

[0081] Specifically, in the embodiment, a spatial rectangular coordinate system is established with the center of the bridge support as the origin, the bridge length direction as the X axis, the bridge width direction as the Y axis, and the vertical direction as the Z axis. The three-dimensional coordinates of the center points of each region are measured using a total station or a laser scanner , and the Min-Max normalization method is used to normalize the three-dimensional coordinates of the center points of each region . k is the region number, and K is the number of regions.

[0082] According to the topological relationship between the bridge regions, a regional adjacency matrix A of the bridge is constructed: wherein, indicates the adjacency relationship between regions, i.e., the topological relationship coding, if the region is connected to the region , , otherwise , , , , b .

[0083] For any region k, the normalized three-dimensional coordinates of the region are spliced with the topological relationship coding in the kth row of the region adjacency matrix to obtain the spatial coordinate coding of the region, i.e., wherein, is the spatial coordinate coding of the kth region, is the topological relationship coding in the kth row of the region adjacency matrix, denotes vector splicing.

[0084] S42, splice the region state vector and the spatial coordinate coding into a bridge state evaluation sub-vector, and then use the bridge state evaluation sub-vectors of different regions to obtain a bridge state evaluation matrix.

[0085] Specifically, in the embodiment, the region state vector and the spatial coordinate coding of the same region are spliced into a bridge state evaluation sub-vector, i.e., wherein, is the kth bridge state evaluation sub-vector, is the region state probability distribution of the kth region, is the damage type probability distribution of the kth region, is the abnormal source probability distribution of the kth region.

[0086] K regions can obtain K bridge state evaluation sub-vectors, and then the first bridge state evaluation sub-vector to the Kth bridge state evaluation sub-vector can be used to construct a bridge state evaluation matrix: wherein, D is the bridge state evaluation matrix, and T is the transpose.

[0087] S43, obtain the evaluation results of experts on the overall health state and the remaining life of the bridge, and combine the bridge state evaluation matrix to construct a secondary model dataset.

[0088] Specifically, in the present embodiment, 30% of the bridge overall sample data is extracted from the standardized multi-modal dataset, and experts are organized to evaluate the overall health status and remaining life of the bridge based on the extracted bridge overall sample data, and the expert evaluation results are cross-validated to improve the evaluation quality. The evaluation results are associated with the bridge state evaluation matrix, and then a secondary model dataset is constructed.

[0089] The overall health status of the bridge includes three categories of health, early warning and danger, which correspond to the health, slight damage and serious damage of the three bridge region health statuses in turn. Of course, in other alternative embodiments, the overall health status of the bridge can also include other categories.

[0090] S44, constructing a secondary bridge state evaluation model based on the Transformer architecture according to the secondary model dataset, the input of which is the bridge state evaluation matrix, and the output of which is the predicted overall health status and remaining life of the bridge.

[0091] Specifically, in the present embodiment, the overall architecture of the secondary bridge state evaluation model is similar to that of the primary bridge state evaluation model, but its input is the bridge state evaluation matrix and it has two independent output heads of the bridge overall health status classification head and the remaining life regression head.

[0092] The bridge overall health status classification head first applies global average pooling to the feature matrix output by the encoder to obtain a global feature vector, and then inputs the global feature vector into a fully connected layer followed by a Softmax activation function, and outputs the probability distribution of the bridge overall health status as the predicted bridge overall health status. The remaining life regression head is also based on the global feature vector, which is input into another fully connected layer using the ReLU activation function and outputs a scalar value representing the predicted remaining life.

[0093] The secondary model dataset is divided into a training set and a validation set in a ratio of 7:3, and the training and validation of the secondary bridge state evaluation model are completed. During the training process of the secondary bridge state evaluation model, the Adam optimization algorithm is used to update and adjust the parameters of the model, and the training target is to minimize a multi-task loss function, which satisfies: wherein, is the total loss; is the cross-entropy loss function, which is used to measure the difference between the predicted probability distribution of the bridge overall health status and the expert evaluation of the bridge overall health status; is the mean square error loss function, which is used to measure the difference between the predicted remaining life and the expert evaluation of the remaining life; and are hyperparameters, and their initial values are set to 0.6 and 0.4 respectively.

[0094] S45, using the secondary bridge state evaluation model to obtain the current overall health state and current remaining life of the bridge as the current overall state evaluation result of the bridge.

[0095] Specifically, in the embodiment, the current bridge state evaluation matrix is obtained based on the current regional state vector and the spatial coordinate code, and is input into the secondary bridge state evaluation model, and then the prediction result of the current overall health state and the current remaining life of the bridge is obtained, and the prediction result is taken as the current overall state evaluation result of the bridge.

[0096] It should be noted that in some cases, the actions described in the specification can be performed in different orders and still achieve the desired results, and in the embodiment, the order of the steps given is only to make the embodiment look clearer and more understandable, and is not a limitation.

[0097] In an alternative embodiment, see Figure 4 In order to improve the practicability of the method and facilitate the popularization of the method, the application further provides a bridge state evaluation system based on a large model, which comprises a data acquisition device 1, a data output device 2, a processor 3 and a storage 4. The storage 4 comprises a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program comprises program instructions, and the program instructions enable the processor 3 to implement the bridge state evaluation method based on a large model provided by the application when the program instructions are executed by the processor 3.

[0098] In summary, the application has at least the following beneficial effects: 1. The method collects multi-modal data and improves data quality through a standardization process, which can reduce the negative impact of noise interference and missing values on subsequent model training, eliminate the dimensional difference between different modal data, and enable more accurate capture of key information in the subsequent feature extraction stage. The use of multi-modal data to construct a standardized multi-modal data set for bridge state evaluation can solve the problem of low accuracy of single modal data in evaluating bridge state and improve the accuracy of bridge state evaluation.

[0099] 2. The method extracts multi-modal features by pre-training a domain large model, and generates weak supervision signals based on dynamic clustering and reconstruction error, which not only utilizes the professional knowledge of the domain model to improve the feature representation ability and provides reliable data input for bridge regional state evaluation, but also reduces the dependence on manual annotation through weak supervision, to a certain extent, alleviating the problem of high data annotation cost.

[0100] 3. The first model of the method realizes deep interaction between images and time sequence features through a cross-modal attention mechanism, can fully mine the correlation between heterogeneous data such as images and time sequences, breaks through the information loss bottleneck of traditional weighted average fusion, improves the accuracy of bridge state evaluation, and provides more optional solutions for bridge regional state evaluation; the final output regional state vector can provide an input with both semantic information and quantitative indicators for the second-level evaluation.

[0101] 4. The method constructs a second-level evaluation model, associates the regional state vector with the spatial position by introducing spatial coordinate coding, generates a bridge state evaluation matrix, and trains the model based on the overall state information labeled by experts, thereby realizing systematic evaluation from local regions to the whole bridge, making up for the lack of local or global evaluation in the prior art, and finally realizing the cooperative work of the two-level model, which reduces the dependence on large-scale labeled data, improves the model generalization ability, deepens the correlation mining of multi-modal data, and improves the local-to-global evaluation mechanism, thereby alleviating the problems of high data labeling cost, insufficient generalization ability, shallow fusion, and imperfect evaluation mechanism, and improving the intelligent level and accuracy of bridge state evaluation.

[0102] 5. A system adapted to the method is provided, which can improve the practicability of the method and facilitate the popularization of the method.

[0103] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent substitutions for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.

Claims

1. A method for bridge state assessment based on a large model, characterized in that, The method comprises the following steps: Collecting historical multi-modal data of different regions on the bridge, and generating a standardized multi-modal data set using the historical multi-modal data; Using a pre-trained domain large model to extract multi-modal features from the standardized data set, while generating data state labels as weak supervision signals; Building a first-level bridge state evaluation model based on a Transformer architecture according to the multi-modal features and the weak supervision signals, for evaluating the regional state of the bridge and generating a regional state vector; Building a second-level bridge state evaluation model based on a Transformer architecture, and then obtaining the overall state evaluation result of the bridge according to the regional state vector and spatial coordinate encoding.

2. The bridge state evaluation method based on a large model according to claim 1, wherein: The historical multi-modal data includes historical image data and historical discrete time series data of different regions of the bridge, and the historical discrete time series data includes historical strain data, historical acceleration data, historical displacement data, historical temperature data and historical humidity data. 3.The bridge state evaluation method based on a large model according to claim 1, characterized in that, The collecting of the historical multi-modal data of different regions on the bridge, and the generating of a standardized multi-modal data set using the historical multi-modal data, comprises the following steps: Collecting the historical image data and the historical discrete time series data through a sensor network deployed on the bridge; Respectively pre-processing the historical image data and the historical discrete time series data to obtain standardized multi-modal data; Using the standardized multi-modal data to build a standardized multi-modal data set.

4. The bridge state evaluation method based on a large model according to claim 3, wherein: The standardized multi-modal data includes standardized historical image data and standardized historical discrete time series data; The respective pre-processing of the historical image data and the historical discrete time series data to obtain standardized multi-modal data comprises the following steps: Denoising and scaling the historical image data to obtain the standardized historical image data; Performing outlier rejection, missing value supplementation and normalization processing on the historical discrete time series data to obtain the standardized historical discrete time series data. 5.The bridge state evaluation method based on a large model according to claim 1, wherein, The using of a pre-trained domain large model to extract multi-modal features from the standardized data set, while generating data state labels as weak supervision signals, comprises the following steps: Using an image large model to extract an image feature vector of the standardized historical image data, and generating an image state label based on a clustering algorithm; Extracting a time series feature vector of the standardized historical discrete time series data through a time series large model, and generating a time series state label based on a reconstruction error; Weighted fusion of the image state label and the time series state label into a weak supervision signal.

6. The bridge state evaluation method based on a large model according to claim 5, characterized in that, The using of an image large model to extract an image feature vector of the standardized historical image data, and generating an image state label based on a clustering algorithm, comprises the following steps: Using a Vision Transformer model fine-tuned on a bridge image data set to extract the image feature vector of the standardized historical image data; The image feature vectors are dynamically clustered by a K-means++ algorithm to generate the image state labels containing three states of health, slight damage and severe damage.

7. The bridge state evaluation method based on a large model according to claim 5, characterized in that, The steps of extracting the time sequence feature vector of the standardized historical discrete time sequence data by the time sequence large model and generating a time sequence state label based on reconstruction error include the following steps: The Informer model is used to extract the time sequence feature vector of the standardized historical discrete time sequence data. The Euclidean distance between the original data and the reconstruction data of the Informer model is calculated, and the 95% quantile value and the 99% quantile value are dynamically set as threshold values to generate the time sequence state labels of normal, slight abnormality and severe abnormality. 8.The bridge state evaluation method based on a large model according to claim 5, wherein, The steps of constructing a first-level bridge state evaluation model based on the Transformer architecture according to the multi-modal feature and the weak supervision signal for evaluating the regional state of the bridge and generating a regional state vector include the following steps: The image feature vector and the time sequence feature vector are combined into a joint feature vector through a cross-modal attention mechanism; The joint feature vector is associated with the weak supervision signal to construct a pre-training data set; The real regional health state, real damage type and real abnormal source labeled by experts are obtained, and a model fine-tuning data set is constructed in combination with the joint feature vector; A first-level bridge state evaluation model based on the Transformer architecture is constructed according to the pre-training data set and the model fine-tuning data set, which takes the joint feature vector as input and outputs the predicted regional health state, predicted damage type and predicted abnormal source; The predicted regional health state, predicted damage type and predicted abnormal source output by the first-level bridge state evaluation model are spliced into a regional state vector. 9.The bridge state evaluation method based on a large model according to claim 8, characterized in that, The steps of constructing a second-level bridge state evaluation model based on the Transformer architecture and then obtaining the overall state evaluation result of the bridge according to the regional state vector and the spatial coordinate code include the following steps: The three-dimensional spatial coordinate positions and connection relationships of different regions of the bridge are determined to obtain the spatial coordinate code of the different regions of the bridge; The regional state vector and the spatial coordinate code are spliced into a bridge state evaluation sub-vector, and then the bridge state evaluation matrix is obtained using the bridge state evaluation sub-vectors of different regions; The evaluation results of the overall health state and the remaining life of the bridge by experts are obtained, and a second-level model data set is constructed in combination with the bridge state evaluation matrix; A second-level bridge state evaluation model based on the Transformer architecture is constructed according to the second-level model data set, which takes the bridge state evaluation matrix as input and outputs the predicted overall health state and remaining life of the bridge; The current overall health state and current remaining life of the bridge are obtained using the second-level bridge state evaluation model, which are used as the current overall state evaluation result of the bridge. 10.A bridge state evaluation system based on a large model, characterized by, The bridge state evaluation system based on a large model comprises a data acquisition device, a data output device, a processor and a storage, the storage comprises a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions enable the processor to implement the bridge state evaluation method based on a large model according to any one of claims 1-9 when the program instructions are executed by the processor.

Images