Multi-bridge monitoring data anomaly identification method and system based on multi-modal feature fusion

Abstract

The application discloses a multi-bridge monitoring data anomaly identification method and system based on multi-modal feature fusion, relates to the field of data anomaly identification, and comprises the following steps: one, collecting SHM time series data of multiple bridges, performing normalization processing and segmentation; two, visualizing the normalized time series data, simultaneously extracting statistical features in the data, and manually marking labels for the image and the statistical features to construct a data set; three, establishing a multi-modal convolutional neural network model based on the input of the gray image and the statistical features, using the data set from the multiple bridges, simultaneously inputting the gray image and the corresponding statistical features into the network model for training, four, using the trained convolutional neural network model to perform anomaly identification on target bridge SHM data. The application realizes automatic identification of bridge SHM data anomalies, improves the identification precision of abnormal data under the condition that the categories of the bridge monitoring data set are unbalanced and the data volume is limited, and provides a basis for bridge operation state evaluation and early warning.

Description

Technical Field

[0001] This invention belongs to the field of data anomaly identification technology, specifically relating to a method and system for identifying multi-bridge monitoring data anomalies based on multimodal feature fusion. Background Technology

[0002] With the development of advanced sensing, the Internet of Things, big data, artificial intelligence, and other new-generation information technologies, structural health monitoring (SHM) technology is now widely used in the monitoring and maintenance of civil engineering structures. Long-term monitoring will generate massive amounts of SHM data, which record rich performance evolution information during bridge operation, including structural loads and response behavior patterns under normal operating conditions, as well as rare loads and responses during special events such as typhoons, ship collisions, earthquakes, fires, and traffic accidents.

[0003] By mining and analyzing the massive amounts of monitoring data collected by the SHM system, the safety and suitability of bridge structures during their service life can be assessed. However, due to monitoring system malfunctions, such as improper sensor installation, quality damage, and long-term use, the monitoring data collected by the SHM system may exhibit various abnormal interferences, severely affecting the data analysis results. Therefore, how to quickly and effectively identify and detect abnormal data is a major challenge in the field of bridge health monitoring.

[0004] To address the various anomaly patterns in bridge SHM data, training convolutional neural network (CNN) models for automatic anomaly identification and classification is an effective method. CNNs possess powerful feature extraction capabilities. Existing techniques typically visualize time-series data and input it into the CNN for training, transforming the anomaly identification problem into an image classification problem. However, this method is limited by image pixel size, potentially leading to the loss or interference of spatial information. To overcome this limitation, multimodal information fusion has been proposed as an effective solution. By integrating data from different modalities, it more comprehensively represents the intrinsic characteristics of the data, significantly improving the accuracy and robustness of anomaly identification. Simultaneously, SHM data for individual bridges often suffers from severe data imbalance, meaning the number of anomaly categories varies greatly. This is reflected in two aspects: firstly, the imbalance between normal and anomaly data (normal monitoring data typically far exceeds anomaly data); and secondly, the imbalance in the number of samples for different anomaly patterns within the anomaly data. This imbalance in individual bridge monitoring data often leads to confusion between the identification results of some categories, affecting the final accuracy of anomaly identification. Summary of the Invention

[0005] In order to solve the technical problems existing in the background art, the present invention aims to provide a method and system for anomaly identification of multi-bridge monitoring data based on multimodal feature fusion, which is intended to solve the problems of low accuracy in identifying some anomaly data caused by the imbalance of single-bridge monitoring data in current image-based anomaly data classification methods.

[0006] To solve the technical problem, the technical solution of the present invention is as follows:

[0007] An automatic anomaly identification method for multi-bridge monitoring data based on multimodal feature fusion, the method comprising:

[0008] S1: Normalize the original time series dataset and divide it into a series of fixed-length segments to obtain normalized time series data segments;

[0009] S2: Convert the obtained normalized time series data segments into grayscale images, extract statistical features from the time series data, manually label the images and statistical features, and output the grayscale image dataset, the statistical feature dataset, and the labeled normal / abnormal labels.

[0010] S3: Based on the output of step S2, establish a multimodal convolutional neural network model. Using datasets from multiple bridges, input grayscale images and corresponding statistical features into the network model simultaneously for training to obtain a trained multimodal deep learning model.

[0011] S4: Use the trained multimodal deep learning model to identify anomalies in the target bridge SHM data.

[0012] It is understandable that there is an imbalance in the SHM data of a single bridge: the amount of normal data is much larger than the amount of abnormal data, and the amount of data of different abnormal categories varies greatly. Therefore, when creating the dataset, SHM data of multiple bridges are used to complement each other to form a more balanced merged dataset.

[0013] Understandably, the designed multimodal convolutional neural network combines image features and statistical features for classification. The image encoder uses a ResNet model to extract rich image features, while the statistical feature encoder extracts statistical features (peak intensity, standard deviation, linearity, and linear fit slope) from time-series data. These features provide statistical information about the data, helping the model better understand the distribution and trends of the data. In the classification stage, the model concatenates these two features to form a more comprehensive feature vector, making full use of information from both modalities.

[0014] Furthermore, prior to step S1, the method further includes:

[0015] SHM time-series data for multiple bridges were collected, including acceleration, deflection, stress, and temperature data, to form the original time-series dataset.

[0016] Further, step S2 includes:

[0017] S201: Read the segmented time series data, plot each sample data as a 224×224 grayscale image, and extract statistical features. Save the image and statistical features in png and npy formats respectively.

[0018] S202: Manually label the generated data images and statistical feature samples with abnormal patterns.

[0019] Furthermore, the statistical features include: peak intensity, standard deviation, linearity, and linear fitting slope; in image-based anomaly data classification methods, the statistical features of the data are introduced to enhance the distinguishing effect between different anomaly patterns.

[0020] The formula for peak intensity is:

[0021]

[0022] In formula (1), D 0.9 x is the distance between the upper and lower bounds of the data values ​​where 90% of the data points in the sample are located. max x is the maximum value in the data. min The minimum value in the data is the feature index, which is mainly used to identify outliers in the data.

[0023] The formula for standard deviation is:

[0024]

[0025] In formula (2), x i For each data point, μ is the sample mean and N is the total number of data points in the sample. This feature index is mainly used to identify the second smallest and over-range anomalies in the data.

[0026] The formula for linearity is:

[0027]

[0028] In formula (3), max(|Δx) i |) represents the maximum absolute value of the deviation between the smooth curve and the fitted line for all data points in the sample, x max and x min These are the maximum and minimum values ​​in the sample, respectively. This feature index is mainly used to identify drift anomalies in the data.

[0029] The linear fit slope is the slope of the fitted line obtained by using least squares linear regression on all data points in the sample. It represents the degree of baseline shift and is used to identify trend anomalies in the data.

[0030] Furthermore, step S3 specifically includes:

[0031] The multimodal convolutional neural network model includes convolutional layers, pooling layers, and fully connected layers. Its overall structure consists of three parts: an image encoder, a statistical feature encoder, and a fusion classifier. The image encoder, based on a ResNet neural network with the last fully connected layer removed, receives a 224×224 grayscale image input and extracts its features, outputting 512 image feature vectors. The statistical feature encoder comprises two fully connected layers activated by the ReLU function. It receives numerical inputs corresponding to four statistical features and maps them to 512-dimensional features, consistent with the dimension of the image features. The fusion classifier concatenates and fuses the extracted image features and statistical features, achieving final classification after passing through the fully connected layers.

[0032] The training steps of the multimodal convolutional neural network model include:

[0033] S301: Manually label the abnormal patterns in the SHM datasets of multiple bridges, and divide the datasets into training and validation sets in an 8:2 ratio. The training set should mainly consist of the data samples of the bridges to be tested, and supplement the abnormal patterns with other bridge data samples for the smaller number of abnormal patterns.

[0034] S302: Based on the designed multimodal convolutional neural network model, train the network weight parameters suitable for anomaly data identification, and adjust the hyperparameters according to the training results to obtain the optimal training model.

[0035] Furthermore, the ResNet neural network introduces a residual block structure, which consists of two convolutional layers, with each convolutional layer followed by a batch normalization layer and a ReLU activation function;

[0036] The purpose of batch normalization is to make the data distribution of each layer in the network more stable, thereby accelerating the learning speed of the model. The batch normalization formula is as follows:

[0037]

[0038] In formula (4): x i For the input sample, μ is the sample mean, σ is the sample variance, ∈ is a minimum value used to prevent the denominator from being zero, γ and β are scaling and offset parameters, respectively, and the standardized data is linearly transformed.

[0039] The ReLU activation function has low computational complexity and can effectively mitigate the gradient explosion and vanishing problems that may occur during neural network training. The formula is:

[0040] f(x) = max(0,x) (5).

[0041] Furthermore, step S4 includes:

[0042] The SHM data of the bridge under test is segmented and converted into grayscale images, and four statistical features are extracted. The data samples of the bridge under test are then input into a trained multimodal convolutional neural network model for identification and classification to obtain the corresponding abnormal patterns.

[0043] An automatic anomaly identification system for multi-bridge monitoring data based on multimodal feature fusion, characterized in that the system is applied to any of the methods described above, and the system includes:

[0044] Data collection module: used to collect SHM time series data of multiple bridges, including acceleration, deflection, stress and temperature data, to form the raw time series dataset;

[0045] Data preprocessing module: used to normalize the original time series dataset and divide it into a series of fixed-length segments to obtain normalized time series data segments;

[0046] Feature extraction and labeling module: It is used to convert the obtained normalized time series data segments into grayscale images, extract statistical features from the time series data, manually label the images and statistical features, and output grayscale image dataset, statistical feature dataset and labeled normal / abnormal labels;

[0047] Deep learning training module: Based on the output of the feature extraction and annotation module, a multimodal convolutional neural network model is established. Using datasets from multiple bridges, grayscale images and corresponding statistical features are simultaneously input into the network model for training, resulting in a trained multimodal deep learning model.

[0048] Anomaly detection module: Uses the trained multimodal deep learning model to identify anomalies in the target bridge SHM data.

[0049] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the automatic identification method for multi-bridge monitoring data anomalies based on multimodal feature fusion as described above.

[0050] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion as described above.

[0051] Compared with the prior art, the advantages of the present invention are as follows:

[0052] This invention collects and integrates SHM time-series data from multiple bridges as a dataset, and normalizes data from different sources. By combining image features and statistical features of the time-series data, the accuracy of anomaly identification and classification is improved. After manually labeling the dataset integrating SHM data from multiple bridges, image samples and statistical features are simultaneously input into a designed multimodal convolutional neural network model for training. Based on deep learning algorithms, anomalies of different patterns in the bridge SHM data are automatically identified and classified with high accuracy, providing a basis for bridge operation status assessment and early warning. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the process of the present invention;

[0054] Figure 2 These are common anomaly patterns in bridge SHM data (taking acceleration data as an example);

[0055] Figure 3 This is a schematic diagram of a multimodal convolutional neural network structure;

[0056] Figure 4 This is a diagram of the residual block structure.

[0057] Figure 5 This is a schematic diagram of the training process of a multimodal convolutional neural network. Detailed Implementation

[0058] The specific implementation of the present invention is described below with reference to embodiments:

[0059] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0060] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0061] Example 1:

[0062] An automatic anomaly identification method for multi-bridge monitoring data based on multimodal feature fusion, such as Figure 1 As shown, it includes the following steps:

[0063] Step 1: Collect SHM time series data (acceleration, deflection, stress, temperature, etc.) for multiple bridges;

[0064] Step 2: Normalize the original SHM time-series data of multiple bridges, scaling the data to the range [-1, 1]. The purpose of this step is to eliminate the differences in data characteristics between bridge SHM data from different sources. Data normalization is a common technique in data preprocessing and can effectively eliminate errors caused by different units of measurement. Subsequently, a non-overlapping sliding window of length w is used to segment the original SHM data of the bridges to be analyzed. The value of w is related to the sampling frequency of the data. When the sampling frequencies of data from different sources differ significantly, the value of w should be modified to make the number of data points in a single sample more similar.

[0065] Step 3: Visualize the normalized time series data by converting it into grayscale images. At the same time, extract the statistical features from the time series data and construct a training set by randomly selecting some images and statistical features through manual labeling.

[0066] Step 3.1: Plot the segmented data as a 224×224 grayscale image for easy input into the neural network model. To avoid interference, hide the coordinate axes in the image and save the image in PNG format. Simultaneously, extract four statistical features from the time-series data (peak intensity, standard deviation, linearity, and linear fit slope) and save them in .npy format.

[0067] Peak intensity:

[0068]

[0069] In formula (1), D 0.9 x is the distance between the upper and lower bounds of the data values ​​where 90% of the data points in the sample are located. max The maximum value in the data, x min I represents the minimum value in the data. This feature index is mainly used to identify outliers in the data. When outliers exist in the sample, I... p The value is close to 0;

[0070] Standard deviation:

[0071]

[0072] In formula (2), x i For each data point, μ is the sample mean, and N is the total number of data points in the sample. This feature index is mainly used to identify the second smallest and over-range anomalies in the data. The standard deviation extracted from the over-range data is significantly higher than that extracted from the normal data, while the standard deviation of the second smallest data is significantly lower than that of the normal data.

[0073] Linearity:

[0074]

[0075] In formula (3), max(|Δx) i |) represents the maximum absolute value of the deviation between the smooth curve and the fitted line for all data points in the sample, x max and x min These represent the maximum and minimum values ​​in the sample, respectively. This feature index is primarily used to identify drift anomalies in the data. The R-value corresponding to normal data... L The value should be close to 0, while drift anomalies are usually manifested as irregular fluctuations in the data. L Larger; the trend anomaly is manifested by a skewed baseline in the data, but the corresponding R... L The value is close to 0, just like normal data.

[0076] Linear fit slope: The slope of the fitted line obtained by using least squares linear regression on all data points in the sample. It represents the degree of baseline deviation. This feature is mainly used to identify trend anomalies in the data.

[0077] Step 3.2: Manually label the generated image samples and statistical features with anomalous patterns. Common data anomaly patterns include outliers, second smallest values, drift, trends, missing values, and overrange oscillations. Taking acceleration data from the bridge SHM system as an example, common anomaly pattern samples include... Figure 2 As shown;

[0078] Step 4: Build a multimodal convolutional neural network model. Use datasets from multiple bridges and input grayscale images and corresponding statistical features into the network model for training. By fusing data from multiple bridges, the problem of low accuracy in some abnormal pattern recognition caused by dataset imbalance can be eliminated.

[0079] The multimodal convolutional neural network designed in this invention includes convolutional layers, pooling layers, and fully connected layers. Its overall structure is divided into three parts: an image encoder, a statistical feature encoder, and a fusion classifier. Figure 3As shown in the diagram, the image encoder, based on a ResNet neural network, removes the last fully connected layer, receives a 224×224 grayscale image input, extracts image features, and outputs 512 image feature vectors. The statistical feature encoder consists of two fully connected layers activated by the ReLU function, receives numerical inputs corresponding to four statistical features, and maps them to 512-dimensional features (consistent with the dimension of the image features). The fusion classifier concatenates and fuses the extracted image features and statistical features. Image features capture spatial patterns, while statistical features provide contextual information. The final classification is achieved after passing through the fully connected layers.

[0080] The ResNet model introduces a residual block structure, such as Figure 4 As shown, the residual block consists of two convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function.

[0081] The purpose of batch normalization is to make the data distribution of each layer in the network more stable, thereby accelerating the learning speed of the model. The batch normalization formula is as follows:

[0082]

[0083] In formula (4): x i The input sample is μ, the sample mean is σ, the sample variance is σ, ∈ is a minimum value used to prevent the denominator from being zero, and γ and β are scaling and offset parameters, respectively, which perform a linear transformation on the standardized data.

[0084] The ReLU activation function has low computational complexity and can effectively mitigate the gradient explosion and vanishing problems that may occur during neural network training. The formula is:

[0085] f(x) = max(0,x) (5)

[0086] The structure of residual blocks can alleviate the gradient vanishing or gradient explosion phenomenon caused by increasing network depth, and can also reduce the occurrence of overfitting in deep networks.

[0087] Step 4.1: Manually label the abnormal patterns in the SHM datasets of multiple bridges, and divide the datasets into training and validation sets in an 8:2 ratio; the training set should mainly consist of the data samples of the bridges to be tested, and for the abnormal patterns that are less numerous, supplement them with data samples from other bridges to avoid the impact of data imbalance.

[0088] Step 4.2: Train the network weight parameters suitable for anomaly data identification based on the designed multimodal convolutional neural network model, and adjust the hyperparameters according to the training results to obtain the optimal training model;

[0089] The hyperparameters of the multimodal convolutional neural network in this invention include the learning rate, batch input size, and regularization coefficient. These hyperparameters are tuned during training based on the training results. The number of iterations is set to 50, and the model is trained as follows: Figure 5 As shown, with the increase of the number of iterations, the training accuracy and validation accuracy steadily improve. When the number of iterations reaches 50, the validation accuracy exceeds 98%. The training results indicate that when multi-bridge SHM data samples are fused, the training set data structure is more balanced. At the same time, the multimodal convolutional neural network model effectively combines image features and statistical features, while also retaining the independent representation capabilities of each modality, enabling high-precision automatic identification and classification of SHM data anomalies.

[0090] Step 5: Use the trained convolutional neural network model to identify anomalies in the target bridge SHM data.

[0091] Step 5.1: Convert the SHM data of the bridge under test into grayscale images in segments and hide the coordinate axes.

[0092] Step 5.2: Input the bridge data sample to be tested into the trained multimodal convolutional neural network model for identification and classification to obtain the corresponding abnormal mode.

[0093] This invention uses a multimodal convolutional neural network model for anomaly data identification. It integrates SHM time-series data from multiple bridges, combining their image features and statistical characteristics, and simultaneously inputs these data into a designed multimodal convolutional neural network for training. Based on deep learning algorithms, it performs high-precision automatic identification and classification of anomalies in different modes in the bridge SHM data, providing a basis for bridge operation status assessment and early warning.

[0094] Example 2:

[0095] This invention provides a multi-bridge monitoring data anomaly identification system based on multimodal feature fusion. This system can be used to implement the aforementioned multi-bridge monitoring data anomaly identification method based on multimodal feature fusion. Specifically, the system includes:

[0096] Data collection module: used to collect SHM time series data of multiple bridges, including acceleration, deflection, stress and temperature data, to form the raw time series dataset;

[0097] Data preprocessing module: used to normalize the original time series dataset and divide it into a series of fixed-length segments to obtain normalized time series data segments;

[0098] Feature extraction and labeling module: It is used to convert the obtained normalized time series data segments into grayscale images, extract statistical features from the time series data, manually label the images and statistical features, and output grayscale image dataset, statistical feature dataset and labeled normal / abnormal labels;

[0099] Deep learning training module: Based on the output of the feature extraction and annotation module, a multimodal convolutional neural network model is established. Using datasets from multiple bridges, grayscale images and corresponding statistical features are simultaneously input into the network model for training, resulting in a trained multimodal deep learning model.

[0100] Anomaly detection module: Uses the trained multimodal deep learning model to identify anomalies in the target bridge SHM data.

[0101] Example 3:

[0102] This embodiment provides a terminal device, which includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used in the operation of a multi-bridge monitoring data anomaly identification method based on multimodal feature fusion, including the following steps:

[0103] Step 1: Collect SHM time series data (acceleration, deflection, stress, temperature, etc.) for multiple bridges;

[0104] Step 2: Normalize the original SHM time series data of multiple bridges and divide it into a series of fixed-length segments;

[0105] Step 3: Visualize the normalized time series data by converting it into grayscale images. Simultaneously, extract the statistical features from the time series data and manually label the images and statistical features to construct a dataset.

[0106] Step 4: Build a multimodal convolutional neural network model based on grayscale images and statistical features as input. Use datasets from multiple bridges and simultaneously input grayscale images and corresponding statistical features into the network model for training.

[0107] Step 5: Use the trained convolutional neural network model to identify anomalies in the target bridge SHM data.

[0108] Example 4:

[0109] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0110] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the multi-bridge monitoring data anomaly identification method based on multimodal feature fusion in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0111] Step 1: Collect SHM time series data (acceleration, deflection, stress, temperature, etc.) for multiple bridges;

[0112] Step 2: Normalize the original SHM time series data of multiple bridges and divide it into a series of fixed-length segments;

[0113] Step 3: Visualize the normalized time series data by converting it into grayscale images. Simultaneously, extract the statistical features from the time series data and manually label the images and statistical features to construct a dataset.

[0114] Step 4: Build a multimodal convolutional neural network model based on grayscale images and statistical features as input. Use datasets from multiple bridges and simultaneously input grayscale images and corresponding statistical features into the network model for training.

[0115] Step 5: Use the trained convolutional neural network model to identify anomalies in the target bridge SHM data.

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

[0121] Many other changes and modifications can be made without departing from the concept and scope of this invention. It should be understood that this invention is not limited to the specific embodiments, and the scope of this invention is defined by the appended claims.

Claims

1. A method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion, characterized in that, The method includes: S1: Normalize the original time series dataset and divide it into a series of fixed-length segments to obtain normalized time series data segments; S2: Convert the obtained normalized time series data segments into grayscale images, extract statistical features from the time series data, manually label the images and statistical features, and output the grayscale image dataset, the statistical feature dataset, and the labeled normal / abnormal labels. S3: Based on the output of step S2, establish a multimodal convolutional neural network model. Using datasets from multiple bridges, input grayscale images and corresponding statistical features simultaneously into the input of the network model for training to obtain a trained multimodal deep learning model. S4: Use the trained multimodal deep learning model to identify anomalies in the target bridge SHM data; Step S3 specifically includes: The multimodal convolutional neural network model includes convolutional layers, pooling layers, and fully connected layers. Its overall structure consists of three parts: an image encoder, a statistical feature encoder, and a fusion classifier. The image encoder, based on a ResNet neural network with the last fully connected layer removed, receives a 224×224 grayscale image input and extracts its features, outputting 512 image feature vectors. The statistical feature encoder comprises two fully connected layers activated by the ReLU function. It receives numerical inputs corresponding to four statistical features and maps them to 512-dimensional features, consistent with the dimension of the image features. The fusion classifier concatenates and fuses the extracted image features and statistical features, achieving final classification after passing through the fully connected layers. The training steps of the multimodal convolutional neural network model include: S301: Manually label the abnormal patterns in the SHM datasets of multiple bridges, and divide the datasets into training and validation sets in an 8:2 ratio. The training set should mainly consist of the data samples of the bridges to be tested, and supplement the abnormal patterns with other bridge data samples for the smaller number of abnormal patterns. S302: Based on the designed multimodal convolutional neural network model, train the network weight parameters suitable for anomaly data identification, and adjust the hyperparameters according to the training results to obtain the optimal training model.

2. The method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion according to claim 1, characterized in that, Prior to step S1, the method further includes: SHM time-series data for multiple bridges were collected, including acceleration, deflection, stress, and temperature data, to form the original time-series dataset.

3. The method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion according to claim 1, characterized in that, Step S2 includes: S201: Read the segmented time series data, plot each sample data as a 224×224 grayscale image, and extract statistical features. Save the image and statistical features in png and npy formats respectively. S202: Manually label the generated data images and statistical feature samples with abnormal patterns.

4. The method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion according to claim 1, characterized in that, The statistical features include peak intensity, standard deviation, linearity, and linear fitting slope; in image-based anomaly data classification methods, statistical features of the data are introduced to enhance the distinguishing effect between different anomaly patterns. The formula for peak intensity is: （1） In formula (1), D 0.9 x is the distance between the upper and lower bounds of the data values ​​where 90% of the data points in the sample are located. max x is the maximum value in the data. min The minimum value in the data; The formula for standard deviation is: （2） In formula (2), x i The value for each data point, Here, N is the sample mean, and N is the total number of data points in the sample. The formula for linearity is: （3） In formula (3), max( | Δx) i | ) represents the maximum absolute value of the deviation between the smoothed curve and the fitted line for all data points in the sample, x max and x min These are the maximum and minimum values ​​in the sample, respectively. The linear fit slope is the slope of the fitted line obtained by using least squares linear regression on all data points in the sample. It represents the degree of baseline shift and is used to identify trend anomalies in the data.

5. The method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion according to claim 1, characterized in that, The ResNet neural network introduces a residual block structure, which consists of two convolutional layers, with each convolutional layer followed by a batch normalization layer and a ReLU activation function. The purpose of batch normalization is to make the data distribution of each layer in the network more stable, thereby accelerating the learning speed of the model. The batch normalization formula is as follows: （4） In formula (4): x i The input sample is μ, the sample mean is σ, the sample variance is ϵ, which is a minimum value used to prevent the denominator from being zero, and γ and β are scaling and offset parameters, respectively. The standardized data is then subjected to a linear transformation. The ReLU activation function has low computational complexity and can effectively mitigate the gradient explosion and vanishing problems that may occur during neural network training. The formula is: （5） 6. The method for automatic identification of anomalies in multi-bridge monitoring data based on multimodal feature fusion according to claim 1, characterized in that, Step S4 includes: The SHM data of the bridge under test is segmented and converted into grayscale images, and four statistical features are extracted. The data samples of the bridge under test are then input into a trained multimodal convolutional neural network model for identification and classification to obtain the corresponding abnormal patterns.

7. An automatic anomaly identification system for multi-bridge monitoring data based on multimodal feature fusion, characterized in that, The system is applied to the method according to any one of claims 1-6, and the system comprises: Data collection module: used to collect SHM time series data of multiple bridges, including acceleration, deflection, stress and temperature data, to form the raw time series dataset; Data preprocessing module: used to normalize the original time series dataset and divide it into a series of fixed-length segments to obtain normalized time series data segments; Feature extraction and labeling module: It is used to convert the obtained normalized time series data segments into grayscale images, extract statistical features from the time series data, manually label the images and statistical features, and output grayscale image dataset, statistical feature dataset and labeled normal / abnormal labels; Deep learning training module: Based on the output of the feature extraction and annotation module, a multimodal convolutional neural network model is established. Using datasets from multiple bridges, grayscale images and corresponding statistical features are simultaneously input into the network model for training, resulting in a trained multimodal deep learning model. Anomaly detection module: Uses the trained multimodal deep learning model to identify anomalies in the target bridge SHM data.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the automatic identification method for multi-bridge monitoring data anomalies based on multimodal feature fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the automatic identification method for multi-bridge monitoring data anomalies based on multimodal feature fusion as described in any one of claims 1 to 6.

Images