Multi-bridge monitoring data anomaly identification method and system based on multi-modal feature fusion

Abstract

The invention discloses a multi-bridge monitoring data anomaly identification method and system based on multi-modal feature fusion, and relates to the field of data anomaly identification, and the method comprises the following steps: 1, collecting SHM time series data of a plurality of bridges, and carrying out the normalization processing and segmentation; 2, visualizing the normalized time series data, extracting statistical characteristics in the data, and manually labeling the images and the statistical characteristics to construct a data set; 3, establishing a multi-modal convolutional neural network model based on the input of the grayscale images and the statistical features, and inputting the grayscale images and the corresponding statistical features into the network model at the same time for training by using a data set from a plurality of bridges, and 4, performing anomaly recognition on the SHM data of the target bridge by using the trained convolutional neural network model. According to the invention, automatic identification of the bridge SHM data abnormity is realized, the identification precision of the abnormal data is improved under the condition that the types of the bridge monitoring data set are unbalanced and the data volume is limited, and a basis is provided for evaluation and early warning of the bridge operation state.

Description

Technical Field

[0001] The present invention belongs to the technical field of data anomaly recognition, and particularly relates to a multi-bridge monitoring data anomaly recognition method and system based on multi-modal feature fusion. Background Art

[0002] With the development of a new generation of information technologies such as advanced sensing, Internet of Things, big data, and artificial intelligence, structural health monitoring technology has currently been widely applied to the monitoring and maintenance of civil engineering structures. Long-term monitoring will generate a large amount of SHM data, and these monitoring data record rich performance evolution information during the operation of the bridge, including the structural load and response behavior rules under normal operating conditions, as well as the rare loads and responses in special events such as typhoons, ship collisions, earthquakes, fires, and traffic accidents.

[0003] By mining and analyzing the large amount of monitoring data collected by the SHM system, the safety and applicability of the bridge structure during its service life can be evaluated. However, due to monitoring system failures, such as improper sensor installation, quality damage, and long-term use, etc., the monitoring data collected by the SHM system may be interfered by various anomalies, seriously affecting the data analysis results. Therefore, how to quickly and effectively identify and detect abnormal data is a major challenge in the field of bridge health monitoring.

[0004] For various data anomaly patterns existing in bridge SHM data, training a convolutional neural network model to automatically identify and classify abnormal data is an effective method. The convolutional neural network has a powerful feature extraction ability. Existing technologies usually visualize the time-series data and input it into the convolutional neural network for training, converting the abnormal recognition problem of time-series data into an image classification problem. However, this method is limited by the pixel size of the image, which may lead to the loss or interference of the spatial information of the data. To overcome this limitation, the fusion of multi-modal information is proposed as an effective solution. By integrating data from different modalities, the internal characteristics of the data can be more comprehensively characterized, significantly improving the accuracy and robustness of abnormal recognition. At the same time, there is often a serious data imbalance problem in the SHM data of a single bridge, that is, the number of each abnormal category in the data set varies greatly. On the one hand, this is reflected in the imbalance between the sample sizes of normal data and abnormal data. Usually, the normal monitoring data is much more than the abnormal data. At the same time, it is also reflected in the imbalance of the sample numbers of each abnormal pattern within the abnormal data. This imbalance problem of the single-bridge monitoring data often leads to confusion in the recognition results between some categories, affecting the final abnormal data recognition accuracy. Summary of the Invention

[0005] To solve the technical problems existing in the background art, the present invention aims to provide a multi-bridge monitoring data anomaly recognition method and system based on multi-modal feature fusion, aiming to solve the problem of low recognition accuracy of some abnormal data caused by the current image-based abnormal data classification method and the imbalance of single-bridge monitoring data.

[0006] To solve the technical problems, the technical solution of the present invention is:

[0007] A multi-bridge monitoring data anomaly automatic recognition method based on multi-modal feature fusion, the method comprising:

[0008] S1: Normalize the original time series data set and segment it into a series of fixed-length segments to obtain the normalized time series data segments;

[0009] S2: Convert the obtained normalized time series data segments into grayscale images, simultaneously extract the statistical features in the time series data, and manually label the images and statistical features with tags, and output the grayscale image data set, the statistical feature data set, and the labeled normal / abnormal tags;

[0010] S3: Based on the output result of step S2, establish a multi-modal convolutional neural network model, use the data sets from multiple bridges, and input the grayscale images and the corresponding statistical features into the network model for training simultaneously to obtain a trained multi-modal deep learning model;

[0011] S4: Use the trained multi-modal deep learning model to perform anomaly recognition on the target bridge SHM data.

[0012] It can be understood that there is an imbalance in the SHM data of a single bridge: the amount of normal data is much larger than that of abnormal data, and the amount of data in different abnormal categories in the abnormal data varies greatly. Therefore, when creating the data set, the SHM data from multiple bridges are used to complement each other to form a combined and more balanced data set.

[0013] It can be understood that the designed multi-modal convolutional neural network combines image features and statistical features for classification. The image encoder part uses the ResNet model to extract rich image features, and the statistical feature encoder part extracts statistical features (peak intensity, standard deviation, linearity, and linear fitting slope) from the time series data. These features provide statistical information of the data, which helps the model better understand the distribution and trend of the data. In the classification stage, the model concatenates these two features to form a more comprehensive feature vector, making full use of the information of both modalities.

[0014] Further, before the step S1, the method further comprises:

[0015] Collect the SHM time series data of multiple bridges, including acceleration, deflection, stress, and temperature data, to form an original time series data set.

[0016] Further, step S2 includes:

[0017] S201: Read the segmented time series data, plot each sample data as a 224×224 grayscale image, and extract statistical features. The image and statistical features are saved in png and npy formats respectively;

[0018] S202: Manually label the abnormal pattern labels for the generated data images and statistical feature samples.

[0019] Further, the statistical features include: peak intensity, standard deviation, linearity, and linear fitting slope; in the image-based abnormal data classification method, the statistical features of the data are introduced to strengthen the discrimination effect between abnormal patterns;

[0020] The formula for peak intensity is:

[0021]

[0022] In formula (1), D 0.9 is the distance between the upper and lower bounds of the data values where 90% of the data points in the sample are located, x max is the maximum value in the data, x min is the minimum value in the data. This feature index is mainly used to identify the outlier anomalies in the data;

[0023] The formula for standard deviation is:

[0024]

[0025] In formula (2), x i is the value of each data point, μ is the sample mean, and N is the total number of data points in the sample. This feature index is mainly used to identify the sub-small and over-range anomalies in the data;

[0026] The formula for linearity is:

[0027]

[0028] In formula (3), max(|Δx i |) is the maximum value of the absolute deviation between the smooth curve and the fitting line of all data points in the sample, x max and x min are the maximum and minimum values in the sample respectively. This feature index is mainly used to identify the drift anomalies in the data;

[0029] The slope of the linear fit is the slope of the fitted line obtained by using the least squares linear regression for all data points in the sample, which represents the degree of baseline shift. This characteristic index is used to identify trend anomalies in the data.

[0030] Further, step S3 specifically includes:

[0031] The multi-modal convolutional neural network model includes a convolutional layer, a pooling layer, and a fully connected layer. The overall structure is divided into three parts: an image encoder, a statistical feature encoder, and a fusion classifier. Among them, the image encoder is based on the ResNet neural network. The last fully connected layer is removed, and it receives a grayscale image input of size 224×224 and extracts the image features therein, outputting 512 image feature vectors. The statistical feature encoder consists of two fully connected layers, activated by the ReLU function in the middle, receives the numerical inputs corresponding to 4 statistical features and maps them to 512-dimensional features, which is consistent with the dimension of the image features. The fusion classifier splices and fuses the extracted image features and statistical features, and realizes the final classification through the fully connected layer.

[0032] The training steps of the multi-modal convolutional neural network model include:

[0033] S301: Manually label the abnormal mode labels for the SHM datasets of multiple bridges, and divide the datasets into a training set and a validation set according to a ratio of 8:2. The training set should mainly include the data samples of the bridges to be measured. For the abnormal modes with a small number, the data samples of other bridges are used for supplementation.

[0034] S302: Based on the designed multi-modal convolutional neural network model, train the network weight parameters suitable for abnormal data recognition, adjust the hyperparameters according to the training results, and obtain the optimal training model.

[0035] Further, the ResNet neural network introduces the structure of residual blocks. The residual blocks are composed of two convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function.

[0036] The purpose of batch normalization is to make the data distribution of each layer input in the network more stable and accelerate the learning speed of the model. The batch normalization formula is as follows:

[0037]

[0038] In formula (4): x i is the input sample, μ is the sample mean, σ is the sample variance, ∈ is a very small value used to prevent the denominator from being zero, γ and β are the scaling parameter and the offset parameter, and the standardized data is linearly transformed.

[0039] The ReLU activation function features low computational complexity and can effectively mitigate the problems of gradient explosion and gradient disappearance that may occur during neural network training. The formula is as follows:

[0040] f(x) = max(0, x) (5).

[0041] Furthermore, the step S4 includes:

[0042] Segment the SHM data of the bridge to be measured into grayscale images and extract 4 statistical features. Input the data samples of the bridge to be measured into the trained multi-modal convolutional neural network model for identification and classification to obtain the abnormal pattern it belongs to.

[0043] A multi-bridge monitoring data anomaly automatic recognition system based on multi-modal feature fusion, characterized in that the system is applied to any of the above methods, and the system includes:

[0044] Data collection module: used to collect the SHM time series data of multiple bridges, including acceleration, deflection, stress, and temperature data, to form an original time series data set;

[0045] Data preprocessing module: used to perform normalization processing on the original time series data set and segment it into a series of fixed-length segments to obtain the normalized time series data segments;

[0046] Feature extraction and annotation module: used to convert the obtained normalized time series data segments into grayscale images, extract the statistical features in the time series data at the same time, and manually mark the images and statistical features with labels, and output the grayscale image data set, the statistical feature data set, and the labeled normal / abnormal labels;

[0047] Deep learning training module: Based on the output results of the feature extraction and annotation module, establish a multi-modal convolutional neural network model, use the data sets from multiple bridges, input the grayscale images and the corresponding statistical features into the network model for training at the same time, and obtain the trained multi-modal deep learning model;

[0048] Anomaly detection module: Use the trained multi-modal deep learning model to identify anomalies in the SHM data of the target bridge.

[0049] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the multi-bridge monitoring data anomaly automatic recognition method based on multi-modal feature fusion described in any of the above.

[0050] A computer-readable storage medium stores a computer program thereon, and when the program is executed by a processor, it implements the multi-bridge monitoring data anomaly automatic recognition method based on multi-modal feature fusion described in any one of the above.

[0051] Compared with the prior art, the advantages of the present invention are as follows:

[0052] The present invention collects and fuses the SHM time-series data of multiple bridges as a data set, and normalizes the data from different sources. By combining the image features and statistical features of the time-series data, the recognition and classification accuracy of data anomalies is improved; after manually labeling the data set that fuses the SHM data of multiple bridges, the image samples and statistical features are simultaneously input into the designed multi-modal convolutional neural network model for training, and based on the deep learning algorithm, different patterns of anomalies in the bridge SHM data are automatically recognized and classified with high precision, providing a basis for bridge operation status evaluation and early warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic flow chart of the present invention;

[0054] Figure 2 is a common anomaly pattern in bridge SHM data (taking acceleration data as an example);

[0055] Figure 3 is a schematic structural diagram of a multi-modal convolutional neural network;

[0056] Figure 4 is a structural diagram of a residual block;

[0057] Figure 5 is a schematic diagram of the training process of a multi-modal convolutional neural network. DETAILED DESCRIPTION OF THE INVENTION

[0058] The following describes the specific embodiments of the present invention in conjunction with the embodiments:

[0059] It should be noted that the structures, ratios, sizes, etc. shown in this specification are only used to cooperate with the content disclosed in the specification for those skilled in this technology to understand and read, and are not used to limit the implementation conditions of the present invention. Any modification of the structure, change of the proportional relationship, or adjustment of the size should still fall within the scope covered by the technical content disclosed in the present invention without affecting the effects that the present invention can produce and the purposes that can be achieved.

[0060] Meanwhile, terms such as "upper", "lower", "left", "right", "middle", and "one" cited in this specification are only for the convenience of clear description and do not limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships, without substantial changes in the technical content, should also be regarded as the scope of implementation of the present invention.

[0061] Example 1:

[0062] An automatic abnormal recognition method for multi-bridge monitoring data based on multi-modal feature fusion, as Figure 1 shown, includes the following steps:

[0063] Step 1: Collect SHM time series data (acceleration, deflection, stress, temperature, etc.) of multiple bridges;

[0064] Step 2: Normalize the original SHM time series data of multiple bridges and scale the data within the range of [-1, 1]; the purpose of this step is to eliminate the data feature differences existing in the bridge SHM data from different sources. Data normalization is a common technique in data preprocessing and can effectively eliminate errors caused by different dimensions. Subsequently, an overlapping sliding window with a length of w is used to segment the original SHM data of the bridge to be analyzed. The value of w is related to the sampling frequency of the data. When the sampling frequencies of data from different sources vary greatly, the value of w should be modified to make the number of data points in a single sample close.

[0065] Step 3: Visualize the normalized time series data, convert it into a grayscale image, extract statistical features from the time series data at the same time, and randomly select some images and statistical features through manual marking to construct a training set;

[0066] Step 3.1: Plot the segmented data into a grayscale image with a size of 224×224 for convenient subsequent input into the neural network model. At the same time, to avoid interference, hide the coordinate axes in the image, and save the image in png format; extract 4 statistical features (peak intensity, standard deviation, linearity, and linear fitting slope) from the time series data and save them in npy format, where:

[0067] Peak intensity:

[0068]

[0069] In formula (1), D 0.9 is the distance between the upper and lower bounds of the data values where 90% of the data points in the sample are located, x max is the maximum value in the data, x min is the minimum value in the data. This feature index is mainly used to identify outliers in the data. When there are outliers in the sample, the value of I p is close to 0;

[0070] Standard deviation:

[0071]

[0072] In formula (2), x i is the value of each data point, μ is the sample mean, and N is the total number of data points in the sample. This characteristic index is mainly used to identify sub-minimum and out-of-range anomalies in the data. The standard deviation extracted from out-of-range data is significantly higher than that extracted from normal data, while the standard deviation of sub-minimum data is significantly smaller than that of normal data;

[0073] Linearity:

[0074]

[0075] In formula (3), max(|Δx i |) is the maximum value of the absolute deviation between the smooth curve and the fitting straight line of all data points in the sample, and x max and x min are the maximum and minimum values in the sample respectively. This characteristic index is mainly used to identify drift anomalies in the data. The R L value corresponding to normal data should be close to 0, while drift anomalies usually show irregular up and down fluctuations of the data, and the R L is larger; trend anomalies are manifested as the baseline tilt of the data, but the corresponding R L value is as close to 0 as that of normal data;

[0076] Linear fitting slope: The slope of the fitting straight line obtained by using the least squares linear regression for all data points in the sample, which represents the degree of baseline offset. This characteristic index is mainly used to identify trend anomalies in the data.

[0077] Step 3.2: Manually label the abnormal pattern labels for the generated image samples and statistical features. Common data abnormal patterns include outliers, sub-minimum, drift, trend, missing, out-of-range oscillation, etc.; Taking the acceleration data in the bridge SHM system as an example, common samples of various abnormal patterns are as Figure 2 shown;

[0078] Step 4: Establish a multi-modal convolutional neural network model. Use the data sets from multiple bridges, and input the grayscale images and the corresponding statistical features into the network model for training at the same time; By fusing the data of multiple bridges, the problem of low recognition accuracy of some abnormal patterns caused by data set imbalance can be eliminated;

[0079] The multi-modal convolutional neural network designed by the present invention includes a convolutional layer, a pooling layer and a fully connected layer. The overall structure is divided into three parts: an image encoder, a statistical feature encoder and a fusion classifier, as Figure 3As shown in the figure. Among them, the image encoder is based on the ResNet neural network. The last fully connected layer is removed. It receives grayscale image inputs of size 224×224 and extracts the image features therein, outputting 512 image feature vectors. The statistical feature encoder consists of two fully connected layers, activated by the ReLU function in the middle. It receives the numerical inputs corresponding to 4 statistical features and maps them to 512-dimensional features (the same dimension as the image features). The fusion classifier concatenates and fuses the extracted image features and statistical features. The image features capture spatial patterns, and the statistical features provide context information. The final classification is achieved through a fully connected layer.

[0080] The ResNet model introduces the structure of residual blocks. As Figure 4 shown, the residual block consists of two convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function.

[0081] The purpose of batch normalization is to make the data distribution of each layer input in the network more stable and accelerate the learning speed of the model. The batch normalization formula is as follows:

[0082]

[0083] In formula (4): x i is the input sample, μ is the sample mean, σ is the sample variance, ∈ is a very small value used to prevent the denominator from being zero, and γ and β are the scaling parameter and the offset parameter, which perform a linear transformation on the standardized data.

[0084] The ReLU activation function has the characteristics of low computational complexity and can effectively alleviate the problems of gradient explosion and gradient disappearance that may occur in the training of neural networks. The formula is:

[0085] f(x) = max(0, x) (5)

[0086] The structure of the residual block can alleviate the phenomenon of gradient disappearance or gradient explosion caused by the deepening of the network layers, and at the same time can reduce the occurrence of overfitting in deep networks.

[0087] Step 4.1: Manually label the abnormal pattern labels for the SHM datasets of multiple bridges, and divide the datasets into a training set and a validation set according to the ratio of 8:2. The training set should mainly include the data samples of the bridges to be measured. For the abnormal patterns with a small number, use the data samples of other bridges for supplementation to avoid the influence of data imbalance.

[0088] Step 4.2: Train the network weight parameters suitable for abnormal data recognition based on the designed multi-modal convolutional neural network model, adjust the hyperparameters according to the training results, and obtain the optimal training model.

[0089] The hyperparameters of the multi-modal convolutional neural network in the present invention include the learning rate, the size of batch input, the regularization coefficient, etc., and the hyperparameters are optimized according to the training results during the training process. The number of iterations is set to 50, and the model is trained. The training process is as Figure 5 shown. As the number of iterations increases, the training accuracy and the validation accuracy increase steadily. When the number of iterations reaches 50, the validation accuracy exceeds 98%. The training results show that in the case of fusing multi-bridge SHM data samples, the data structure of the training set is more balanced. At the same time, the multi-modal convolutional neural network model effectively combines image features and statistical features, and also retains the independent representation ability of each modality, and can realize automatic recognition and classification of SHM data anomalies with high accuracy.

[0090] Step 5: Use the trained convolutional neural network model to identify anomalies in the target bridge SHM data.

[0091] Step 5.1: Segment the SHM data of the bridge to be measured and convert it into a grayscale image, hiding the coordinate axes.

[0092] Step 5.2: Input the data samples of the bridge to be measured into the trained multi-modal convolutional neural network model for recognition and classification to obtain the abnormal patterns to which they belong.

[0093] The present invention uses a multi-modal convolutional neural network model to identify abnormal data, fuses the SHM time-series data of multiple bridges, fuses the SHM time-series data of multiple bridges and combines their image features and statistical features, and synchronously inputs them into the designed multi-modal convolutional neural network for training, and automatically identifies and classifies different patterns of anomalies in the bridge SHM data with high accuracy according to the deep learning algorithm, providing a basis for bridge operation status evaluation and early warning.

[0094] Embodiment 2:

[0095] The present invention provides a multi-bridge monitoring data anomaly recognition system based on multi-modal feature fusion. This system can be used to implement the above-mentioned multi-bridge monitoring data anomaly recognition method based on multi-modal feature fusion. Specifically, the system includes:

[0096] Data collection module: used to collect the SHM time-series data of multiple bridges, including: acceleration, deflection, stress, and temperature data, to form an original time series data set;

[0097] Data preprocessing module: used to perform normalization processing on the original time series data set and segment it into a series of fixed-length segments to obtain normalized time series data segments;

[0098] Feature extraction and annotation module: used to convert the obtained normalized time-series data segments into grayscale images, extract statistical features from the time-series data, and manually label the images and statistical features with tags, and output a grayscale image dataset, a statistical feature dataset, and labeled normal / anomaly tags;

[0099] Deep learning training module: Based on the output results of the feature extraction and annotation module, a multi-modal convolutional neural network model is established. Using datasets from multiple bridges, the grayscale images and corresponding statistical features are simultaneously input into the network model for training to obtain a trained multi-modal deep learning model;

[0100] Anomaly detection module: Use the trained multi-modal deep learning model to identify anomalies in the SHM data of the target bridge.

[0101] Embodiment 3:

[0102] This embodiment provides a terminal device, which includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the multi-bridge monitoring data anomaly recognition method based on multi-modal feature fusion, including the following steps:

[0103] Step 1: Collect SHM time-series data (acceleration, deflection, stress, temperature, etc.) of multiple bridges;

[0104] Step 2: Normalize the original SHM time-series data of multiple bridges and divide it into a series of fixed-length segments;

[0105] Step 3: Visualize the normalized time-series data, convert it into a grayscale image, extract statistical features from the time-series data at the same time, and manually label the images and statistical features with tags to construct a dataset;

[0106] Step 4: Establish a multi-modal convolutional neural network model based on the input of the grayscale image and statistical features. Use the dataset from multiple bridges, and input the grayscale image and the corresponding statistical features into the network model for training simultaneously;

[0107] Step 5: Use the trained convolutional neural network model to perform anomaly recognition on the SHM data of the target bridge.

[0108] Example 4:

[0109] This example provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is the memory device in the terminal device, used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides storage space, and this storage space stores the operating system of the terminal. And, in this storage space, there are also stored one or more instructions suitable for being loaded and executed by the processor. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0110] One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the multi-bridge monitoring data anomaly recognition method based on multi-modal feature fusion in the above examples; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0111] Step 1: Collect the SHM time-series data (acceleration, deflection, stress, temperature, etc.) of multiple bridges;

[0112] Step 2: Normalize the original SHM time-series data of multiple bridges and divide it into a series of fixed-length segments;

[0113] Step 3: Visualize the normalized time-series data, convert it into a grayscale image, extract the statistical features in the time-series data at the same time, and manually label the image and the statistical features to construct a dataset;

[0114] Step 4: Establish a multi-modal convolutional neural network model based on the input of the grayscale image and statistical features. Use the dataset from multiple bridges, and input the grayscale image and the corresponding statistical features into the network model for training simultaneously;

[0115] Step 5: Use the trained convolutional neural network model to perform anomaly recognition on the target bridge SHM data.

[0116] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0117] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0118] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0120] The above has described the preferred embodiments of the present invention in detail, but the present invention is not limited to the above embodiments. Within the knowledge scope of those of ordinary skill in the art, various changes can be made without departing from the gist of the present invention.

[0121] Many other changes and modifications can be made without departing from the spirit and scope of the present invention. It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for automatically identifying abnormalities in multi-bridge monitoring data based on multimodal feature fusion, characterized in that: The method comprises: S1: Normalize the original time series data set and divide it into a series of fixed-length segments to obtain normalized time series data segments; S2: Convert the normalized time series data segments into grayscale images, extract statistical features from the time series data, and manually label the images and statistical features, outputting grayscale image datasets, statistical feature datasets, and labeled normal / abnormal labels; S3: Based on the output result of step S2, a multimodal convolutional neural network model is established, and the grayscale image and the corresponding statistical features are simultaneously input into the network model for training using data sets from multiple bridges to obtain a trained multimodal deep learning model; S4: Use the trained multimodal deep learning model to identify anomalies in the SHM data of the target bridge.

2. According to claim 1, a method for automatically identifying abnormalities in multi-bridge monitoring data based on multimodal feature fusion, characterized in that: Before step S1, the method further includes: The SHM time series data of multiple bridges, including acceleration, deflection, stress and temperature data, are collected to form the original time series data set.

3. The method for automatically identifying abnormalities in multi-bridge monitoring data based on multimodal feature fusion according to claim 1 is characterized in that: The step S2 comprises: S201: read the segmented time series data, draw each sample data into a 224×224 grayscale image, and extract statistical features. The image and statistical features are saved in png and npy formats respectively; S202: Manually label the generated data images and statistical feature samples with abnormal pattern labels.

4. The method for automatically identifying abnormalities in multi-bridge monitoring data based on multimodal feature fusion according to claim 1 is characterized in that: The statistical features include: peak intensity, standard deviation, linearity and linear fitting slope; in the image-based abnormal data classification method, the statistical features of the data are introduced to enhance the distinction between the abnormal patterns; The peak intensity formula is: In formula (1), D 0.9 is the distance between the upper and lower bounds of the data value where 90% of the data points in the sample are located, x max is the maximum value in the data, x min It is the minimum value in the data. This characteristic index is mainly used to identify outliers in the data. The standard deviation formula is: In formula (2), x i is the value of each data point, μ is the sample mean, and N is the total number of data points in the sample. This characteristic indicator is mainly used to identify sub-minimum and over-range anomalies in the data; The linearity formula is: In formula (3), max(|Δx i |) is the maximum absolute value of the deviation between the smooth curve and the fitted straight line of all data points in the sample, x max and x min are the maximum and minimum values ​​in the sample, respectively. This characteristic indicator is mainly used to identify drift anomalies in the data; The linear fit slope is the slope of the fitted line obtained by using the least squares linear regression method for all data points in the sample. It indicates the degree of baseline deviation. This characteristic indicator is used to identify trend anomalies in the data.

5. The method for automatically identifying abnormalities in multi-bridge monitoring data based on multi-modal feature fusion according to claim 1 is characterized in that: The step S3 specifically includes: The multimodal convolutional neural network model includes a convolution layer, a pooling layer and a fully connected layer, and the overall structure is divided into three parts: an image encoder, a statistical feature encoder and a fusion classifier; wherein the image encoder is based on the ResNet neural network, removes the last fully connected layer, receives a grayscale image input of size 224×224, extracts the image features therein, and outputs 512 image feature vectors; the statistical feature encoder consists of two fully connected layers, with the ReLU function activated in the middle, receives the numerical input corresponding to the four statistical features and maps them to 512-dimensional features, which is consistent with the dimension of the image features; the fusion classifier splices and fuses the extracted image features and statistical features, and realizes the final classification through the fully connected layer; The training steps of the multimodal convolutional neural network model include: S301: Manually label abnormal patterns of SHM data sets of multiple bridges, and divide the data sets into training set and validation set in a ratio of 8:

2. The training set should mainly be composed of data samples of the bridge to be tested, and other bridge data samples should be used to supplement the abnormal patterns with a small number. S302: Based on the designed multimodal convolutional neural network model, network weight parameters suitable for abnormal data recognition are trained, and hyperparameters are adjusted according to the training results to obtain the optimal training model.

6. The method for automatically identifying abnormalities in multi-bridge monitoring data based on multi-modal feature fusion according to claim 1 is characterized in that: The ResNet neural network introduces a residual block structure, which consists of two convolutional layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function; The purpose of batch normalization is to make the data distribution of each layer input in the network more stable and accelerate the learning speed of the model. The batch normalization formula is as follows: In formula (4): x i is the input sample, μ is the sample mean, σ is the sample variance, ∈ is a minimum value used to prevent the denominator from being zero, γ and β are scaling parameters and offset parameters, which linearly transform the standardized data; The ReLU activation function has the characteristics of low computational complexity and can effectively alleviate the gradient explosion and gradient disappearance problems that may occur in neural network training. The formula is: f(x)=max(0,x) (5).

7. The method for automatically identifying abnormalities in multi-bridge monitoring data based on multimodal feature fusion according to claim 1 is characterized in that: The step S4 comprises: The SHM data of the bridge to be tested are segmented and converted into grayscale images and four statistical features are extracted. The data samples of the bridge to be tested are input into the trained multimodal convolutional neural network model for recognition and classification to obtain the corresponding abnormal patterns.

8. An automatic recognition system for abnormalities in multi-bridge monitoring data based on multimodal feature fusion, characterized in that: The system is applied to the method described in any one of claims 1 to 7, and the system comprises: Data collection module: used to collect SHM time series data of multiple bridges, including acceleration, deflection, stress and temperature data, to form the original time series data set; Data preprocessing module: used to normalize the original time series data set and divide it into a series of fixed-length segments to obtain normalized time series data segments; Feature extraction and labeling module: used to convert the normalized time series data segments into grayscale images, extract statistical features from the time series data, manually label the images and statistical features, and output grayscale image data sets, statistical feature data sets, and labeled normal / abnormal labels; Deep learning training module: Based on the output results of the feature extraction and annotation module, a multimodal convolutional neural network model is established. Using data sets from multiple bridges, grayscale images and corresponding statistical features are simultaneously input into the network model for training to obtain a trained multimodal deep learning model. Anomaly detection module: Use the trained multimodal deep learning model to identify anomalies in the SHM data of the target bridge.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for automatically identifying abnormalities in multi-bridge monitoring data based on multimodal feature fusion as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for automatically identifying abnormalities in multi-bridge monitoring data based on multi-modal feature fusion as described in any one of claims 1 to 7.

Images