A method and device for identifying data image migration between different bridges
By using the ResNet 50 network model for data transfer learning in bridge monitoring, the problem of imbalanced bridge monitoring data was solved, and the accuracy of abnormal data identification was improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-27
- Publication Date
- 2026-03-24
AI Technical Summary
In bridge monitoring, sensors are exposed to the outdoor environment for a long time, which leads to data anomalies. Traditional manual inspections are inefficient, and existing automated methods have low accuracy in identifying data imbalances.
By transferring monitoring data from other bridges to the target bridge dataset and using a pre-trained ResNet 50 network model for transfer learning, the target bridge dataset is expanded, improving the model's accuracy in identifying various data categories.
It alleviated the problem of data imbalance and improved the accuracy of abnormal data identification for target bridges, especially significantly improving the identification rate for certain types of data.
Smart Images
Figure CN116229165B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of engineering structure monitoring data analysis, and particularly relates to a method and device for identifying data image migration between different bridges. BACKGROUND
[0002] A structure health monitoring system is widely used for managing and maintaining infrastructure and making early judgments on the condition of engineering structures to avoid potential safety risks. The structure health monitoring system involves a large number of sensor devices. However, due to the long-term exposure of these sensors to outdoor environments, the sensors are affected by natural environments or external factors, resulting in sensor failure and multiple data anomalies collected, which need to be cleaned by anomaly detection technology.
[0003] Traditional manual inspection can easily identify missing data, jump points and other data anomalies. However, due to the very large amount of monitoring data, the efficiency of manually identifying abnormal data is very limited, which will cause a large waste of time and labor costs. Therefore, taking automatic means to identify abnormal data can greatly reduce time and labor costs, and is more conducive to engineering practical application.
[0004] With the development of machine learning and deep learning, data anomaly detection methods based on convolutional neural networks have been developed. Ni, Zhang, and Noori (2020) designed a one-dimensional convolutional neural network to extract directly from the input signal and detected acceleration data from a long-span suspension bridge SHM system in China. Zhang et al. (2022) proposed a new convolutional neural network framework that first down-samples, stacks, and inputs acceleration data into a CNN as a training set, designs and trains a CNN model, and obtains intermediate results through the model. Then, statistical features are applied to analyze and classify ambiguous patterns. It combines the advantages of CNN and statistical features, and can achieve data anomaly detection faster and more accurately than using CNN alone. Shajihan et al. (2022) converted time series data into 3-channel images composed of time, spectral graph, and probability density function, and then used a designed convolutional neural network to identify the images, achieving high overall accuracy and recall rate.
[0005] In order to reduce the training time of the network model and obtain better image recognition effect, the transfer learning is applied to the convolutional neural network, and the pre-trained model is used for image recognition. However, due to accidental factors, the number of images of a certain type of data in the selected data set may not be sufficient, which will affect the recognition of the model to the data of this type. Based on this, the application proposes a data transfer method, which transfers other bridge data to the target bridge data set, expands the data of a certain type of data in the target bridge data set, so that the model can complete the classification of all types of abnormal data, and solves the problem that the model cannot accurately recognize the data of a certain type due to the lack of data of the target bridge. SUMMARY
[0006] The application aims to provide a different bridge data image transfer recognition method and device, which solves the problem of data imbalance in the transfer learning process.
[0007] The technical scheme of the application is as follows: a different bridge data image transfer recognition method, converting bridge monitoring data time domain into an image, classifying the image according to the contour features of the converted image; the pre-trained model is a Resnet 50 network model, and the transfer learning method is adopted to train the classified image by using the adjusted network model; for the problem of unbalanced target bridge data set, the data of each type of other bridge data is supplemented to improve the recognition accuracy of the Resnet 50 network model for each type of data of the target bridge.
[0008] Specifically, the steps include the following:
[0009] Step 1: image conversion;
[0010] The sampling frequency of the acceleration sensor arranged on the bridge is f=x Hz, x data are collected per second, and 3600*x data are collected per hour at the point; taking time as the horizontal coordinate and the acceleration value at the time point as the vertical coordinate, the data points collected by the acceleration sensor in one hour are connected in sequence to draw a two-dimensional image by using Matlab, and the contour features of the image represent the main features of the acceleration data in one hour;
[0011] Step 2: abnormal data feature classification;
[0012] According to the contour features of the image, the bridge monitoring data is divided into six data categories: normal, drift, local gain, missing, noise and jump point;
[0013] Step 3: other bridge data quality judgment;
[0014] Other bridge monitoring data image data sets are selected, and the adjusted Resnet 50 network model is trained to judge the image quality in the other bridge monitoring data image data sets;
[0015] Step four: data set expansion;
[0016] Other bridge monitoring data images meeting the judgment condition are added to the target bridge monitoring data image training set, and the target bridge monitoring data image data set is expanded;
[0017] Step five: taking transfer learning to train the expanded target bridge monitoring data image data set by using the adjusted Resnet 50 network model;
[0018] The pre-trained model in the MATLAB toolbox mainly uses the adjusted Resnet 50 network model; the adjusted Resnet 50 network model is as follows: a new fully connected layer and a classification layer are set, the internal parameters InputSize of the new fully connected layer and the internal parameters OutputSize of the new classification layer are set to d; the original fully connected layer and the classification layer of the Resnet 50 network model are searched in the network layer of the adjusted Resnet 50 network model, and the new fully connected layer and the new classification layer are replaced; the hyperparameters of the adjusted Resnet 50 network model are initialized and set, the expanded target bridge monitoring data image data set is input to train the adjusted Resnet 50 network model, and the hyperparameters are adjusted according to the accuracy and loss function curve obtained by training.
[0019] The acceleration amplitude of the normal data image has an orderly fluctuation above and below the image center line, and the fluctuation is obvious; the overall data of the drift data image deviates from the center line and presents an inclined trend or partially deviates from the fluctuation above and below the center line. In the gain data image, the acceleration amplitude in part of the time period presents a sudden increase trend, and the amplitude is much higher than that in other time periods. In the noise data image, the acceleration amplitude in the whole time period tends to be consistent, and there is no obvious fluctuation. In the missing data image, part of the data is missing, the acceleration amplitude at the missing position is 0, and a straight line is displayed at the image center line, or all the data is missing and the whole image is blank. In the jump point data image, the acceleration amplitude suddenly increases at a certain moment, and one or more isolated data points are presented in the image, and the overall feature is obviously reduced.
[0020] A recognition device for data image migration between different bridges comprises:
[0021] The acquisition module is configured to acquire bridge monitoring data;
[0022] The storage is configured to store the acquired bridge monitoring data and a computer program; the computer program comprises an image conversion program and an image training program;
[0023] a processor for executing a computer program stored in the memory, when the computer program is executed, the processor is used for:
[0024] reading the stored bridge monitoring data, the bridge monitoring data is collected and stored at the same sampling frequency; according to the obtained bridge monitoring data, the bridge monitoring data is converted into an image by using an image conversion program; and then the expanded target bridge monitoring data image data set is trained by using an image training program.
[0025] The method and device provided by the application can alleviate the data imbalance problem, avoid the problem that the selected target bridge data set has too few data images of a certain type, and make the model have a low recognition accuracy for the data of this type. The target bridge monitoring data image data set expanded by the method has a significantly improved recognition accuracy of the network model for the data of this type. BRIEF DESCRIPTION OF DRAWINGS
[0026] Fig. 1(a) is a time-domain image of converted data on a target bridge;
[0027] Fig. 1(b) is a time-domain image of converted data on a non-target bridge;
[0028] Fig. 2(a) is a main feature of normal data in a time-domain image;
[0029] Fig. 2(b) is a main feature of drift data in a time-domain image;
[0030] Fig. 2(c) is a main feature of local gain data in a time-domain image;
[0031] Fig. 2(d) is a main feature of noise data in a time-domain image;
[0032] Fig. 2(e) is a main feature of missing data in a time-domain image;
[0033] Fig. 2(f) is a main feature of jump point data in a time-domain image;
[0034] Figure 3 is a curve graph of the recognition accuracy of a non-target bridge;
[0035] Figure 4 is a process graph of image recognition by an adjusted Resnet 50 network model in a pre-training model;
[0036] Fig. 5(a) is a confusion matrix graph of target bridge data migration before a bridge;
[0037] Fig. 5(b) is a confusion matrix graph of target bridge data migration after a bridge. DETAILED DESCRIPTION
[0038] The embodiments of the application are further illustrated in combination with the technical solutions.
[0039] Taking a certain bridge as an example, anomaly detection was performed on its acceleration data using a ResNet 50 network model adjusted from a pre-trained model.
[0040] Step 1: Image Conversion
[0041] The accelerometers deployed on the target bridge have a sampling frequency of f = 200Hz, collecting 200 data points per second, resulting in 3600 * 200 = 720,000 acceleration data points collected per hour. Using time as the x-axis and the acceleration value at that time point as the y-axis, Matlab was used to connect and plot the 720,000 data points collected by the accelerometers in one hour into a two-dimensional image. The features of this image roughly represent the main characteristics of the acceleration data within one hour. The converted image is shown in Figure 1(a). The accelerometers deployed on the non-target bridge have a sampling frequency of f = 100Hz, collecting 100 data points per second, resulting in 3600 * 100 = 360,000 acceleration data points collected per hour. Using time as the x-axis and the acceleration value at that time point as the y-axis, Matlab was used to connect and plot the 360,000 data points collected by the sensor in one hour into a two-dimensional image. The features of this image roughly represent the main characteristics of the acceleration data within one hour. The converted image is shown in Figure 1(b).
[0042] Step 2: Classification of Abnormal Data Features
[0043] Based on the contour features of the converted images, the images are divided into six data categories: normal, drift, local gain, missing, noise, and jump points, and stored in different folders. The main features of each data category are shown in Figure 2.
[0044] Step 3: Quality assessment of other bridge data
[0045] The adjusted ResNet50 network model was trained on a dataset of images of non-target bridges, and the accuracy change curve was obtained after training. Figure 3 ,from Figure 3 The curves of the training set and the validation set are closely aligned, indicating a good fit between them. The vertical axis shows that both the training accuracy and the validation accuracy are over 90%, demonstrating that the model trained on this dataset has a good classification ability for the six classes of data. This dataset can be used as an expanded training set.
[0046] Step 4: Dataset Expansion
[0047] (4) Add the non-target bridge data image set to the target bridge training set to expand the target bridge data set. According to the comparison between the data amount of each category of the original target bridge data set in Table 1 and the data amount of each category of the expanded target bridge data set in Table 2, it can be seen that the quantity expansion presents a change of order of magnitude;
[0048] Step five: take transfer learning to train the data set by using the pre-trained model;
[0049] Table 1: Data amount of each category of the original target bridge data set
[0050] Data category Number of images Normal 1463 Drift 4621 Local gain 244 Noise 21293 Missing 10503 Jump 1524
[0051] Table 2: Data amount of each category of the expanded target bridge data set
[0052] Data category Number of images Normal 16076 Drift 6423 Local gain 1653 Noise 21558 Missing 12353 Jump 1677
[0053] The pre-trained model in the MATLAB toolbox, i.e. the adjusted Resnet 50 network model, is set for a 1000-output category task. In order to make it adapt to the 6-output task of bridge anomaly data detection, the pre-trained model is fine-tuned. The fine-tuning operation is as follows: first, set a new fully connected layer and a classification layer, set the internal parameter InputSize of the new fully connected layer to 6, and set the internal parameter OutputSize of the new classification layer to 6. Then search for the fully connected layer and the classification layer in the network layer of the pre-trained model automatically, and replace them with the newly defined fully connected layer and classification layer. Then initialize the hyperparameters, input the training set to train the data image; set the hyperparameters of the adjusted Resnet 50 network model preliminarily, initialLearnRate-0.01, optimizer-SGD, MiniBitchsize-32, MaxEpochs-1. The identification process of the pre-trained model, i.e. the adjusted Resnet 50 network model, is as follows: Figure 4As shown, the confusion matrix after training is shown in FIG. 5. FIG. 5(a) is a confusion matrix obtained by training the model of the target bridge data set before data migration, and then testing the model using the data of the bridge in May. From FIG. 5(a), it can be seen that the model has a serious low recognition rate for normal, drift, and local gain data compared to noise, missing, and jump point data. The adjusted Resnet 50 network model cannot effectively identify the above three types of data. The recognition accuracy of the adjusted Resnet 50 network model has a certain relationship with the image data. In theory, the more the number of images, the more sufficient the model learns the characteristics of the data, and the higher the recognition accuracy of the model for this type of data in the final test. The reason for the low recognition accuracy of the local gain category data in FIG. 5(a) can be attributed to the small amount of training data for the above category and the serious data imbalance. To illustrate the effectiveness of data migration, a comparison verification is performed. The target bridge data is migrated, and the non-target bridge data is used to expand the data set. The network model is retrained, and then the target bridge data in May is used for testing to obtain FIG. 5(b). From FIG. 5(b), it can be seen that the overall accuracy of the adjusted Resnet 50 network model for the six types of data is 93.1%, and the recognition ability is good. The recognition recall rate of normal data, drift, and local gain data is significantly improved, increasing by 30%, 8%, and 22.5%, respectively. The recognition accuracy of noise, missing, and jump point data is normal fluctuation, with no significant decrease. After expanding the target bridge data images, the recognition of normal data, drift, and local gain data is significantly improved, which illustrates the effectiveness of this method.
Claims
1. A method for recognizing data image migration between different bridges, characterized in that, Bridge monitoring data is converted into images in the time domain. Based on the contour features of the converted images, the images are classified. A transfer learning method is adopted, and the adjusted ResNet 50 network model is used to train the classified images. To address the imbalance problem of the target bridge dataset, data from other bridges are used to supplement the data of each category, thereby improving the recognition accuracy of the adjusted ResNet 50 network model for each category of the target bridge. The specific steps are as follows: Step 1: Image conversion; The sampling frequency of the accelerometers deployed on the bridge is f= Hz, sampling per second This data point is collected at an acceleration rate of 3600* per hour. Next, using time as the horizontal axis and the acceleration value at that time point as the vertical axis, Matlab is used to connect the data points collected by the accelerometer in one hour to draw a two-dimensional image. The contour features of this image represent the main features of the acceleration data in one hour. Step 2: Classification of abnormal data features; Based on the contour features of the images, the bridge monitoring data are divided into six categories: normal, drift, local gain, missing, noise, and jump points. Step 3: Quality assessment of other bridge data; Other bridge monitoring image datasets were selected, and an adjusted ResNet 50 network model was trained to evaluate the image quality in other bridge monitoring image datasets. Step 4: Dataset expansion; Other bridge monitoring data images that meet the judgment criteria are added to the target bridge monitoring data image training set to expand the target bridge monitoring data image dataset; Step 5: Employ transfer learning to train the adjusted ResNet 50 network model on the expanded target bridge monitoring data image dataset; The pre-trained model in the MATLAB toolbox is mainly an adjusted ResNet50 network model. The adjusted ResNet50 network model is as follows: A new fully connected layer and a classification layer are set, with the inputSize of the new fully connected layer and the outputSize of the new classification layer set to d. Existing fully connected and classification layers in the adjusted ResNet50 network model are searched for and replaced with the new fully connected and classification layers. The hyperparameters of the adjusted ResNet50 network model are initialized, and the expanded target bridge monitoring image dataset is input for training. The hyperparameters are then adjusted based on the accuracy and loss function curves obtained during training.
2. The method for recognizing data image migration between different bridges according to claim 1, characterized in that, The normal data image shows orderly fluctuations along the center line, with obvious fluctuations; the drift data image deviates from the center line as a whole, showing a tilting trend or partially deviating from the center line while fluctuating up and down; the gain data image shows a sudden increase in acceleration amplitude during certain time periods, with its amplitude higher than that of data in other time periods; the noisy data image shows a consistent acceleration amplitude throughout the entire time period; the missing data image shows partial data loss, with the acceleration amplitude at the missing point being 0, displaying a straight line at the image center line, or all data loss resulting in a blank image; the jump point data image shows a sudden increase in acceleration amplitude at a certain moment, presenting one or more isolated data points, and the overall feature is reduced.
3. The method for recognizing data image migration between different bridges according to claim 1 or 2, characterized in that, The value of d is 6.
4. A device for identifying data image migration between different bridges, used to implement the identification method according to any one of claims 1-3, characterized in that, The device for recognizing data image migration between different bridges includes: The data acquisition module is used to acquire bridge monitoring data; The memory is used to store the acquired bridge monitoring data and computer programs; the computer programs include image conversion programs and image training programs. A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, the processor is configured to: Read the stored bridge monitoring data, which is collected and stored at the same sampling frequency; based on the acquired bridge monitoring data, use an image conversion program to convert the bridge monitoring data into images; then use an image training program to train the expanded target bridge monitoring data image dataset.