A bridge vibration displacement visual measurement method and system based on deep learning
Through deep learning network model, combined with technologies such as ResNet50 and feature pyramid module, high-precision and low-cost measurement of bridge vibration displacement is achieved, noise and error problems in traditional methods are solved, and non-contact and long-distance measurement capabilities are provided.
Patent Information
- Application Number
- CN202111520529.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-13
AI Technical Summary
In traditional bridge vibration measurement methods, acceleration sensors are susceptible to noise, fiber optic sensors are expensive and have limited contact distances, large visual displacement measurement errors, and existing object detection algorithms are not suitable for complex scenarios of large-span bridges.
Using the visual measurement method of bridge vibration displacement based on deep learning, the deep learning network model is constructed, and the ResNet50 feature extraction, feature pyramid module, path enhancement module, regional candidate network module and side-perceptual boundary positioning module are used to realize non-contact, long-distance, Multi-point synchronization measurement of bridge vibration displacement.
High-precision and low-cost bridge vibration displacement measurement is achieved, avoiding noise influence and error problems in traditional methods, and has the advantages of non-contact and long-distance measurement, and can accurately detect vibration displacement on large-span bridges.
Smart Images

Figure CN114972153B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a deep learning-based bridge vibration displacement visual measurement method and system, and belongs to the field of visual vibration displacement measurement and computer vision. Background Art
[0002] In actual scenarios, large-span bridges will generate large-amplitude vibrations after being disturbed by external forces, affecting the normal operation of traffic, driving safety and the service life of the bridge. Therefore, monitoring the vibration displacement of large-span bridges is of great significance for their health monitoring and bridge safety early warning.
[0003] In traditional bridge vibration measurement and detection methods, commonly used sensors include accelerometers and fiber optic sensors. Accelerometers are one of the most commonly used sensors in detection because of their small size, high reliability, and wide measurement frequency range. However, in actual measurements, acceleration signals are often mixed with various noise signals, and fiber optic sensors are expensive and have limited non-contact distance. Therefore, using traditional sensors to measure the displacement of long-span bridges will face many challenges. Furthermore, the template matching algorithm based on target detection is suitable for single-scene measurements with large color differences between the target and the environment, and is not suitable for target matching with higher requirements in actual application scenarios. The tracking algorithm commonly used in visual displacement measurement will have large errors during the displacement measurement process due to the influence of sub-pixel accuracy. Summary of the invention
[0004] The present invention provides a deep learning-based visual measurement method and system for bridge vibration displacement, which are used to realize the visual measurement of bridge vibration displacement in a non-contact manner.
[0005] The technical solution of the present invention is: a visual measurement method for bridge vibration displacement based on deep learning, comprising:
[0006] Using an image acquisition device to collect a bridge structure data set and divide it into a training data set and a verification data set;
[0007] Using annotation tools, the training data set and the validation data set are annotated to obtain the training set and the validation set respectively;
[0008] Build deep learning network models;
[0009] Modify the hyperparameters in the configuration file to obtain the training parameters;
[0010] Call the training set and configuration file to start training the deep learning network model, and obtain the candidate weights after the training.
[0011] The performance of the candidate weights is evaluated using a validation set to quantify the performance of the candidate weights, and the optimal weights are loaded into the deep learning network model to obtain the deep learning network model after the optimal weights are loaded;
[0012] The newly acquired image of the bridge structure to be inspected is input into the deep learning network model with the optimal weights for inspection to obtain the inspection bounding box.
[0013] It also includes: verification of the deep learning network model after loading the optimal weights: verifying the error between the bounding box obtained by the present invention and the target edge by comparing the performance with traditional digital image processing algorithms and deep learning algorithms with various different positioning principles; performing curve fitting comparison on the displacement curve obtained by the bounding box according to the present invention and the displacement curve obtained by the displacement signal collected by the acceleration sensor, performing fast Fourier transform on the displacement curve, and comparing the frequency domain diagrams obtained by the fast Fourier transform; jointly realizing the verification of the time-frequency domain characteristics.
[0014] The displacement curve obtained by the boundary box according to the present invention is specifically as follows: the position information of the target in the image of the first frame is used as a reference frame for calculating the displacement offset, the vibration displacement of the bridge structure in the pixel coordinate system is obtained by calculating the displacement offset between the remaining frames and the first frame, and then the vibration displacement in the pixel coordinate system is converted into the physical coordinate system to obtain the real displacement curve.
[0015] The image acquisition device acquires the bridge structure data set specifically by: using a high-speed camera to capture vibration data of the excited bridge model according to the shooting speed.
[0016] The labeling is specifically as follows: the training data set and the validation data set are labeled by the labelImg toolkit, and after the labeling is completed, a training set for training and a validation set for validation are obtained; wherein the specific operation of the labeling is to label the detection targets on the bridge in the data set image with a labeling box of the same length and width.
[0017] The construction of the deep learning network model specifically includes: using the backbone network of ResNet50 feature extraction, the feature pyramid module, the path enhancement module, the region candidate network module and the side perception boundary positioning module to construct the deep learning network model.
[0018] The backbone network of the ResNet50 feature extraction is divided into 4 residual blocks to obtain feature maps S1, S2, S3, S4, and S5; wherein the number of residual units used in the residual blocks is 3, 4, 6, and 3 respectively;
[0019] The feature pyramid module obtains feature maps P2, P3, P4, P5 by top-down, lateral connection and convolution fusion operations based on feature maps S2, S3, S4, S5 output by the backbone network; wherein the nearest neighbor upsampling method is used in the top-down upsampling process;
[0020] The path enhancement module is based on the feature maps P2, P3, P4, and P5 output by the feature pyramid module. The path enhancement starts from the feature map P2 and gradually approaches the feature map P5 to obtain feature maps N2, N3, N4, and N5; wherein the feature map N2 is the same as the feature map P2;
[0021] The region candidate network module first passes through a 3×3 convolution respectively according to the feature maps N2-N5 output by the path enhancement module, and then passes through two 1×1 convolutions to obtain a classification branch and a regression branch; the classification branch is a binary classification of the target and the background, generating a total of 2K outputs, and the regression branch is to determine the candidate box position of the target area, each box contains 4 values, and a total of 4K outputs are generated; the 6K outputs are ROIAligned with the above N2-N5 feature maps to obtain an ROI feature map; wherein K=W×H×m, W and H are the width and height of the feature map output by the path enhancement module, respectively, and m represents the number of anchor boxes generated by each anchor point;
[0022] The side perception boundary positioning module relaxes the target area in the ROI feature map output by the region candidate network module according to the proportional coefficient σ, and compresses the relaxed ROI feature map through 1×1 convolution to obtain feature F. Feature F is strengthened by self-attention in the X-axis direction and Y-axis direction respectively to obtain X-axis feature map and Y-axis feature map. Then, feature F is aggregated along the X-axis and Y-axis by 1×3 convolution and 3×1 convolution respectively to extract horizontal feature F. X and vertical feature F Y , and then to F X 、F Y Upsample to get features of size 1×2k and 2k×1, and then convert the horizontal feature F X and vertical feature F Y Segmented into side-aware features F left 、F right 、F top and F down ; Then perform bucket estimation to obtain the approximate boundary position B left , B right , B top and B down Finally, the fine-tuned boundary position X is obtained by the offset of the Bucker center line left , X right , X top and X down .
[0023] The training of the deep learning network model is as follows:
[0024] Train the deep learning network model; the training objects include: the coordinates and categories of the upper left corner and lower right corner of the annotation box; the training results are expressed in the form of weights obtained after training;
[0025] Call the file of the deep learning network model to start training. The specific process of training is as follows:
[0026] 1) Load the hyperparameters set in the file into the deep learning network model;
[0027] 2) Randomly select a batch size of images from the training set as the current training sample;
[0028] 3) Each image in the training sample is sequentially placed in the deep learning network model to update the weight parameters; the weight parameter update is specifically as follows: the weight parameters are randomly initialized and then the forward propagation calculation of the convolutional neural network is performed to obtain a set of intermediate parameters, and then the intermediate parameters are used for back propagation to update the weight parameters; the new weight parameters will iterate the old weight parameters used to calculate the forward propagation before;
[0029] 4) Complete the forward and backward propagation process for all images in the training sample as one training of the deep learning network model, and repeat steps 1) to 3) until the number of network model training reaches the set number of iterations;
[0030] 5) After the entire training is completed, filter out the candidate weight files.
[0031] A bridge vibration displacement visual measurement system based on deep learning, comprising:
[0032] A division unit, used for collecting a bridge structure data set by using an image acquisition device and dividing the data set into a training data set and a verification data set;
[0033] A labeling unit, used to label the training data set and the validation data set by using a labeling tool to obtain a training set and a validation set;
[0034] Construction unit, used to build deep learning network models;
[0035] A first obtaining unit is used to modify the hyperparameters in the configuration file to obtain the training parameters;
[0036] The second obtaining unit is used to call the training set and the configuration file to start training the deep learning network model, and obtain the candidate weights after the training;
[0037] The third obtaining unit is used to evaluate the performance of the candidate weights using the validation set to quantify the performance of the candidate weights, load the optimal weights into the deep learning network model, and obtain the deep learning network model after the optimal weights are loaded;
[0038] The fourth obtaining unit is used to input the newly acquired bridge structure image to be detected into the deep learning network model after loading the optimal weight for detection, so as to obtain the detected bounding box.
[0039] It also includes a verification unit for verifying the deep learning network model after loading the optimal weights: verifying the error between the bounding box obtained by the present invention and the target edge by performing a performance comparison with a traditional digital image processing algorithm and a deep learning algorithm with a plurality of different positioning principles; performing curve fitting comparison between a displacement curve obtained by the bounding box according to the present invention and a displacement curve obtained by a displacement signal collected by an acceleration sensor, performing a fast Fourier transform on the displacement curve, and comparing the frequency domain diagrams obtained by the fast Fourier transform; and jointly realizing the verification of the time-frequency domain characteristics.
[0040] The beneficial effects of the present invention are as follows: the present invention uses the bridge structure in the high-speed video as the object of vibration displacement measurement, introduces the target detection method based on deep learning into the field of visual vibration measurement, and verifies the feasibility of the deep learning method in visual vibration measurement from multiple angles, and the measurement method has the advantages of non-contact, long-distance, and multi-point synchronous measurement; specifically, firstly, ResNet50 is used to extract features of the input image, and the extracted features are divided into shallow feature maps and deep feature maps according to different convolution depths; further, a feature fusion module is added after the feature extraction network to fuse features of different scales and depths, which can ensure The feature map in the subsequent operation contains not only rich contour shapes and edge details, but also semantic information and key feature pixels that can express complex targets. This can avoid the deficiency of a single weak shallow semantic feature that is difficult to distinguish categories due to the large similarity of heterogeneous information, and can also avoid the deficiency of only extracting high-level semantic information unique to the category. On the basis of the above, adding a path enhancement module can enhance the fusion function of the feature pyramid and add accurate positioning information. Furthermore, the region candidate network module is used to obtain the predicted frame of the target and cooperate with the boundary-aware positioning module to finely locate and fine-tune the target area, so as to obtain the minimum true area to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a structural block diagram of the present invention;
[0042] Figure 2 Collecting drawings for bridge structures;
[0043] Figure 3 LabelImg tool to label the bridge detection target.
[0044] Figure 4 This is the schematic diagram of the ResNet50 residual unit;
[0045] Figure 5 It is the principle diagram of the feature pyramid network module and the path enhancement module;
[0046] Figure 6 This is the specific step diagram of the path enhancement module;
[0047] Figure 7 It is the schematic diagram of the regional candidate network module;
[0048] Figure 8 This is the schematic diagram of the side perception boundary positioning module;
[0049] Fig. 9 A diagram showing a specific implementation of the process of training a model;
[0050] Fig.10 A comparison chart of the detection targets of the network model and other traditional detection methods;
[0051] Fig.11 Enlarge the comparison chart for the test results;
[0052] Fig.12 Detect target result graph for deep learning network model;
[0053] Fig.13 This is the specific structure diagram of the deep learning network model;
[0054] Fig.14 It is the time domain diagram of the acceleration sensor and image displacement;
[0055] Fig.15 Fast Fourier transform spectra of displacement obtained in different ways. DETAILED DESCRIPTION
[0056] The invention is further described below in conjunction with the accompanying drawings and embodiments, but the content of the invention is not limited to the scope of the embodiments.
[0057] Example 1: Figure 1-15 As shown, a bridge vibration displacement visual measurement method based on deep learning includes:
[0058] Using an image acquisition device to collect a bridge structure data set and divide it into a training data set and a verification data set;
[0059] Using annotation tools, the training data set and the validation data set are annotated to obtain the training set and the validation set respectively;
[0060] Build deep learning network models;
[0061] Modify the hyperparameters in the configuration file to obtain the training parameters;
[0062] Call the training set and configuration file to start training the deep learning network model, and obtain the candidate weights after the training.
[0063] The performance of the candidate weights is evaluated using a validation set to quantify the performance of the candidate weights, and the optimal weights are loaded into the deep learning network model to obtain the deep learning network model after the optimal weights are loaded;
[0064] The newly acquired image of the bridge structure to be inspected is input into the deep learning network model with the optimal weights for inspection to obtain the inspection bounding box.
[0065] Furthermore, it can be arranged to also include: verification of the deep learning network model after loading the optimal weights: verifying the error between the bounding box obtained by the present invention and the target edge by comparing the performance with the traditional digital image processing algorithm and the deep learning algorithms with various different positioning principles; performing curve fitting comparison on the displacement curve obtained by the bounding box according to the present invention and the displacement curve obtained by the displacement signal collected by the acceleration sensor, performing fast Fourier transform on the displacement curve, and comparing the frequency domain diagram obtained by the fast Fourier transform; jointly realizing the verification of the time-frequency domain characteristics.
[0066] Furthermore, the displacement curve obtained by the boundary box according to the present invention can be set specifically as follows: the position information of the target in the image of the first frame is used as a reference frame for calculating the displacement offset, the vibration displacement of the bridge structure in the pixel coordinate system is obtained by calculating the displacement offset between the remaining frames and the first frame, and then the vibration displacement in the pixel coordinate system is converted into the physical coordinate system to obtain the real displacement curve.
[0067] Furthermore, the image acquisition device may be configured to acquire the bridge structure data set specifically by: acquiring vibration data of the excited bridge model by photographing with a high-speed camera according to a photographing speed.
[0068] Furthermore, the labeling can be specifically set as follows: labeling the training data set and the validation data set through the labelImg toolkit, and obtaining a training set for training and a validation set for validation after labeling; wherein the specific operation of labeling is to label the detection targets on the bridge in the data set image with a labeling frame of the same length and width. Since the present invention is mainly used for visual measurement of bridge vibration displacement, it is obvious that the present invention uses a labeling frame of the same size to label the detection targets on the bridge in the data set image, which is more in line with the motion change process of the detection targets on the bridge during bridge vibration.
[0069] Furthermore, a deep learning network model can be constructed by setting up a backbone network using ResNet50 feature extraction, a feature pyramid module, a path enhancement module, a region candidate network module, and a side perception boundary positioning module.
[0070] Furthermore, the backbone network of the ResNet50 feature extraction can be divided into 4 residual blocks to obtain feature maps S1, S2, S3, S4, and S5; wherein the number of residual units used by the residual blocks is 3, 4, 6, and 3 respectively;
[0071] The feature pyramid module obtains feature maps P2, P3, P4, P5 by top-down, lateral connection and convolution fusion operations based on feature maps S2, S3, S4, S5 output by the backbone network; wherein the nearest neighbor upsampling method is used in the top-down upsampling process;
[0072] The path enhancement module is based on the feature maps P2, P3, P4, and P5 output by the feature pyramid module. The path enhancement starts from the feature map P2 and gradually approaches the feature map P5 to obtain feature maps N2, N3, N4, and N5; wherein the feature map N2 is the same as the feature map P2;
[0073] The region candidate network module first passes through a 3×3 convolution respectively according to the feature maps N2-N5 output by the path enhancement module, and then passes through two 1×1 convolutions to obtain a classification branch and a regression branch; the classification branch is a binary classification of the target and the background, generating a total of 2K outputs, and the regression branch is to determine the candidate box position of the target area, each box contains 4 values, and a total of 4K outputs are generated; the 6K outputs are ROIAligned with the above N2-N5 feature maps to obtain an ROI feature map; wherein K=W×H×m, W and H are the width and height of the feature map output by the path enhancement module, respectively, and m represents the number of anchor boxes generated by each anchor point;
[0074] The side perception boundary positioning module relaxes the target area in the ROI feature map output by the region candidate network module according to the proportional coefficient σ, and compresses the relaxed ROI feature map through 1×1 convolution to obtain feature F. Feature F is strengthened by self-attention in the X-axis direction and Y-axis direction respectively to obtain X-axis feature map and Y-axis feature map. Then, feature F is aggregated along the X-axis and Y-axis by 1×3 convolution and 3×1 convolution respectively to extract horizontal feature F. X and vertical feature F Y , and then to F X 、F Y Upsample to get features of size 1×2k and 2k×1, and then convert the horizontal feature F X and vertical feature F Y Segmented into side-aware features F left 、F right 、F top and F down ; Then perform bucket estimation to obtain the approximate boundary position B left , B right , Btop and B down Finally, the fine-tuned boundary position X is obtained by the offset of the Bucker center line left , X right , X top and X down .
[0075] In the field of visual vibrometer, the slight error of whether the prediction box selects the minimum real area is extremely fatal in the field of visual vibrometer. It is the main reason for the large displacement offset of the target in the images of continuous frames. The model of the present invention is crucial to the detection of the precise positioning information of the target through the mutual cooperation of various modules.
[0076] Furthermore, the deep learning network model may be set to be trained as follows:
[0077] Train the deep learning network model; the training objects include: the coordinates and categories of the upper left corner and lower right corner of the annotation box; the training results are expressed in the form of weights obtained after training;
[0078] Call the file of the deep learning network model to start training. The specific process of training is as follows:
[0079] 1) Load the hyperparameters set in the file into the deep learning network model;
[0080] 2) Randomly select a batch size of images from the training set as the current training sample;
[0081] 3) Each image in the training sample in step 2) is sequentially placed in the deep learning network model to update the weight parameters; the weight parameter update is specifically as follows: randomly initialize the weight parameters and then perform forward propagation calculation of the convolutional neural network to obtain a set of intermediate parameters, and then use the intermediate parameters to perform back propagation to update the weight parameters; the new weight parameters will iterate the old weight parameters used to calculate the forward propagation before;
[0082] 4) The process of completing one forward and backward propagation for all images in the training sample described in step 2) is recorded as one training of the deep learning network model, and steps 1) to 3) are repeated until the number of network model training times reaches the set number of iterations;
[0083] 5) After the entire training is completed, filter out the candidate weight files.
[0084] A bridge vibration displacement visual measurement system based on deep learning, comprising:
[0085] A division unit, used for collecting a bridge structure data set by using an image acquisition device and dividing the data set into a training data set and a verification data set;
[0086] A labeling unit, used to label the training data set and the validation data set by using a labeling tool to obtain a training set and a validation set;
[0087] Construction unit, used to build deep learning network models;
[0088] A first obtaining unit is used to modify the hyperparameters in the configuration file to obtain the training parameters;
[0089] The second obtaining unit is used to call the training set and the configuration file to start training the deep learning network model, and obtain the candidate weights after the training;
[0090] The third obtaining unit is used to evaluate the performance of the candidate weights using the validation set to quantify the performance of the candidate weights, load the optimal weights into the deep learning network model, and obtain the deep learning network model after the optimal weights are loaded;
[0091] The fourth obtaining unit is used to input the newly acquired bridge structure image to be detected into the deep learning network model after loading the optimal weight for detection, so as to obtain the detected bounding box.
[0092] Furthermore, a verification unit may be provided to verify the deep learning network model after the optimal weights are loaded: the error between the bounding box obtained by the present invention and the target edge is verified by comparing the performance with the traditional digital image processing algorithm and the deep learning algorithms with different positioning principles; the displacement curve obtained by the bounding box of the present invention is compared with the displacement curve obtained by the displacement signal collected by the acceleration sensor by curve fitting, the displacement curve is fast Fourier transformed, and the frequency domain diagram obtained by the fast Fourier transform is compared; and the verification of the time-frequency domain characteristics is jointly realized.
[0093] Furthermore, this application provides the following specific implementation steps:
[0094] Step 1: Collect a bridge structure data set and divide it into a training data set and a verification data set;
[0095] Step 2: Use annotation tools to annotate the training data set and validation data set to obtain the training set and validation set respectively;
[0096] Step 3: Use the ResNet50 feature extraction backbone network, feature pyramid module, path enhancement module, region candidate network module and side perception boundary positioning module to build a deep learning network model;
[0097] Step 4: Before formal training, modify the hyperparameters in the configuration file to obtain the training parameters;
[0098] Step 5: Call the training set and configuration file to start training the deep learning network model, and obtain the candidate weights after the training.
[0099] Step 6: Evaluate the performance of the candidate weights using the validation set to quantify the performance of the candidate weights.
[0100] Step 7, conduct a comparative experiment on the deep learning network model, and verify the deep learning network model after loading the optimal weight: verify the error between the bounding box obtained by the present invention and the edge of the target by comparing the performance with the traditional digital image processing algorithm and the deep learning algorithms with different positioning principles; compare the displacement curve obtained by the bounding box according to the present invention with the displacement curve obtained by the displacement signal collected by the acceleration sensor by curve fitting, perform fast Fourier transform on the displacement curve, and compare the frequency domain graph obtained by the fast Fourier transform; jointly realize the verification of the time-frequency domain characteristics. The deep learning network model is composed of a ResNet50 feature extraction network, a feature pyramid network module, a path enhancement module, a region candidate network module, and a side perception boundary positioning module (i.e., Our in the figure). The traditional digital image processing algorithm is an ordinary NDSSM; the deep learning algorithms with different positioning principles mainly include those based on anchor boxes and those without anchor boxes, among which the classic algorithms based on anchor boxes include the single-stage SSD algorithm and the two-stage Faster-RCNN, and the algorithms without anchor boxes include CenterNet.
[0101] The bridge structure data set is the vibration data of the bridge model after being excited, which is obtained by camera shooting. In the absence of natural light, only ordinary LED fill light is used for light compensation during the acquisition process. The continuous frame resolution of the acquisition is 256x256. The acquired images are as follows Figure 2 At the same time, the NI9234 acquisition card and the PCB acceleration sensor were used to synchronously collect the vibration data of the bridge model after excitation, with a sampling frequency of 1.3kHz, for comparative experiments.
[0102] The bridge structure data set is divided into a training data set and a verification data set. In this embodiment, the flexible bridge structure data sets used in the present invention are all collected on the cable-stayed bridge model. A total of 13,000 cable-stayed bridge model image data sets are collected at a shooting speed of 1,300 frames per second by a high-speed camera; 1,000 collected cable-stayed bridge model vibration data sets are sequentially taken from the collected data, and the training data set and verification data set account for 80% and 20% of the bridge structure data set respectively: 800 training data sets and 200 verification data sets; subsequently, the bridge structure images that need to be tested can be added or collected for further testing.
[0103] The specific steps of step 2 are as follows: first set the labeling category to magnet on the labelImg toolkit, and label the training data set and the verification data set through the labelImg toolkit. After the labeling is completed, a training set for training and a verification set for verification are obtained; the specific operation of labeling is to use a labeling box of the same size to label the detection target on the bridge. The labeling effect of the labelImg toolkit is as follows: Figure 3 shown.
[0104] The residual unit principle of ResNet50 in step 3 is as follows Figure 4 As shown in the figure, the input is subjected to a 1×1 convolution for data feature extraction, and then a normalization layer is used to speed up the training and convergence of the network, and then a Relu function is activated, and then a 3×3 convolution with the same convolution width is used to extract the output features. Then, it is activated by a normalization layer and a Relu activation function, and finally a 1×1 convolution is used to increase the channel information before output. The ResNet50 network is divided into 4 residual blocks, and the number of residual units used in all blocks is 3, 4, 6, and 3 respectively. The residual unit can obtain more channel information from the input features, so that the information of the previous residual block can flow into the next residual block without hindrance, which improves the information flow and avoids the gradient disappearance and degradation problems caused by the network being too deep.
[0105] The principle of the feature pyramid network module in step 3 is as follows Figure 5 As shown in Figure 1, it includes the top-down, lateral connection and convolution fusion operation process, where Figure 5 The bottom-up operation on the left is the forward propagation process of the ResNet50 network, where S1-S5 represents the feature map output by the last residual unit in each residual block stage. The top-down operation in the middle combined with the horizontal connection is the upsampling process plus feature fusion. The specific upsampling method is the nearest neighbor upsampling method. First, the feature map S5 of the top layer is convolved with 1×1 to obtain the feature map P5. The feature map P5 is upsampled once, and then the corresponding feature map S4 in the previous layer is convolved with 1×1 to generate feature map P4. Repeat the iterative process to generate feature map P3 and feature map P2. The final convolution fusion process is to eliminate the aliasing effect caused by the upsampling process through a 3×3 convolution kernel.
[0106] The principle of the path enhancement module in step 3 is as follows Figure 5 As shown in Figure 1, P2-P5 represent the feature maps generated by the feature pyramid network module. Path enhancement starts from feature map P2 and gradually approaches feature map P5. We use N2-N5 to represent the feature maps after path enhancement, where feature map N2 is exactly the same as feature map P2 without any processing. The specific process of path enhancement is as follows: Figure 6 As shown. Feature map N i After a convolutional layer with a kernel size of 3×3 and a stride of 2, the feature map size is reduced to half of its original size and then combined with the feature map P i+1 The result of the addition is passed through a convolution layer with a kernel size of 3×3 and a step size of 1 to obtain the feature map N. i+1 , the feature map size remains unchanged, and all convolutional layers are followed by a ReLU layer. Where i is 2, 3, or 4.
[0107] The principle of the region candidate network module in step 3 is as follows Figure 7 As shown in the figure, the main function of the region candidate network module is to extract candidate frames from the feature maps N2-N5 finally generated by the previous path enhancement module through a series of operations. The series of operations of the region candidate network module mainly pass the feature maps N2-N5 through a 3×3 convolution respectively, and then pass them through two 1×1 convolutions respectively to obtain classification branches and regression branches, where the classification branch is a binary classification of the target and the background, generating a total of 2K outputs, and the regression branch is to determine the position of the candidate frame of the target area, each box contains 4 values, and a total of 4K outputs are generated; the 6K outputs obtained by the feature maps N2-N5 through the region candidate network module are ROIAligned with the above N2-N5 feature maps to obtain the ROI feature map. The region candidate network module is a fully convolutional network that can simultaneously predict the bounding box coordinates and target scores of the target, and end-to-end training generates high-quality proposal frames. At the same time, the region candidate network module shares convolutional features with the detection network. Among them, each feature map of the present invention corresponds to an anchor box area size, and each anchor box has three aspect ratios, namely 1:1, 1:2, and 2:1, that is, each anchor point of the feature map generates 3 anchor boxes, m=3, K=W×H×m, W and H are the width and height of the feature map, respectively.
[0108] The principle of the boundary perception positioning module in step 3 is as follows Figure 8 As shown in the figure, the main steps are to divide the target area into multiple buckets, first determine which bucket its border is on, and then regress the offset of its border line from the center of the bucket. In this way, high-quality border coordinate information is obtained through accurate regression of the four borders. The specific steps are to compress the channel through 1×1 convolution, strengthen the X-axis direction features and Y-axis features respectively through self-attention, obtain the X-direction feature map and the Y-direction feature map, and then use 1×3 convolution and 3×1 convolution to aggregate the feature F along the X-axis and Y-axis respectively to extract the horizontal feature and vertical feature F. X 、F Y , and then to F X 、F Y Upsample to get features of size 1×2k and 2k×1, and then convert the horizontal feature F Xand vertical feature F Y Segmented into side-aware features F left 、F right 、F top and F down Then, bucket estimation is performed. In order to ensure that the candidate boundary can completely cover the entire object, that is, to completely separate the object from the background, the candidate boundary area needs to be widened by a proportional factor σ (σ>1). For each side of the bounding box, the boundary-aware positioning module first divides the extended proposal target space into multiple buckets, and uses the side perception function to search for the center of the bucket closest to the target boundary. The center line of the bucket is used as a rough estimate to determine the location of the edge of the target, thereby determining the approximate boundary position B of the object. left , B right , B top and B down Finally, the fine-tuned boundary position X is obtained by the offset of the Bucker center line left , X right , X top and X down , thus obtaining a high-precision positioning boundary.
[0109]
[0110]
[0111] l x =(δB right -δB left ) / 2k (3)
[0112] l y =(δB down -δB top ) / 2k (4)
[0113] The formula used to enhance the feature through self-attention is shown in formula (1)(2). Mx and My represent the attention map, which are obtained by convolution of feature F by 1×1. Mx and My are normalized along the X direction and Y direction to obtain the weighted values in the horizontal direction and vertical direction, and then weighted with feature F to obtain the X-direction feature map and Y-direction feature map. The width and height of a single bucket are calculated as shown in formula (3)(4). left , B right , B top and B down They represent the upper, lower, left, and right bucket boundaries after rough positioning. k is an integer, and 7 or 14 is generally the best. xIndicates the height of a single bucket, l y Indicates the height of the bucket.
[0114] The specific implementation method of the process of training the model in step 5 is as follows: Fig. 9 As shown, first set the number of pictures of the extracted batch size to 4 in the train_res50_fpn.py configuration file in the master, the hyperparameter learning rate = 0.001, momentum = 0.9, weight decay coefficient = 0.0005, number of iterations = 1000, and decay the learning rate by 10 times after 720 and 840 iterations of training, and the other parameters are default. Start training, load the deep learning network model, and call in images in batches according to the size of the batch pictures for training. According to the set parameters, when the number of iterations reaches the set number, output the weight once, and then screen out the candidate weights. The performance of the weights is quantitatively evaluated through the validation set, and the evaluation results such as accuracy, recall, average precision mean, etc. are checked for the performance of the overall quantitative optimal weights.
[0115] The specific steps of the comparative experiment in step 7 are as follows:
[0116] Step 7.1: Randomly select a portion of images from the untrained image sequence for prediction under the same other conditions, and qualitatively compare the positioning effects of all algorithms. The specific comparison results are as follows: Fig.10 As shown in the figure, the results from left to right are FasterR-CNN, SSD, CenterNet, NDSSM, Our, and GT (Our represents the method of the present invention, and GT represents the ideal state of manual annotation). Fig.10 It can be seen that all algorithms can accurately locate the target, and the confidence level of the detected target is as high as over 95%. In order to further analyze the differences in positioning accuracy of different algorithms, we will Fig.10 The upper and lower edges of the qualitative comparison results are cut out and enlarged for qualitative comparison, and the inaccurate positioning areas are framed with rectangles, such as Fig.11 As shown in the figure, the results from left to right are FasterR-CNN, SSD, CenterNet, NDSSM, Our, and GT. From the qualitative analysis, it can be seen that in different images, the FasterR-CNN, SSD, CenterNet, and NDSSM algorithms not only have large positioning errors in target positioning, but also the positioning errors of the same algorithm in different images are different, showing a certain volatility. The detection results of the network model described in the present invention are as follows Fig.12 As shown in FIG. 1 , the obtained bounding box almost coincides with the edge of the target, the error is small, and the positioning accuracy under different images maintains good accuracy and stability, that is, the effectiveness of the present invention is verified from the time domain characteristics. The structure of the deep learning network model is as follows Fig.13 shown.
[0117] Step 7.2: Under the same conditions, the performance of the present invention is compared with the displacement signal synchronously collected by the acceleration sensor. The specific steps are as follows:
[0118] Step 7.2.1, process the signal collected in step 1. We use the position information of the target in the first frame as the reference frame for calculating the offset. We calculate the displacement offset between the remaining frames and the first frame to obtain the vibration displacement of the flexible structure in the pixel coordinate system, and then convert the image to the physical coordinate system to obtain the real displacement. The comparison results of the curve fitting between different algorithms and the acceleration displacement at different time nodes are listed in Table 1. The detection algorithm of the present invention, the best FasterR-CNN algorithm among other detection effects, and the acceleration sensor are compared. The time domain waveform of the displacement signal obtained is as follows: Fig.14 As shown, from Fig.14 From the entire displacement signal in (a), it can be seen that the displacement curves obtained by all visual algorithms have approximately the same change trend as the displacement curve obtained by the acceleration sensor. Therefore, after the structure is stimulated, the displacement quickly reaches the peak from zero and then gradually decays regularly. After analyzing the local areas in the figure, we found that the displacement of multiple local areas in FasterR-CNN (5.5s to 8s) exceeds the acceleration displacement.
[0119] Table 1. Comparison of curve fitting between different algorithms and acceleration displacement at different time nodes
[0120]
[0121] Step 7.2.2: The time domain waveform of the displacement signal obtained in step 7.2.1 Fig.14 Further analysis, from Fig.14 In (a), the displacement signal within 1.5 seconds is randomly selected and analyzed. Fig.14 After zooming in on (b), we can find that only the displacement curve fitting of the algorithm of the present invention is closest to the acceleration displacement signal, while the other FasterR-CNN algorithms with the best detection effects show more or less amplitude inconsistency in some time periods. Fig.14 (b) A uniform displacement peak signal is randomly selected and then Fig.14 After partial zooming in (c), it is found that the displacement signal obtained by FasterR-CNN changes with time in a wavy pattern and has poor smoothness. Whether analyzed from the overall or local perspective, the displacement curve obtained by the algorithm of the present invention has the highest fit with the acceleration displacement signal, and it maintains a high degree of matching over time.
[0122] Will Fig.14After the time domain waveform in is transformed by fast Fourier transform, we can get Fig.15 As shown in the frequency domain diagram, it can be seen from the figure that the spectrum characteristics obtained by the method of the present invention are closest to the acceleration signal, while the low-frequency amplitude obtained by the Faster R-CNN algorithm with the best detection effect is close to the acceleration signal, but a large amount of high-frequency noise appears.
[0123] In summary, from the perspective of time-frequency domain characteristics, the algorithms of the present invention all show better measurement results.
[0124] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A bridge vibration displacement visual measurement method based on deep learning, characterized by: include: Using an image acquisition device to collect a bridge structure data set and divide it into a training data set and a verification data set; Using annotation tools, the training data set and the validation data set are annotated to obtain the training set and the validation set respectively; Build deep learning network models; Modify the hyperparameters in the configuration file to obtain the training parameters; Call the training set and configuration file to start training the deep learning network model, and obtain the candidate weights after the training. The performance of the candidate weights is evaluated using a validation set to quantify the performance of the candidate weights, and the optimal weights are loaded into the deep learning network model to obtain the deep learning network model after the optimal weights are loaded; Input the newly acquired bridge structure image to be inspected into the deep learning network model after loading the optimal weights for inspection, and obtain the inspection bounding box; The construction of the deep learning network model specifically comprises: constructing the deep learning network model using a backbone network of ResNet50 feature extraction, a feature pyramid module, a path enhancement module, a region candidate network module, and a side perception boundary positioning module; The backbone network of the ResNet50 feature extraction is divided into 4 residual blocks to obtain feature maps S1, S2, S3, S4, and S5; wherein the number of residual units used by the residual blocks is 3, 4, 6, and 3 respectively; The feature pyramid module obtains feature maps P2, P3, P4, P5 by top-down, lateral connection and convolution fusion operations based on feature maps S2, S3, S4, S5 output by the backbone network; wherein the nearest neighbor upsampling method is used in the top-down upsampling process; The path enhancement module is based on the feature maps P2, P3, P4, and P5 output by the feature pyramid module. The path enhancement starts from the feature map P2 and gradually approaches the feature map P5 to obtain feature maps N2, N3, N4, and N5; wherein the feature map N2 is the same as the feature map P2; The region candidate network module first passes through a 3×3 convolution on the feature maps N2-N5 output by the path enhancement module, and then passes through two 1×1 convolutions to obtain a classification branch and a regression branch; the classification branch is a binary classification of the target and the background, generating a total of 2K outputs, and the regression branch is to determine the candidate box position of the target area, each box contains 4 values, and a total of 4K outputs are generated; the 6K outputs are ROI Aligned with the above N2-N5 feature maps to obtain the ROI feature map; where K=W×H× m , W and H are the width and height of the feature map output by the path enhancement module, m Indicates the number of anchor boxes generated by each anchor point; The side perception boundary positioning module relaxes the target area in the ROI feature map output by the region candidate network module. σ The scale factor of , the relaxed ROI feature map is compressed by 1×1 convolution to obtain feature F, and feature F is strengthened by self-attention in the X-axis direction and Y-axis direction to obtain feature maps in the X direction and Y direction. Then, feature F is aggregated along the X-axis and Y-axis by 1×3 convolution and 3×1 convolution respectively to extract horizontal feature F X and vertical feature F Y , and then to F X 、F Y Upsample to get features of size 1×2k and 2k×1, and then convert the horizontal feature F X and vertical feature F Y Segmented into side-aware features F left 、F right 、F top and F down ; Then perform bucket estimation to obtain the approximate boundary position B left , B right , B top and B down Finally, the fine-tuned boundary position X is obtained by the offset of the Bucker center line left , X right , X top and X down .
2. The bridge vibration displacement visual measurement method based on deep learning according to claim 1 is characterized in that: Also includes: Verification of the deep learning network model after loading the optimal weights: Verify the error between the bounding box and the target edge by comparing the performance with traditional digital image processing algorithms and deep learning algorithms with different positioning principles; perform curve fitting comparison between the displacement curve obtained based on the bounding box and the displacement curve obtained by the displacement signal collected by the acceleration sensor, perform fast Fourier transform on the displacement curve, and compare the frequency domain diagrams obtained by the fast Fourier transform; jointly realize the verification of the time-frequency domain characteristics.
3. The bridge vibration displacement visual measurement method based on deep learning according to claim 2 is characterized in that: The displacement curve obtained according to the bounding box is specifically as follows: the position information of the target in the image of the first frame is used as a reference frame for calculating the displacement offset, the vibration displacement of the bridge structure in the pixel coordinate system is obtained by calculating the displacement offset between the remaining frames and the first frame, and then the vibration displacement in the pixel coordinate system is converted into the physical coordinate system to obtain the real displacement curve.
4. The bridge vibration displacement visual measurement method based on deep learning according to claim 1 is characterized in that: The image acquisition device acquires the bridge structure data set specifically by: using a high-speed camera to capture vibration data of the excited bridge model according to the shooting speed.
5. The bridge vibration displacement visual measurement method based on deep learning according to claim 1 is characterized in that: The labeling is specifically as follows: the training data set and the validation data set are labeled by the labelImg toolkit, and after the labeling is completed, a training set for training and a validation set for validation are obtained; wherein the specific operation of the labeling is to label the detection targets on the bridge in the data set image with a labeling box of the same length and width.
6. The bridge vibration displacement visual measurement method based on deep learning according to claim 1 is characterized by: The training of the deep learning network model is as follows: Train the deep learning network model; the training objects include: the coordinates and categories of the upper left corner and lower right corner of the annotation box; the training results are expressed in the form of weights obtained after training; Call the file of the deep learning network model to start training. The specific process of training is as follows: 1) Load the hyperparameters set in the file into the deep learning network model; 2) Randomly select a batch size of images from the training set as the current training sample; 3) Each image in the training sample is sequentially placed in the deep learning network model to update the weight parameters; the weight parameter update is specifically as follows: the weight parameters are randomly initialized and then the forward propagation calculation of the convolutional neural network is performed to obtain a set of intermediate parameters, and then the intermediate parameters are used for back propagation to update the weight parameters; the new weight parameters will iterate the old weight parameters used to calculate the forward propagation before; 4) Complete the forward and backward propagation process for all images in the training sample as one training of the deep learning network model, and repeat steps 1) to 3) until the number of network model training reaches the set number of iterations; 5) After the entire training is completed, filter out the candidate weight files.
7. A bridge vibration displacement visual measurement system based on deep learning, characterized by: include: A division unit, used for collecting a bridge structure data set by using an image acquisition device and dividing the data set into a training data set and a verification data set; A labeling unit, used to label the training data set and the validation data set by using a labeling tool to obtain a training set and a validation set; Construction unit, used to build deep learning network models; A first obtaining unit is used to modify the hyperparameters in the configuration file to obtain the training parameters; The second obtaining unit is used to call the training set and the configuration file to start training the deep learning network model, and obtain the candidate weights after the training; The third obtaining unit is used to evaluate the performance of the candidate weights using the validation set to quantify the performance of the candidate weights, load the optimal weights into the deep learning network model, and obtain the deep learning network model after the optimal weights are loaded; A fourth obtaining unit is used to input the newly acquired bridge structure image to be detected into the deep learning network model after loading the optimal weight for detection, so as to obtain a detected bounding box; The construction of the deep learning network model specifically comprises: constructing the deep learning network model using a backbone network of ResNet50 feature extraction, a feature pyramid module, a path enhancement module, a region candidate network module, and a side perception boundary positioning module; The backbone network of the ResNet50 feature extraction is divided into 4 residual blocks to obtain feature maps S1, S2, S3, S4, and S5; wherein the number of residual units used in the residual blocks is 3, 4, 6, and 3 respectively; The feature pyramid module obtains feature maps P2, P3, P4, P5 by top-down, lateral connection and convolution fusion operations based on feature maps S2, S3, S4, S5 output by the backbone network; wherein the nearest neighbor upsampling method is used in the top-down upsampling process; The path enhancement module is based on the feature maps P2, P3, P4, and P5 output by the feature pyramid module. The path enhancement starts from the feature map P2 and gradually approaches the feature map P5 to obtain feature maps N2, N3, N4, and N5; wherein the feature map N2 is the same as the feature map P2; The region candidate network module first passes through a 3×3 convolution on the feature maps N2-N5 output by the path enhancement module, and then passes through two 1×1 convolutions to obtain a classification branch and a regression branch; the classification branch is a binary classification of the target and the background, generating a total of 2K outputs, and the regression branch is to determine the candidate box position of the target area, each box contains 4 values, and a total of 4K outputs are generated; the 6K outputs are ROI Aligned with the above N2-N5 feature maps to obtain the ROI feature map; where K=W×H× m , W and H are the width and height of the feature map output by the path enhancement module, m Indicates the number of anchor boxes generated by each anchor point; The side perception boundary positioning module relaxes the target area in the ROI feature map output by the region candidate network module. σ The scale factor of , the relaxed ROI feature map is compressed by 1×1 convolution to obtain feature F, and feature F is strengthened by self-attention in the X-axis direction and Y-axis direction to obtain feature maps in the X direction and Y direction. Then, feature F is aggregated along the X-axis and Y-axis by 1×3 convolution and 3×1 convolution respectively to extract horizontal feature F X and vertical feature F Y , and then to F X 、F Y Upsample to get features of size 1×2k and 2k×1, and then convert the horizontal feature F X and vertical feature F Y Segmented into side-aware features F left 、F right 、F top and F down ; Then perform bucket estimation to obtain the approximate boundary position B left , B right , B top and B down Finally, the fine-tuned boundary position X is obtained by the offset of the Bucker center line left , X right , X top and X down .
8. The bridge vibration displacement visual measurement system based on deep learning according to claim 7 is characterized in that: It also includes a verification unit for verifying the deep learning network model after loading the optimal weights: verifying the error between the bounding box and the target edge by comparing the performance with traditional digital image processing algorithms and deep learning algorithms with various different positioning principles; performing curve fitting comparison between the displacement curve obtained according to the bounding box and the displacement curve obtained by the displacement signal collected by the acceleration sensor, performing fast Fourier transform on the displacement curve, and comparing the frequency domain diagrams obtained by the fast Fourier transform; jointly realizing the verification of the time-frequency domain characteristics.
