Suspension cable vibration recognition system and method based on machine vision

Abstract

The invention discloses a suspension cable vibration recognition system and method based on machine vision, and belongs to the bridge suspension cable vibration monitoring technology. According to the method, monitoring equipment is arranged on a non-suspension cable, and the method comprises the steps that a high-definition camera is arranged at each monitoring point and used for obtaining image data of a bridge, and a video stream in a suspension cable monitoring area is obtained; secondly, preprocessing including denoising and enhancing is carried out on the collected image sequence, and the vibration recognition capacity is improved; then, a key area of a cable is positioned through improved U-Net, and a motion track of the cable in continuous frames is captured in combination with an improved DeepSort algorithm; after the midpoint of the cable is positioned, measuring pixel displacement frame by frame to generate a vibration displacement time sequence, and calculating motion vectors of key points in adjacent frames by using an optical flow method to form vibration waveform data; and finally, according to the motion information of the feature points extracted from the image, analyzing the vibration mode and amplitude of the suspension cable, and using Fourier transform to obtain inherent parameters such as fundamental frequency of the suspension cable. According to the method, the vibration monitoring and recognition effects of the bridge suspension cable are improved by improving the U-Net and DeepSort algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge monitoring, and particularly to a suspension cable vibration recognition system and method based on machine vision. Background Art

[0002] In traditional bridge or suspension cable structure health monitoring, devices such as accelerometers and displacement sensors are often used to detect vibration conditions in real time. However, these devices usually need to be installed on the structure, which may cause interference to the structure, and there are certain limitations in terms of their measurement range, accuracy, and cost:

[0003] 1. Require direct contact with the structure: Accelerometers, strain sensors, and displacement sensors usually need to be installed on the suspension cable or bridge structure to monitor vibration. This not only may have a certain impact on the structure, but also the installation and maintenance are relatively complex, especially in large-scale bridges or high-altitude structures, where the operation difficulty and cost are high.

[0004] 2. Difficult installation: Suspension cable structures often have large spans and complex geometries, making it very difficult to install sensors. Installing sensors in suspended or inaccessible places is prone to safety problems or errors.

[0005] 3. Limited by the number and location of sensors: The number of traditional sensors is usually limited and cannot cover every vibration node of the entire suspension cable. This makes the monitored data may not comprehensively reflect the overall vibration characteristics of the suspension cable.

[0006] 4. High cost: Especially in large-scale monitoring systems, a large number of sensors and wiring are required, increasing the installation and maintenance costs of the system.

[0007] Therefore, researching and developing a non-contact, efficient, and accurate suspension cable vibration recognition technology has important application value. Machine vision, as a non-contact monitoring technology, can obtain the surface image of the structure through a camera and process it to achieve comprehensive monitoring of the structure. The method based on image processing and pattern recognition can identify the vibration characteristics of the suspension cable in real time and accurately, thus providing a new solution for structural health monitoring. Summary of the Invention

[0008] Object of the Invention: Aiming at the limitations of traditional suspension cable vibration recognition in terms of measurement range, accuracy, and cost, the present invention provides a suspension cable vibration recognition system and method based on machine vision, which uses machine vision, utilizes a high-definition camera to measure the vibration video of the suspension cable of a suspension bridge, and combines machine recognition technology to accurately identify the suspension cable vibration.

[0009] Technical Solution: A suspension cable vibration recognition method based on machine vision includes the following steps:

[0010] (1) Deploy monitoring devices, including high-definition cameras to obtain image data of the bridge and get the video stream within the suspension cable monitoring area;

[0011] (2) Perform preprocessing including denoising and enhancement on the collected image sequence to improve the vibration recognition ability;

[0012] (3) Improve the U-Net to locate the key areas of the cable, and combine with the improved DeepSort algorithm to capture the movement trajectory of the cable in consecutive frames;

[0013] (4) After locating the midpoint of the cable, generate the vibration displacement time series by measuring the pixel displacement frame by frame, and use the optical flow method to calculate the motion vectors of the key points in adjacent frames to form the vibration waveform data;

[0014] (5) Analyze the vibration mode and amplitude of the suspension cable according to the motion information of the feature points extracted from the image, and use Fourier transform to obtain the cable natural parameters such as the fundamental frequency of the suspension cable.

[0015] Further, step (1) is to obtain image data by using a high-speed camera for the suspension cable of the bridge.

[0016] Further, the method performs per-pixel classification of the image by improving the U-Net algorithm, including introducing a multi-scale fusion module between the encoder and the decoder, enhancing the model's perception of the target area and edge details through context information at different scales, and introducing cross-layer attention gating to enhance the response of the suspension cable edge features. The specific improvements to the U-Net algorithm include:

[0017] Attention module, calculate the similarity between the encoder and decoder features at the skip connection and generate the attention weight: A = σ(W1F enc +W2F dec +b); where W1 and W2 are trainable weight matrices, b is the bias term, σ is the Sigmoid activation function, F enc is the encoder feature, F dec is the decoder feature;

[0018] The gated output is: F out = A⊙F enc , where ⊙ represents the element-wise product;

[0019] The multi-scale convolution calculation is:

[0020]

[0021] where Conv is the convolution operation, d i is the dilation rate of the dilated convolution, ω i is the weight of each scale feature.

[0022] Furthermore, the DeepSort algorithm in this method processes the motion trajectory of the suspension cable through a harmonic Kalman filter model. The harmonic Kalman filter model includes introducing an acceleration variable into the state matrix in the traditional Kalman filter. At the same time, the harmonic Kalman filter model maps the state vector to the measurement space through the measurement matrix.

[0023] In the harmonic Kalman filter model, it is assumed that the measurement is the position z k , where V k is the measurement noise, then the measurement matrix is H = [1 0 0];

[0024] The harmonic Kalman filter model performs the following processing steps:

[0025] 1) Initialize the state estimate and the covariance matrix P0

[0026] 2) Predict the state and covariance at the current moment through the state transition equation. The expression is as follows:

[0027]

[0028] 3) Update the state estimate and covariance through the measurement. The expression is as follows:

[0029]

[0030] Where: is the predicted state, Q is the process noise matrix, R is the measurement noise reflecting the error of the measurement device, and K k is the Kalman gain, which determines the influence degree of the new measurement value on the update;

[0031] 4) The updated covariance matrix describes the accuracy of the new state estimate. Update the covariance matrix P k as follows:

[0032]

[0033] Where I is the identity matrix.

[0034] A suspension cable vibration recognition system based on machine vision. This system executes the method as described above. This system further includes:

[0035] Data acquisition module: including a high-speed camera, an IMU, and a data processing unit, used to collect continuous frame images and auxiliary motion data of the bridge;

[0036] Data verification module: mutually verifies the data of multiple monitoring points to ensure the accuracy and reliability of the data;

[0037] Key area location module: Locate the key area of the suspension cable based on the improved U-Net network;

[0038] Motion trajectory capture module: Capture the motion trajectory of the suspension cable in consecutive frames based on the improved DEEPSORT algorithm;

[0039] Vibration analysis module: Generate vibration displacement time series, analyze vibration modes and extract inherent parameters.

[0040] Beneficial effects: Compared with the prior art, the technical solution provided by the present invention is contactless identification. Secondly, the present invention adopts the improved U-Net algorithm and the improved DeepSort algorithm to realize the vibration monitoring and identification of the suspension cable of the bridge, improving the monitoring effect and identification accuracy of the vibration of the suspension cable of the bridge. Description of the Drawings

[0041] Figure 1 is the monitoring deployment schematic diagram of the present invention;

[0042] Figure 2 is the structural diagram of the improved U-Net network;

[0043] Figure 3 is the time-history vibration diagram of the suspension cable in the embodiment;

[0044] Figure 4 is the spectrogram of the suspension cable in the embodiment. Detailed Embodiments

[0045] The suspension cable is a component used to transfer the load of the bridge to the bridge tower. It is usually woven from multiple strands of steel wire strands and is fixed to the bridge tower or anchor block through anchors to support the bridge, including the main cable, side cable and inclined cable. Existing monitoring schemes include obtaining vibration images based on cameras, but after obtaining video data, it still has to be identified and judged manually. Some existing technologies can perform certain image recognition, but the error is very large, and it is also impossible to combine key positions (monitoring points) to improve the recognition accuracy of the device.

[0046] The present invention first deploys a high-speed camera independently outside the suspension cable, combined with Figure 1The deployment method shown can be to install a high-speed camera through a mounting bracket near the bridge, and also includes installing the camera on a fixed structure (such as a pier or a bridge tower) near the bridge, adding an anti-shake device (such as a shock absorber) to the bracket to further improve the image stability. Ensure that the suspension cable is within the camera's field of view, and collect an image sequence of the suspension cable surface through this camera. To avoid data errors or interference from its own position of a single camera, the present invention uses multiple synchronous cameras to collect the video stream of the suspension cable. Then, perform preprocessing operations such as denoising and enhancement on the collected image sequence to improve the accuracy of subsequent vibration recognition, and further monitor the vibration of the suspension cable based on the improved U-Net and Deepsort algorithms.

[0047] Combined with Figure 2 As shown, the present invention realizes the key point positioning of the suspension cable by image segmentation and pixel-by-pixel classification of the video stream through the U-Net algorithm. The structure of this algorithm includes the following main parts:

[0048] Encoder (downsampling): The encoder part gradually reduces the spatial resolution of the image through convolution and pooling operations, while extracting deeper features.

[0049] Convolutional layer: Each layer extracts image features through convolution operations.

[0050] ReLU activation function: Use the ReLU activation function to increase the non-linear expression ability of the network.

[0051] Max pooling: Reduce the size of the image (downsampling) through pooling operations to reduce the computational amount and improve the abstraction level.

[0052] The goal of the encoder is to extract global features while maintaining semantic information at different scales.

[0053] Decoder (upsampling): The decoder part gradually restores the spatial resolution of the image through upsampling operations, and combines the low-level features in the encoder with the high-level features in the decoder through skip connections, thereby helping to restore the details of the image.

[0054] Upsampling: Restore the spatial resolution through upsampling.

[0055] Skip connection: Skip connect from each layer of the encoder to the corresponding layer of the decoder, so that the decoder can make full use of the low-level detail information.

[0056] The traditional U-Net algorithm has certain shortcomings in the following aspects:

[0057] Edge details: U-Net is prone to problems such as blurring or discontinuity when dealing with edge details.

[0058] Feature fusion: In traditional skip connections, encoder features are directly passed to the decoder, without fully fusing multi-scale context information.

[0059] Based on this, a multi-scale fusion module (multi-scale convolution) is introduced between the encoder and the decoder in the traditional U-Net network to make full use of context information at different scales, improve the model's perception of the target area and edge details, and at the same time introduce cross-layer attention gating to enhance the response of the cable edge features.

[0060] Attention calculation: Calculate the similarity between the encoder and decoder features at the skip connection and generate attention weights. The specific calculation is as follows:

[0061] A = σ(W1F enc + W2F dec + b)

[0062] Where:

[0063] W1 and W2 are trainable weight matrices, b is the bias term, σ is the Sigmoid activation function, and F enc is the encoder feature, and F dec represents the decoder feature.

[0064] Gating mechanism: Weight the features through a gating unit to suppress irrelevant background noise and enhance the target edge response. Gating output:

[0065] F out = A ⊙ F enc

[0066] ⊙ represents element-wise multiplication

[0067] The expression for multi-scale features in multi-scale convolution is:

[0068]

[0069] where Conv is the convolution operation, d i is the dilation rate of the dilated convolution, and ω i is the weight of each scale feature.

[0070] Core steps of DeepSort:

[0071] 1. Object detection: Use an object detection algorithm to detect objects in each frame. (Improved U-Net algorithm)

[0072] 2. Kalman filtering: Predict the position of the object to help the system maintain tracking even when the object temporarily disappears or is occluded.

[0073] 3. Deep feature embedding: Extract the visual features of the object through a deep convolutional neural network.

[0074] 4. Target Association: Based on the prediction results of the Kalman filter and the depth features, target matching is performed to ensure the unique identification of each target.

[0075] The DeepSort algorithm adopts the traditional Kalman filter, assuming that the target motion model is linear. There are significant errors in predicting the vibration of the suspension bridge cables in the application scenario of this patent. Therefore, the present invention adopts a harmonic Kalman filter model, which conforms to the periodic vibration law of the cables.

[0076] The harmonic Kalman filter introduces an acceleration variable into the state matrix in the traditional Kalman filter. The state matrix is expressed as:

[0077]

[0078] where x k represents displacement, v k represents velocity, and a k represents acceleration.

[0079] X k = F·X k-1 + w k-1

[0080] where w k-1 is the process noise, and F is the state transition matrix

[0081]

[0082] Note: It is assumed that the acceleration remains unchanged

[0083]

[0084] In the measurement update step of the Kalman filter, the measurement value is used to update the state estimate. For the harmonic motion model, the measurement value is the position, and a measurement matrix can be defined to map the state vector to the measurement space.

[0085] Assume that the measurement is the position z k , then the measurement model is:

[0086]

[0087] where V k is the measurement noise, and the measurement matrix is

[0088] H = [1 0 0]

[0089] Furthermore, the Kalman filter steps are as follows:

[0090] 1) Initialization: Initialize the state estimate and the covariance matrix P0.

[0091] 2) Prediction step: Predict the state and covariance at the current moment through the state transition equation.

[0092]

[0093] 3) Update: Update the state estimate and covariance through measurement.

[0094]

[0095]

[0096] Where: is the predicted state, Q is the process noise matrix, R is the measurement noise reflecting the error of the measurement device, and K k is the Kalman gain, which determines the influence degree of the new measurement value on the update

[0097] 4) Update the covariance matrix P k : The updated covariance matrix describes the accuracy of the new state estimate:

[0098]

[0099] I is the identity matrix.

[0100] Through simulation, the present invention uses an image processing algorithm (such as an improved U-Net network) to perform semantic segmentation on the suspension cable and locate the key points of the suspension cable (such as the midpoint or fixed point of the cable). Furthermore, the recognition accuracy of the key points is enhanced through a feature extraction algorithm (such as SIFT, ORB). The improved DEEPSORT is used to capture the motion trajectory of the key points of the suspension cable in consecutive frames. By measuring the pixel displacement of the key points frame by frame, a vibration displacement time series is generated. Then, the pixel displacement is converted into the actual physical displacement. Finally, by filtering the vibration displacement time series again to remove noise, the time-domain signal is converted into a frequency-domain signal using Fourier transform, the natural parameters such as the fundamental frequency of the suspension cable are extracted, the vibration waveform is analyzed, the vibration mode and amplitude of the suspension cable are identified, and the results as shown in Figure 3 and Figure 4 are obtained.

[0101] Through the above solution, a high-speed camera can be efficiently set on the bridge suspension cable to realize real-time monitoring and high-precision analysis of the suspension cable vibration.

Claims

1. A method for identifying the vibration of a suspension cable based on machine vision, characterized in that, It includes the following steps: (1) Deploy monitoring devices, including high-definition cameras to obtain image data of the bridge, and obtain the video stream within the suspension cable monitoring area; (2) Perform preprocessing on the image sequence, including denoising and enhancement, to improve the vibration recognition ability; (3) Locate the key area of the cable through the improved U-Net, and capture the movement trajectory of the cable in consecutive frames by combining the improved DeepSort algorithm; (4) After locating the midpoint of the cable, generate the vibration displacement time series by measuring the pixel displacement frame by frame, and calculate the motion vectors of the key points in adjacent frames using the optical flow method to form the vibration waveform data; (5) Analyze the vibration mode and amplitude of the suspension cable according to the motion information of the feature points extracted from the image, and use Fourier transform to obtain the natural parameters of the suspension cable, including the fundamental frequency of the suspension cable.

2. The method for identifying the vibration of a suspension cable based on machine vision according to claim 1, characterized in that This method performs pixel-by-pixel classification of images through the improved U-Net algorithm, including introducing a multi-scale fusion module between the encoder and the decoder, and enhancing the model's perception of the target area and edge details through context information at different scales. At the same time, cross-layer attention gating is introduced to enhance the edge feature response of the suspension cable.

3. The method for identifying the vibration of a suspension cable based on machine vision according to claim 2, characterized in that, The specific improvements to the U-Net algorithm include: Attention module, which calculates the similarity between the encoder and decoder features at the skip connection and generates attention weights: A = σ(W1F enc + W2F dec + b); where W1 and W2 are trainable weight matrices, b is the bias term, σ is the Sigmoid activation function, F enc is the encoder feature, and F dec is the decoder feature; The gated output is: F out = A ⊙ F enc , where ⊙ represents the element-wise product; The multi-scale convolution calculation is as follows: where Conv is the convolution operation, d i is the dilation rate of the dilated convolution, and ω i is the weight of each scale feature.

4. The method for identifying the vibration of a suspension cable based on machine vision according to claim 1, characterized in that The DeepSort algorithm in this method processes the movement trajectory of the suspension cable through a harmonic Kalman filter model. The harmonic Kalman filter model includes introducing an acceleration variable into the state matrix in the traditional Kalman filter. At the same time, the harmonic Kalman filter model maps the state vector to the measurement space through the measurement matrix.

5. The method for identifying the vibration of a suspension cable based on machine vision according to claim 4, wherein, In the harmonic Kalman filter model, it is assumed that the measurement is the position z k , where V< If \(k\) is the measurement noise, then the measurement matrix is \(H = [1 0 0]\); The harmonic Kalman filter model performs the following processing steps: 1) Initialize the state estimation and the covariance matrix P0 2) Predict the state and covariance at the current moment through the state transition equation, and the expression is as follows: 3) Update the state estimate and covariance through measurement, and the expression is as follows: Wherein: is the prediction state, Q is the process noise matrix, R is the measurement noise reflecting the error of the measuring device, and K k is the Kalman gain, which determines the influence degree of the new measurement value on the update; 4) The updated covariance matrix describes the accuracy of the new state estimate, and the updated covariance matrix P k is as follows: where \(I\) is the identity matrix.

6. A suspension cable vibration recognition system based on machine vision, characterized in that, This system executes the method described in any one of claims 1-5. This system includes: Data acquisition module: including high-speed cameras, IMUs, and data processing units, used to acquire consecutive frame images and auxiliary motion data of the bridge; Data verification module: mutually verify the data of multiple monitoring points to improve accuracy and reliability; Key area positioning module: locate the key area of the suspension cable based on the improved U-Net network; Motion trajectory capture module: capture the motion trajectory of the suspension cable in consecutive frames based on the improved DEEPSORT algorithm; Vibration analysis module: generate the vibration displacement time series, analyze the vibration mode, and extract the natural parameters. The natural parameters include fundamental frequency data.

Images