A method for detecting vibration of a flexible body bridge based on multi-target visual tracking

Abstract

The application discloses a flexible body bridge vibration detection model for multi-target visual tracking and application. The flexible body bridge vibration detection model for multi-target visual tracking constructed in the application combines a target tracking module on the basis of an original YOLOv5-s framework, fuses time domain information and space domain information between frames, and realizes more effective vibration displacement detection of a detection target. In view of the fact that an object to be detected and a camera exist in a tilt and cause inaccurate regression displacement precision, an angle parameter is added to a regression head part, CIoU is added on the basis of KFIoU, and through experimental verification, RIoU used in the application has better rotation frame regression characteristics and faster convergence speed than KFIoU.

Description

Technical Field

[0001] This invention relates to a multi-target visual tracking model for vibration detection of flexible bridges and its application, belonging to the fields of visual vibration displacement measurement and computer vision. Background Technology

[0002] In real-world scenarios, long-span bridges experience significant vibrations when subjected to external forces. Traditional methods for measuring and detecting the vibration of long-span flexible bridges often employ contact sensors such as accelerometers and strain gauges. However, the installation and maintenance of these sensors and data acquisition systems are cumbersome and expensive, limiting the widespread applicability of traditional contact sensors in practical engineering. Therefore, the research community has been actively exploring more universally applicable technologies, such as IoT-based wireless sensing systems, laser Doppler vibration meters based on laser ranging, GPS, and interferometric radar systems. However, these technologies all have specific installation distance requirements, limited vibration amplitude measurement ranges, and are mostly very expensive, hindering their application in practical industrial structural displacement monitoring.

[0003] Deep learning-based visual measurement methods offer new insights into structural vibration measurement. However, most current deep learning algorithms prioritize target recognition accuracy while neglecting localization accuracy and the possibility of target rotation. Furthermore, the tilt of the detected object or camera, and the rotation of long-span bridges after disturbances, further contribute to low bounding box accuracy. Additionally, because target detection algorithms only consider the target in the current frame, ignoring the spatiotemporal correlation between adjacent frames, significant jitter errors occur when extracting vibration displacement signals. Summary of the Invention

[0004] This invention provides a multi-target visual tracking model and application for vibration detection of flexible bridges. By combining a feature extraction module, a PANet module, a head localization module, and a target tracking module, a novel multi-target visual tracking model for vibration detection of flexible bridges is constructed. Furthermore, based on this model, multi-target vibration detection of flexible bridges is achieved in a non-contact manner.

[0005] The technical solution of this invention is:

[0006] According to one aspect of the present invention, a vibration detection model for flexible bridges based on multi-target visual tracking is provided, which is constructed by using a feature extraction module, a PANet module, a head localization module, and a target tracking module.

[0007] The feature extraction module is based on the backbone network of the YOLOv5-s network model. The last C3 module of the YOLOv5-s backbone network is replaced with a transformer self-attention mechanism module and moved to the layer below the SPPF module, thus constructing a feature extraction module based on the C3 module. i The feature extraction module consists of the BS module, C3 module, SPPF module, and transformer self-attention mechanism module.

[0008] The PANet module will use the three output features of the feature extraction module. Figure X 1. X2 and X3 are used as inputs, from features Figure X Starting with 3, characteristics Figure X 3. The feature map S3 is obtained through the C1BS module. After upsampling the feature map S3, it is compared with the feature map S3. Figure X 2. Perform concat stacking, then use the C3 module to extract features from the stacked feature layers to obtain feature map S2; then upsample feature map S2 and compare it with the feature maps. Figure X 1. Perform concat stacking, then use the C3 module to extract features from the stacked feature layers to obtain feature map S1; leave feature map S1 unprocessed to obtain feature map Q1; perform downsampling on feature map Q1 using the C3BS module, then concat stack it with feature map S2, and then extract features from the stacked feature layers using the transformer self-attention mechanism module to obtain feature map Q2; perform downsampling on feature map Q2 using the C3BS module, then concat stack it with feature map S3, and then extract features from the stacked feature layers using the transformer self-attention mechanism module to obtain feature map Q3.

[0009] The head localization module first passes the feature maps Q1, Q2, and Q3 output by the PANet module through a C1BS module to obtain the classification and regression parts; then, each of these two parts is passed through a C3BS module, with the regression part obtaining a regression branch and a background branch, and the classification part obtaining a classification branch.

[0010] The multi-target visual tracking model for vibration detection of flexible bridges employs a rotation frame overlap loss function L. RIoU L RIoU expression:

[0011] L RIoU =L KFIoU +L CIoU

[0012] In the formula: L KFIoU L CIoU Let Ki represent the KiOi loss function and CIiOi loss function, respectively.

[0013] The target tracking module takes the output of the head localization module as its input, uses Kalman filtering to predict the position of the tracking trajectory in the next frame, uses the IoU between the predicted box and the actual detection box as the similarity between the two matches, and then completes the matching through the Hungarian algorithm.

[0014] Based on the confidence level and threshold of the test boxes output by the head localization module, the test boxes are classified into high-scoring boxes and low-scoring boxes, and processed separately: If it is a high-scoring box, Kalman filtering is used to predict the position and size of the bounding box in the next frame, the IoU (Intersection over Union) value between the bounding box in the next frame and the high-scoring bounding box in the current frame is calculated, and then the Hungarian algorithm is used to complete the matching; if it is a low-scoring box, Kalman filtering is used to predict the position and size of the bounding box in the next frame, the IoU value between the bounding box in the next frame and the low-scoring bounding box in the current frame is calculated, and then the Hungarian algorithm is used to complete the matching.

[0015] According to another aspect of the present invention, the above-described multi-target visual tracking model for vibration detection of flexible bridges is used for multi-target detection in the vibration of flexible bridges.

[0016] The beneficial effects of this invention are as follows: This invention constructs a multi-target visual tracking model for the vibration detection of flexible bridges. Based on the original YOLOv5-s framework, it combines a target tracking module to fuse temporal and spatial information between frames, achieving more effective vibration displacement detection. To address the issue of inaccurate displacement regression due to tilting of the detected object and camera, this invention adds an angle parameter to the regression head, incorporating CIoU on top of KFIoU. Experimental verification shows that the RIoU used in this paper has better frame rotation regression characteristics and faster convergence speed compared to KFIoU. Attached Figure Description

[0017] Figure 1 This is a structural block diagram of the present invention;

[0018] Figure 2 Collect images of the bridge structure;

[0019] Figure 3 The labelImg2 tool is used to annotate the bridge detection targets.

[0020] Figure 4 Here is a structural diagram of the feature extraction module;

[0021] Figure 5 Here is the structure diagram of module C3;

[0022] Figure 6 This is the schematic diagram of the SPPF module;

[0023] Figure 7 This is a schematic diagram of the transformer self-attention mechanism module.

[0024] Figure 8 Here is a flowchart illustrating the specific workflow of the PANet module;

[0025] Figure 9 This is a schematic diagram of the head positioning module.

[0026] Figure 10 This is a comparison chart of the loss curves of the RIoU loss function and the KFIoU loss function;

[0027] Figure 11 Here is a structural diagram of the target tracking module;

[0028] Figure 12 A diagram illustrating the specific steps involved in training a vibration detection model for a flexible bridge using multi-target visual tracking.

[0029] Figure 13 This is a time-domain comparison diagram of the accelerometer sensor readings and the image displacement signal.

[0030] Figure 14 This is a time-domain local comparison diagram of the accelerometer sensor readings and the image displacement signal.

[0031] Figure 15 A time-domain detail comparison diagram of the accelerometer sensor readings and the image displacement signal;

[0032] Figure 16 Fast Fourier Transform for Sensor Figure 1 ;

[0033] Figure 17 Fast Fourier Transform for sensor1 Figure 2 . Detailed Implementation

[0034] The invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited to the description.

[0035] Example 1: As Figure 1-17 As shown, according to one aspect of the present invention, a vibration detection model for flexible bridges based on multi-target visual tracking is provided. The model is constructed by using a feature extraction module, a PANet module, a head localization module, and a target tracking module.

[0036] Furthermore, such as Figure 4As shown, the feature extraction module is based on the backbone network of the YOLOv5-s network model. The last C3 module of the YOLOv5-s backbone network is replaced with a transformer self-attention mechanism module and moved to the layer below the SPPF module, thus constructing a feature extraction module based on the C3 module. i The feature extraction module consists of the BS module, the C3 module, the SPPF module, and the transformer self-attention mechanism module; the C... i The BS module consists of an i×i convolutional layer, a BN layer, and a SiLU activation function. The C3 module is as follows: Figure 5 As shown, the C3 module is divided into a backbone and a residual part. In the backbone, from input to output, there are a C1BS module and n Bottleneck modules. The residual part consists of a single C1BS module. Images are concatenated and stacked with images from the backbone and the residual part, and then passed through the C1BS module again. This residual network increases network depth, thereby improving the accuracy of bridge structure vibration detection. Simultaneously, skip connections are used during feature extraction to mitigate the gradient vanishing problem caused by increasing network depth.

[0037] Specifically, in the embodiments of the present invention, the C3 module includes 3 C1BS modules and n Bottleneck modules. There are a total of 3 C3 modules in the backbone network, where the number of Bottleneck modules in the 3 C3 modules is 1, 2, and 3 respectively (i.e., from input to output, the first C3 module has 1 Bottleneck module, the second C3 module has 2 Bottleneck modules, and the third C3 module has 3 Bottleneck modules). Each Bottleneck module consists of 1 C1BS module and 1 C3BS module.

[0038] The SPPF module is as follows Figure 6 As shown, the SPPF module consists of two C1BS modules and three max-pooling layers. The SPPF structure is used before the last C3 module to extract features from the bridge detection target through max pooling, thereby increasing the receptive field of the feature extraction backbone network. This aims to fully extract the bridge model features and facilitate subsequent operations.

[0039] The schematic diagram of the transformer self-attention mechanism module is as follows: Figure 7 As shown. First, a fully connected layer is used to calculate the matrix of query(Q), key(K), and value(V):

[0040] Q=Linear(x),K=Linear(x),V=Linear(x) (1)

[0041] Then, it is processed through a multi-head attention mechanism. The calculation formula for Multi-HeadAttention is:

[0042]

[0043] Where d is the dimension of query / key, with a default value of 8. The feature map obtained by passing it through the multi-head self-attention mechanism is added to the input x, and then fed forward and added to its residual edge to obtain the final feature map.

[0044] Furthermore, such as Figure 8 As shown, the PANet module will extract the three output features from the feature extraction module. Figure X 1. X2 and X3 are used as inputs, from features Figure X Starting with 3, characteristics Figure X 3. The feature map S3 is obtained through the C1BS module. After upsampling the feature map S3, it is compared with the feature map S3. Figure X 2. Perform concat stacking, then use the C3 module to extract features from the stacked feature layers to obtain feature map S2; then upsample feature map S2 and compare it with the feature maps. Figure X 1. Concat stacking is performed, and then the C3 module is used to extract features from the stacked feature layers to obtain feature map S1. Feature map S1 is left unprocessed to obtain feature map Q1. Feature map Q1 is downsampled once by the C3BS module, and then concat stacked with feature map S2. The stacked feature layers are then extracted using the transformer self-attention mechanism module to obtain feature map Q2. Feature map Q2 is downsampled once by the C3BS module, and then concat stacked with feature map S3. The stacked feature layers are then extracted using the transformer self-attention mechanism module to obtain feature map Q3. This module effectively enhances the backbone network's ability to represent shallow positional information and deep semantic information of rotating targets, improving the robustness of target detection and the compactness of localization anchor boxes.

[0045] Furthermore, such as Figure 9 As shown, the head localization module first passes the feature maps Q1, Q2, and Q3 output by the PANet module through a C1BS module to obtain the classification and regression parts; then, these two parts are passed through a C3BS module respectively, and the regression part obtains the regression branch Obj and the background branch Reg, and the classification part obtains the classification branch Cls.

[0046] Furthermore, the multi-target visual tracking flexible bridge vibration detection model adopts the rotation frame overlap loss function L. RIoU L RIoU expression:

[0047] L RIoU =L KFIoU +L CIoU (3)

[0048] In the formula: L KFIoU L CIoU Let Ki represent the KiOi loss function and CIiOi loss function, respectively.

[0049] It should be noted that this paper takes into account that the traditional IoU loss function cannot regress the angle parameters of the rotated box, and that the currently effective angle regression loss function KFIoU has a slow convergence speed when the predicted box and the ground truth box do not overlap. Therefore, this paper adds CIoU to KFIoU to construct the RIoU loss function (rotated box overlap loss). The calculation method of the RIoU loss function is shown in formula (3). In order to quantitatively evaluate the training effect and convergence speed of the loss function model, we compare the loss value change curves obtained by the RIoU algorithm proposed in this invention and KFIoU after 500 epochs of training. Figure 10 It can be seen that the RIoU used in this invention has better convergence than KFIoU.

[0050] As can be seen from formula (3), the RIoU in this paper is divided into two parts: KFIoU and CIoU, which are described in detail below:

[0051] The main principle of KFIoU is to convert the rotated frame [x,y,w,h,θ] into a two-dimensional Gaussian distribution (μ,Σ), then multiply the two Gaussian distributions to obtain the Gaussian distribution of the intersecting region, and finally convert the three Gaussian distributions back into a rotated rectangle, calculate the approximate SkewIoU (skewed rectangle IoU), and calculate the loss between the two rotated frames. The Gaussian distribution conversion formulas are shown in formulas (4) and (5) below:

[0052] μ=(x,y) T (4)

[0053]

[0054] The specific expressions for the KFIoU loss function are shown in formulas (6) and (7) below:

[0055]

[0056] L KFIoU =1-KFIOU (7)

[0057] in, x, y, w, h, and θ represent the x-coordinate, y-coordinate, width, height, and angle of the center point of the rectangle, respectively.

[0058] The CIoU loss function is mainly divided into three parts: distance loss, aspect ratio loss, and IoU loss. The specific calculation formulas are shown in (8), (9), and (10).

[0059]

[0060]

[0061]

[0062] Among them, b, b gt These represent the center points of the predicted and ground truth boxes, respectively. ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the smallest closure region that simultaneously contains both the predicted and ground truth boxes, v measures the aspect ratio similarity, α is the weight parameter, and w... gt and h gt The width and height of the ground truth bounding box are represented by w and h, respectively.

[0063] Furthermore, such as Figure 11 As shown, the target tracking module uses the output of the head localization module as the input of the target tracking module, uses Kalman filtering to predict the position of the tracking trajectory in the next frame, uses the IoU between the predicted box and the actual detection box as the similarity between the two matches, and then completes the matching through the Hungarian algorithm.

[0064] Furthermore, the target tracking module uses the ByteTrack multi-target tracking algorithm to avoid the independent phenomenon between adjacent frames in the detection algorithm, optimizing the correlation of detected targets between frames and more accurately measuring the displacement of the vibrating target. Specifically, based on the confidence level and threshold of the test box output by the head positioning module, the test boxes are divided into high-scoring boxes and low-scoring boxes, and processed separately: if it is a high-scoring box, Kalman filtering is used to predict the position and size of the bounding box in the next frame, the IoU (Intersection over Union) value between the bounding box in the next frame and the high-scoring bounding box in the current frame is calculated, and then the Hungarian algorithm is used to complete the matching; if it is a low-scoring box, Kalman filtering is used to predict the position and size of the bounding box in the next frame, the IoU value between the bounding box in the next frame and the low-scoring bounding box in the current frame is calculated, and then the Hungarian algorithm is used to complete the matching. In the embodiment of the present invention, the threshold is 0.5, and test boxes with a value greater than the threshold are judged as high-scoring boxes, otherwise they are low-scoring boxes.

[0065] According to another aspect of the present invention, the above-described multi-target visual tracking model for flexible bridge vibration detection is used for multi-target detection in the vibration of flexible bridges.

[0066] Furthermore, the following is an optional implementation process for using the above-mentioned multi-target visual tracking model for flexible bridge vibration detection in flexible bridge vibration:

[0067] Step 1: Use a high-speed camera to collect a dataset of flexible bridge structures and divide it into a training dataset and a validation dataset;

[0068] Specifically, the flexible bridge structure dataset is obtained by capturing vibration data of a bridge model under excitation using a high-speed camera. A supplementary lighting was used for light compensation during the acquisition process. The continuous frame resolution was 640×512, and the acquired images are shown below. Figure 2 As shown. Vibration data of the excited bridge model were simultaneously acquired using an NI9234 acquisition card and an accelerometer at a sampling frequency of 25.6 kHz for comparative experiments. The flexible bridge structure dataset was divided into a training dataset and a validation dataset. In this embodiment, the flexible bridge structure dataset used in this invention was acquired from a cable-stayed bridge model. A total of 20,000 images of the cable-stayed bridge model were acquired using a high-speed camera at a shooting speed of 2000 frames per second. 380 vibration data images of the cable-stayed bridge model were selected sequentially from the acquired data. The training dataset and the validation dataset accounted for 90% and 10% of the bridge structure dataset, respectively: 342 images for the training dataset and 38 images for the validation dataset. Further testing can be conducted by adding or acquiring images of the bridge structure to be tested.

[0069] Step 2: Using LabelImg2, label the training and validation datasets respectively to obtain the training and validation sets. Specifically: set the labeling classes as sensor and sensor1 in the LabelImg2 toolkit, and use labelImg2 to label the training and validation sets. After labeling, the training set for training and the validation set for validation are obtained. The specific labeling operation involves using bounding boxes of the same size to label the detection targets on the bridge. The bounding boxes are rotated, and the labelImg2 toolkit is used for labeling. The labeling effect using the labelImg2 toolkit is as follows. Figure 3 As shown. Since the shape and size of the tilted target remain unchanged in the image sequence, this paper uses a rotating bounding box of uniform size for manual annotation to reduce the error caused by manual annotation, and distributes the annotated dataset into training, validation and test sets in a 9:1 ratio.

[0070] Step 3: Construct a multi-target visual tracking model, which includes a feature extraction module, a PANet module, a head localization module, and a target tracking module, and apply it to the vibration detection of flexible bridges;

[0071] Step 4: Obtain a set of parameters for training by modifying the hyperparameters in the file;

[0072] Step 5: Train the multi-target visual tracking flexible bridge vibration detection model by calling the training set to obtain candidate weights. Specifically, the training objects in Step 5 include the coordinates of the center point of the annotation box, the width and height of the annotation box, the tilt angle of the annotation box, and the category of the annotated object. After training, the training weights are obtained and used for model parameter adjustment and prediction. The specific steps for training the multi-target visual tracking flexible bridge vibration detection model are as follows: Figure 12 As shown, the training process is as follows: The trained hyperparameters are applied to the flexible bridge vibration detection model for multi-object visual tracking, and a portion of the images from the training set are used as the current training samples; the samples are sequentially fed into the flexible bridge vibration detection model for multi-object visual tracking for training, resulting in updated training weight parameters; the specific steps for updating the weight parameters are: randomly initializing the weight parameters and calculating the output of the training samples; comparing the output of the training samples with the ground truth bounding boxes and calculating the loss function; using the chain rule to calculate the gradient of the loss function with respect to the weight parameters; updating the weight parameters based on the gradient value and the learning rate to minimize the loss function. The above steps are repeated until the required number of iterations for network model training is completed. In this embodiment of the invention, the configuration file is first set to extract a batch of 4 images, with hyperparameters learning rate = 0.0032, momentum = 0.843, weight decay coefficient = 0.001, and other parameters set to default. The flexible bridge vibration detection model for multi-object visual tracking is loaded, and training images are loaded in batches for training. Based on the set parameters such as learning rate, momentum, and attenuation coefficient, the training weights are obtained after the multi-target visual tracking flexible bridge vibration detection model has undergone a set number of iterations. The performance of the candidate weights is quantitatively evaluated using a validation set, and the optimal weight is selected based on the overall performance of the quantified optimal weight, including accuracy, recall, and mean precision.

[0073] Step 6: Load the optimal weights into the vibration detection model of the flexible bridge using multi-target visual tracking to obtain the vibration detection model of the flexible bridge using multi-target visual tracking after loading the optimal weights.

[0074] Step 7: A series of comparative experiments were conducted on the multi-target visual tracking vibration detection model for flexible bridges after loading the optimal weights. Target detection and performance were compared with traditional detection networks such as the normalized correlation coefficient matching method. The positional offset of the bounding box center point of this invention was regressed, and the displacement trajectory was compared with the displacement trajectory synchronously acquired by the accelerometer. This invention improves the algorithm, and the performance of the algorithm before and after the improvement is compared using different evaluation methods.

[0075] This invention compares the improvements made to the algorithm through various evaluation methods and verifies the effectiveness of the improvements through detailed comparative experiments. Detailed experimental data are shown in Table 1. This invention uses the normalized root mean square error (NRMSE) as a quantitative evaluation index to measure the degree of fit between the displacement curve obtained by the improved algorithm and the accelerometer sensor. The specific formula (11) is shown. As can be seen from the data in Table 1, all innovations in this invention are effective. After each innovation was added, the mNRMSE decreased from the initial 0.01685 to 0.0137, indicating that the improvements made in this invention can effectively improve the accuracy of the rotating frame visual tracking algorithm in bridge displacement vibration measurement.

[0076]

[0077] Among them, P i Q represents the predicted value of a vibration detection model for flexible bridges based on multi-target visual tracking. i Q represents the true value measured by the accelerometer. max and Q min These are the maximum and minimum values ​​of the actual values ​​measured by the accelerometer, respectively.

[0078] Table 1. Comparison Experiments of Network Models

[0079]

[0080] In Table 1: A is the traditional YOLOv5-s model; B is the traditional YOLOv5-s model with its head module using the head localization module of this invention; C is the model B with its loss function using the RIOU function of this invention; D is the model C with the backbone network of the traditional model using the feature extraction module of this invention, that is, introducing a transformer, and also introducing a transformer in the neck module; E is the model D with the introduction of a target tracking module.

[0081] This invention employs various deep learning algorithms with different positioning principles for performance comparison. Specifically, it compares the displacement curve of the bounding box center point obtained according to this invention with the displacement curves of the bounding box center point detected by different deep learning algorithms, as well as the displacement curve obtained from the displacement signal collected by the accelerometer, through curve fitting. Figure 13 , 14 As shown in Figure 15. The different shapes of the lines in the figure represent YOLOv5, R... 3 The displacement curves of the det, Our algorithm, and accelerometer are presented. Because the algorithm of this invention uses the RIoU rotating frame detection algorithm, it has better rotating frame regression characteristics and faster convergence speed compared to traditional detection algorithms. Since the detected object rotates, the vibration signal measured by the algorithm of this invention has a higher degree of fit with the vibration signal collected by the accelerometer.

[0082] The quantitative comparison results of different algorithms are shown in Table 2. mAP@.5 represents the average AP of the network when IOU is 0.5. The data in Table 2 show that the algorithm presented in this paper has a high detection accuracy.

[0083] Table 2. Quantitative comparison among different visual displacement detection algorithms

[0084] algorithm mNRMSE mAP@.5 <![CDATA[R 3 it]]> 0.0367 1.0 YOLOv5 0.01685 0.995 Ours 0.0137 1.0

[0085] This invention compares R 3 The displacement curves measured by det, YOLOv5, the algorithm of this invention, and the accelerometer are displayed on the time-domain waveform. It is found that their displacement curves have approximately the same sampling frequency and the same decay law, but the vibration amplitudes are significantly different. Further processing of the time-domain waveforms using a Fast Fourier Transform, as shown... Figure 16 , 17 As shown, a comparison of the spectral characteristics of different algorithms from the perspective of frequency domain consistency reveals that the spectral characteristics obtained by the method of this invention are closest to the acceleration spectrum signal. Other algorithms maintain approximately local consistency with the acceleration spectrum signal, which reflects the effectiveness of the algorithm of this invention in visual displacement measurement.

[0086] In summary, this invention addresses the challenges of small target size, low image resolution, and camera tilt by proposing the addition of angle parameters and utilizing the RIOU rotating box detection algorithm to improve the inaccurate localization of rotating objects by traditional horizontal boxes. Furthermore, this invention introduces the ByteTrack target detection algorithm, fusing temporal and spatial domain information to achieve more effective vibration displacement detection. The experimental section evaluates and compares this invention's method with current mainstream deep learning algorithms and offset measurements obtained from accelerometers; the algorithm demonstrates superior measurement results in all cases.

[0087] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for vibration detection model of flexible bridge based on multi-target visual tracking, characterized in that, A vibration detection model for flexible bridges with multi-target visual tracking is constructed by using a feature extraction module, a PANet module, a head localization module, and a target tracking module. The vibration detection model for flexible bridges using multi-target visual tracking employs a rotation frame overlap loss function. , expression: ； In the formula: Let KFIoU and CIoU represent the loss functions, respectively. The principle of the KFIoU loss function is to transform the rotated box into a two-dimensional Gaussian distribution. The specific expression is as follows: ； ； ； ； in, x, y, w, h, and θ represent the x-coordinate, y-coordinate, width, height, and angle of the center point of the rectangle, respectively. The CIoU loss function is mainly divided into three parts: distance loss, aspect ratio loss, and IoU loss, as shown below: ； ； ； Among them, b, b gt These represent the center points of the predicted bounding box and the ground truth bounding box, respectively. This represents the Euclidean distance between two center points, c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth boxes, v measures the similarity in aspect ratio, α is the weight parameter, and w gt and h gt w and h represent the width and height of the ground truth bounding box, respectively; The feature extraction module is based on the backbone network of the YOLOv5-s network model. The last C3 module of the YOLOv5-s backbone network is replaced with a transformer self-attention mechanism module and moved to the layer below the SPPF module, thus constructing a feature extraction module based on the C3 module. i The feature extraction module consists of the BS module, C3 module, SPPF module, and transformer self-attention mechanism module; The PANet module takes the three output feature maps X1, X2, and X3 from the feature extraction module as input. Starting with feature map X3, feature map X3 is processed by the C1BS module to obtain feature map S3. Feature map S3 is upsampled once and then concatenated and stacked with feature map X2. Then, the C3 module is used to extract features from the stacked feature layers to obtain feature map S2. Feature map S2 is upsampled once and then concatenated and stacked with feature map X1. Then, the C3 module is used to extract features from the stacked feature layers to obtain feature map S1. Feature map S1 is left unprocessed to obtain feature map Q1. Feature map Q1 is downsampled once by the C3BS module and then concatenated and stacked with feature map S2. Then, the transformer self-attention mechanism module is used to extract features from the stacked feature layers to obtain feature map Q2. Feature map Q2 is downsampled once by the C3BS module and then concatenated and stacked with feature map S3. Then, the transformer self-attention mechanism module is used to extract features from the stacked feature layers to obtain feature map Q3. The head localization module first passes the feature maps Q1, Q2, and Q3 output by the PANet module through a C1BS module to obtain the classification and regression parts; then, each of these two parts is passed through a C3BS module, and the regression part obtains a regression branch and a background branch, while the classification part obtains a classification branch. The target tracking module takes the output of the head localization module as its input, uses Kalman filtering to predict the position of the tracking trajectory in the next frame, uses the IoU between the predicted box and the actual detection box as the similarity between the two matches, and then completes the matching through the Hungarian algorithm.

2. The method for vibration detection model of flexible bridge based on multi-target visual tracking according to claim 1, characterized in that, Based on the confidence level and threshold of the test boxes output by the head localization module, the test boxes are divided into high-scoring boxes and low-scoring boxes, and processed separately: If it is a high-resolution bounding box, Kalman filtering is used to predict the position and size of the bounding box in the next frame, the IoU (Intersection over Union) value between the bounding box in the next frame and the high-resolution bounding box in the current frame is calculated, and then the matching is completed by the Hungarian algorithm. If it is a low-resolution bounding box, Kalman filtering is used to predict the position and size of the bounding box in the next frame, and the IoU (Intersection over Union) value between the bounding box in the next frame and the low-resolution bounding box in the current frame is calculated; then the matching is completed by the Hungarian algorithm.

Images