A Bridge Deformation Measurement Method Based on Airborne Dual Cameras and Target Tracking

Abstract

The present invention discloses a method for measuring bridge deformation based on airborne dual cameras and target tracking, which divides the bridge deformation measurement into three parts: pre-pasting signboards, collecting bridge videos by an unmanned aerial vehicle (UAV), and data analysis. First, a number of measurement targets are arranged at the measurement points to be measured on the side of the bridge. Then, a UAV equipped with a dual-camera system including a long-focus camera and a wide-angle camera is used to collect bridge videos. Finally, a method based on deep learning multi-object tracking is used to calculate the displacement of the surface targets of the bridge. By measuring the moving targets on the side of the bridge and the fixed targets of the bridge piers respectively with the dual cameras, the shaking of the UAV itself is eliminated, and the absolute displacement at the target of the bridge is obtained. The method of the present invention has the advantages of being fast, convenient, low-cost and non-contact, overcomes the problem that the traditional method of arranging sensors is difficult to be applied to bridge deformation measurement, and has a good prospect of being widely applied to the actual bridge deformation monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of structural safety monitoring, and particularly relates to a bridge deformation measurement method and system based on airborne dual cameras and target tracking. By combining an airborne coaxial dual camera and a deep learning tracking method, dynamic displacement measurement and safety status evaluation of a bridge under working conditions are realized. Background Technique

[0002] The deformation measurement of a bridge under the action of vehicle loads and the environment (wind, temperature) is an important part of bridge safety assessment. Especially in the load test of a bridge, the accurately measured bridge deformation response can not only be compared with the bridge deformation situation analyzed by finite element method, but also be used to calculate the deep-level characteristic parameters of the bridge (such as structural frequency response function, modal flexibility) according to the structural dynamics theory, providing a basis for bridge damage identification. The displacement measurement of a bridge is divided into long-term monitoring and short-term detection. Long-term monitoring generally requires installing fixed measurement sensors on the bridge to measure the deformation of a small number of key parts and give early warnings when the bridge is subjected to extreme loads or accidents. This method is costly and generally only applicable to large and important bridges. Short-term deformation response detection is more commonly used in the load test before the bridge is put into service and the regular safety assessment during the service process of the bridge. The existing bridge deformation measurement methods mostly aim at the short-term dynamic deformation measurement of bridges.

[0003] The existing bridge deformation measurement methods mostly rely on traditional measurement and surveying instruments, including static GPS, total station, liquid level connecting pipe, etc. Generally, the measurement frequency of such methods is low, generally not exceeding several Hertz, so they are only suitable for static measurement. In recent years, with the development of instrument science, some advanced sensors have been gradually applied to bridge detection. High-sensitivity acceleration sensors are the most commonly used sensors in current bridge vibration tests. Some scholars use the acceleration data to calculate the bridge deformation by double integration, but also point out the limitations of this method, which need to be used under the conditions of short-duration intervals and small-amplitude displacements. LVDT sensors are a commonly used displacement sensor. Some studies use it to measure the deformation of local positions of bridges. However, LVDT needs to be installed on a fixed base point near the measurement point, so it cannot measure the absolute displacement of the main body of the bridge such as the mid-span, and the measuring range is limited.

[0004] With the rapid development of computer vision technology, vision-based measurement methods have begun to be applied to structural deformation detection. These methods are inexpensive, have a simple system composition, and have been proven to achieve the required measurement accuracy in the measurement of some civil engineering structures. Research has shown that vision measurement methods have obvious application prospects in bridge deformation measurement, but two inherent defects of this method have also been exposed. (1) Vision measurement methods generally calculate the movement of the points to be measured in the images at two consecutive times based on the matching between images. However, the imaging quality of images is easily affected by light interference. Therefore, traditional image processing-based matching methods are very sensitive to interferences such as light and occlusion. (2) When using vision measurement methods to detect long-span bridges, it is very difficult to find a suitable camera layout position. On the one hand, bridges are generally located above rivers. When arranging cameras, most cameras can only be tilted to photograph the bridges. On the other hand, when measuring the mid-span of the bridge, for long-span bridges, the measurement distance may exceed 600m. At this distance, the influence of atmospheric interference and the micro-vibration of the camera itself on the results is very large. Therefore, corresponding methods are urgently needed to solve the above problems.

[0005] Aiming at the problem that traditional optical measurement methods are easily affected by interferences such as light and occlusion, some scholars believe that the widely studied deep learning methods in recent years can be used to solve this problem. Object detection algorithms based on deep learning generally have good anti-interference capabilities. In disease detection, even if the images are affected by light, occlusion, and stain erosion, the diseases in them can be accurately identified. Therefore, some scholars have tried to integrate deep learning technology into vision measurement methods. These methods effectively use various methods in deep learning to improve the stability of optical measurement, but most of them are not applied to the bridge deformation measurement scenario. Moreover, deep learning methods have obvious sample selectivity. Therefore, further research on these methods in the bridge deformation measurement scenario is needed.

[0006] Aiming at the problem of the long measurement distance that is difficult to overcome in the measurement of long-span bridges by vision measurement methods, the increasingly widely used unmanned aerial vehicle (UAV) technology in recent years is expected to become a breakthrough point for solving this problem. UAVs carrying cameras have played an important role in structural detection. Using UAVs carrying cameras to replace standing cameras to measure bridge deformation will greatly reduce the measurement distance, so that the displacement measurement position can be arbitrarily selected. However, the UAV will inevitably shake during flight, resulting in the problem of unstable measurement reference points, which limits the application of UAV-based measurement methods. Most of the existing research uses the fixed points in the structural background photographed by the UAV as references to calculate the displacement of the UAV itself. However, for long-span bridges, since they are generally located on the water surface, it is generally difficult to find fixed points as references in the picture. Summary of the Invention

[0007] Technical problem to be solved: Aiming at the deficiencies of the above-mentioned existing technologies, the present invention provides a bridge deformation measurement method and system based on airborne dual cameras and target tracking, which can quickly and conveniently measure the bridge deformation amount with high measurement accuracy.

[0008] Technical solution:

[0009] A bridge deformation measurement method based on airborne dual cameras and target tracking, the bridge deformation measurement method includes the following steps:

[0010] S1. Arrange a number of measurement targets at the measured points to be measured on the side of the bridge, and use a drone to carry a dual-camera system including a telephoto camera and a wide-angle camera to collect bridge videos; wherein, the positions of the telephoto camera and the wide-angle camera are coaxial and fixed, and both are facing the bridge for continuous shooting; the field of view of the wide-angle camera includes the left and right piers of the bridge; the telephoto camera is only used to collect the image videos of the measurement targets at the measured points to be measured on the side of the bridge.

[0011] S2. Build a target recognition model based on the YOLO v5s network, import the video frame images collected by the telephoto camera and the wide-angle camera into the target recognition model respectively, use the target recognition model to identify the positions of the measurement targets from the bridge video frame images collected by both, and segment the measurement target images from the video frames.

[0012] S3. Use a feature extraction network to distinguish between different positions of the segmented measurement target images, assign a fixed ID to each measurement target, obtain the center coordinates of the bounding boxes of the measurement targets in each frame, and draw a rough displacement trajectory of the measurement targets.

[0013] S4. Based on the multi-target tracking algorithm of DeepSORT, after obtaining the center coordinates of the bounding boxes of the measurement targets in the current frame, use Kalman filtering to predict the predicted trajectory of the current coordinates, calculate the correlation degree between the predicted trajectory and the actual coordinates of the center of the bounding box in the next frame, and correct the rough displacement trajectory of the measurement targets according to the correlation degree to obtain the actual displacement trajectory of the center points of the measurement targets.

[0014] S5. Identify the center points of the measurement target images at the sub-pixel level to obtain the sub-pixel level displacement trajectories of the center points of the measurement targets corresponding to the images collected by the two cameras.

[0015] S6. Combine the sub-pixel level displacement trajectories corresponding to the two cameras and the positional relationship between the two cameras, and eliminate the displacement errors caused by the drone shaking from the measurement results of the moving targets.

[0016] Further, in step S1, before applying the dual-camera system for measurement, the telephoto camera and the wide-angle camera are independently calibrated using the Zhang's calibration method to obtain the internal parameter K of the wide-angle camera w and the internal parameter K of the telephoto camerat 。

[0017] Furthermore, for the measurement of the relative displacement of the pier position, no measurement target is used, and the measurement is carried out by using the matching relationship of the texture of the pier itself.

[0018] Furthermore, in step S2, the process of constructing the target recognition model based on the YOLO v5s network includes the following steps:

[0019] Paste the target on the bridge surface, measure the position of the target, and use the center of the target as the displacement measurement point;

[0020] Collect a number of target images under different lighting conditions, manually mark the points, make a target data set in the format of PASCAL VOC, and use the k-means clustering method to cluster the target images in the data set;

[0021] Import the target data set into the YOLO v5s network for training and verification to obtain the trained target recognition model.

[0022] Furthermore, in step S4, the process of correcting the rough displacement trajectory of the measurement target according to the correlation degree to obtain the actual displacement trajectory of the center point of the measurement target includes:

[0023] Calculate the correlation degree between the predicted trajectory and the actual coordinates of the center of the bounding box in the next frame. If the correlation degree reaches the predicted correlation degree threshold, it is determined that the detection result in the next frame is correct; otherwise, perform IOU matching on the predicted trajectory and the re-detected measurement target to continue depicting the trajectory.

[0024] Furthermore, the shape of the measurement target includes four rectangular marks at the four corners and a circular mark in the middle.

[0025] Furthermore, in step S5, the process of identifying the center point of the measurement target image at the sub-pixel level to obtain the sub-pixel point-level displacement trajectory of the center points of the measurement targets corresponding to the images collected by the two cameras includes the following steps:

[0026] Take the coordinates of the four rectangular marks at the four corners of the target as known points, calculate the scale parameter of the measurement target image, and perform skew correction on the measurement target image;

[0027] Use the method of graphic detection to screen the coordinates P1(x1, y1), P2(x2, y2), P3(x3, y3), P4(x4, y4) of the four rectangular marks, fit to obtain the prototype of the measurement target, use the Hough transform to detect the center coordinates P5(x5, y5) of the circle, and take the average value of these five coordinates as the center coordinates P of the measurement target image c (x c , y c ).

[0028] Further, if some of the rectangular and circular markers in the measured target image are missing, the average value of the coordinates of the remaining markers is used as the central coordinate of the measured target image.

[0029] Further, in step S6, the pure displacement of the bridge after removing the base point displacement of the UAV platform is:

[0030] ΔP b = k ti ΔY t - R t，w k wi ΔY w

[0031] where R t，w and T t，w are the rotation and translation relationships of the two cameras, where k i is the scale factor of the target at time T1, and ΔY w and ΔY t are the vertical displacements of the target measured by the wide-angle camera and the telephoto camera.

[0032] The present invention also mentions a bridge deformation measurement system based on an airborne dual-camera of a UAV and deep learning tracking. The bridge deformation measurement system includes a plurality of measurement targets, a UAV carrying a dual-camera system including a telephoto camera and a wide-angle camera, and a processor;

[0033] The plurality of measurement targets are distributed at the measuring points to be measured on the side of the bridge; the UAV hovers on the side of the bridge according to the control instruction of the processor, and uses the telephoto camera and the wide-angle camera to collect the bridge video, and sends the collected bridge video to the processor;

[0034] The processor uses the bridge deformation measurement method as described above to calculate the vertical displacement of the measurement target at the measuring point to be measured on the side of the bridge

[0035] Beneficial effects:

[0036] The bridge deformation measurement method based on an airborne dual-camera and target tracking of the present invention, on the one hand, simultaneously shoots a large-angle video including different points such as bridge piers and a refined video aiming at the displacement target on the side of the bridge by the coaxial dual-cameras on the UAV, and removes the base point displacement caused by the shaking of the UAV platform from the bridge displacement. On the other hand, the method based on deep learning object detection and multi-object tracking is used to solve the problem that the traditional displacement calculation method is prone to measurement errors under light changes and accidental occlusions. Compared with the prior art, the present invention has the following advantages:

[0037] (1) Through theoretical derivation and experimental verification, it is proved that the method of using a coaxial dual-camera mounted on a drone to simultaneously photograph the fixed point of the pier and the target on the bridge body can effectively eliminate the vibration of the drone itself when measuring the bridge displacement by using the drone. This method takes into account the requirements of large-field-of-view photography of the bridge and small-field-of-view photography of the local target position on the bridge at the same time. Compared with the existing method of using a single-camera drone to photograph the whole bridge simultaneously, it can maximize the accuracy of displacement measurement.

[0038] (2) The proposed displacement measurement method based on deep learning multi-object tracking integrates the latest object detection algorithm, multi-object tracking algorithm and sub-pixel detection algorithm. On the established bridge displacement target dataset, the test accuracy of object detection is 0.998, and the test accuracy of object feature classification is 0.823. This displacement calculation method that combines deep learning and sub-pixel detection avoids the problem that traditional displacement calculation methods such as DIC and optical flow need to be manually adjusted according to the measurement target, and also overcomes the problem that traditional methods are easily affected by complex conditions such as changes in lighting conditions and occlusion of the measured object. The indoor displacement table test shows that the proposed method can effectively avoid the occurrence of outliers in the displacement curve while maintaining a similar accuracy to the DIC method. Description of the Drawings

[0039] Figure 1 Schematic diagram of the bridge deformation measurement method based on airborne dual-cameras and object tracking according to the embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the camera pose calculation principle and the dual-camera measurement principle, where (a) is the schematic diagram of the camera pose calculation principle, and (b) is the schematic diagram of the dual-camera measurement principle;

[0041] Figure 3 Flowchart of the displacement calculation method based on deep learning multi-object tracking;

[0042] Figure 4 Flowchart of processing the trajectory based on the DeepSORT multi-object tracking algorithm;

[0043] Figure 5 Schematic diagram of the training result of the target recognition model based on the YOLO v5s network;

[0044] Figure 6 Schematic diagram of the training result of the inter-class recognition network based on the DeepSORT multi-object tracking algorithm. Detailed Embodiments

[0045] The following embodiments can enable those skilled in the art to understand the present invention more comprehensively, but do not limit the present invention in any way.

[0046] See Figure 1, this embodiment discloses a bridge deformation measurement method based on airborne dual cameras and target tracking. The bridge deformation measurement method includes the following steps:

[0047] S1. Arrange a number of measurement targets at the measurement points to be measured on the side of the bridge, and use an unmanned aerial vehicle (UAV) equipped with a dual-camera system including a telephoto camera and a wide-angle camera to collect bridge videos. Among them, the positions of the telephoto camera and the wide-angle camera are coaxial and fixed, and both are facing the bridge and continuously shooting. The field of view of the wide-angle camera includes the left and right piers of the bridge. The telephoto camera is only used to collect the image videos of the measurement targets at the measurement points to be measured on the side of the bridge.

[0048] S2. Build a target recognition model based on the YOLO v5s network, import the video frame images collected by the telephoto camera and the wide-angle camera into the target recognition model respectively, use the target recognition model to identify the positions of the measurement targets from the bridge video frame images collected by both, and segment the measurement target images from the video frames.

[0049] S3. Use a feature extraction network to distinguish between different positions of the segmented measurement target images, assign a fixed ID to each measurement target, obtain the center coordinates of the bounding boxes of the measurement targets in each frame, and draw a rough displacement trajectory of the measurement targets.

[0050] S4. Based on the multi-target tracking algorithm of DeepSORT, after obtaining the center coordinates of the bounding boxes of the measurement targets in the current frame, use Kalman filtering to predict the predicted trajectory of the current coordinates, calculate the correlation degree between the predicted trajectory and the actual coordinates of the center of the bounding box in the next frame, and correct the rough displacement trajectory of the measurement targets according to the correlation degree to obtain the actual displacement trajectory of the center points of the measurement targets.

[0051] S5. Identify the center points of the measurement target images at the sub-pixel level to obtain the sub-pixel level displacement trajectories of the center points of the measurement targets corresponding to the images collected by the two cameras.

[0052] S6. Combine the sub-pixel level displacement trajectories corresponding to the two cameras and the positional relationship between the two cameras, and eliminate the displacement errors caused by the shaking of the UAV from the measurement results of the moving targets. In this way, based on the absolute displacement values at each time point after removing the errors, the dynamic displacement trajectory curve of the bridge can be drawn.

[0053] This embodiment divides the bridge deformation measurement into three steps: arranging targets, collecting bridge videos, and data analysis. Among them, the targets are arranged at the measurement points that need to be measured, the UAV hovers on the side of the bridge, and shoots the video information of the bridge under load. The algorithms in the data analysis can realize high-frame-rate real-time calculation of the displacement of the preset targets on the bridge, and can be used for the displacement detection of in-service bridges, and effectively evaluate the safety performance of the bridge structure.

[0054] (1) Measurement target and UAV setup

[0055] The target used for laying at the measuring point positions on the side of the bridge is made of lightweight aluminum plate. The pattern of the target consists of four rectangles at the four corners and a circle in the middle. Its advantages are as follows: Firstly, the corner coordinates of the four rectangles at the four corners can be used as known points to calculate the scale parameter of the image, and when the image has a large inclination, the image can be corrected for inclination. Secondly, the center point formed by the coordinates of the four rectangles at the four corners coincides with the center point coordinates of the middle circle. Therefore, even when part of the target is blocked, the center point coordinates can be calculated through a small number of corner coordinates. Assume that the center point coordinates of the target are P c (x c ,y c ). The four corner points detected are P1(x1, y1), P2(x2, y2), P3(x3, y3), P4(x4, y4), and the center point coordinates of the center circle are P5(x5, y5). Then the calculation method of P c is to take the average of these five coordinates.

[0056] The UAV used for shooting the side deformation video of the bridge can adopt a general - type UAV. The camera system carried by the UAV is a parallel - arranged dual - camera system including a long - focal camera and a wide - angle camera. Among them, the long - focal camera is used to shoot the target laid on the side of the bridge, and the wide - angle camera is used to shoot the entire bridge. Before applying the dual - camera for measurement, the Zhang calibration method needs to be used to independently calibrate the dual - cameras, and the obtained camera internal parameters are: the internal parameter of the wide - angle camera is K w , and the internal parameter of the zoom camera is K t . These internal parameters are used for both de - distorting the source data and analyzing the mutual relationship between the displacement results calculated by the two cameras through homography calculation of the dual - cameras, and eliminating the shaking of the UAV itself. Specifically, for eliminating the base - point vibration of the UAV itself, a dual - camera system with a long - focal lens and a wide - angle lens is adopted, and the shooting method of simultaneously shooting the deformation detection points of the bridge body and the fixed reference points of the bridge piers is used. By theoretical derivation of the relationship between the displacements measured by the dual - cameras in this case, the displacement of the UAV itself is eliminated. In actual operation, the UAV equipped with a coaxial dual - camera can be hovered on one side of the bridge, and the camera is directed at the bridge to continuously shoot videos. The camera of the UAV is set to the free mode, that is, the horizontal rotation and vertical rotation angles of the camera remain unchanged, and it does not rotate due to the change of the UAV's attitude, maintaining the attitude of always facing the bridge. Therefore, the relative displacement change caused by the rotation of the camera can be not considered. The wide - angle lens of the camera is set to just shoot a field of view including the left and right bridge piers.

[0057] (2) Calculation of the absolute displacement of the bridge

[0058] Refer to Figure 2 , assume that the world coordinate system is (x w ,y w, z w ), the unstable UAV platform coordinate system is (x u , y u , z u ), the coordinate systems of the two cameras are (x k , y k , z k ), where k = 1, 2. The coordinate transformation of the camera is (α, β, γ, Δx, Δy, Δz), where (α, β, γ) are the rotation angles of the camera around three directions, and (Δx, Δy, Δz) are the translation components of the camera in three directions. Let P i be the fixed target control point within the field of view of camera W, and p i be the projection point of the control point in the image of camera W, where i = 1, 2, 3,..., n. When the camera moves, that is, P i moves relatively, a displacement component of the projection point p i in a certain direction will include the rotation of the camera around the rotation axis in that direction plus the translation component in that direction, which can be expressed (taking the y-axis as an example):

[0059] P′ y =-P x sinγ w +P y cosα w cosγ w +P z sinα w cosγ w +ΔY w

[0060] where α w , β w and γ w are the rotation angles of the camera and the fixed target in the three-axis directions, P x , P y and P z are the coordinates of P i before the movement occurs, and ΔY w is the translation component in the y-axis direction. Then the relative vertical displacement between camera W and the fixed target is:

[0061] H w =P′ y -P y =P y (cosα w cosγ w -1)+P z sinα w cosγ w +ΔY w

[0062] The point displacement in the image coordinate system is as follows:

[0063] Δp iy = v′ i - v i = k wi H w

[0064] where the proportionality coefficient k wi can be calculated from the target size and camera parameters through the homography relationship, and can also be simply calculated from the object distance and focal length when the camera is facing the target directly during shooting. Since the movement in the object distance direction is very small, it can be considered that k wi does not change and is a constant. Since Δp iy can be directly calculated from two frames of images, α w , β w and γ w are also very small under the stable compensation of the three-axis gimbal. Therefore, the above formula can be simplified as:

[0065] Δp iy = k wi ΔY w

[0066] Thus, the vertical displacement of the camera platform during shooting can be obtained.

[0067] Cameras W and T are fixedly connected, so the relationship between the two cameras is:

[0068]

[0069] where R t，w and T t，w are the rotation and translation relationships between the two cameras. Therefore, after obtaining the rotation and translation relationships between the two cameras through dual-camera calibration, the vertical displacement of camera W can also be transmitted to camera T. Similar to camera W, the vertical displacement of the three-dimensional points captured by camera T is:

[0070]

[0071] Then, the pure displacement of the bridge excluding the base point displacement of the UAV platform is:

[0072] ΔP b = k ti ΔY t - R t，w k wi ΔY w .

[0073] (3) Displacement of Deep Learning Multi-Target Tracking

[0074] See Figure 3, for the images detected by the object detection network, the sizes of the detected bounding boxes can be quickly extracted, and the center coordinates of the detected objects can be calculated. Connecting the center coordinates frame by frame can be regarded as the rough pixel displacement of the object. Aiming at the problems of the change of the ID of the object detection after occlusion and the fact that directly calculating the center point can only obtain the pixel-level displacement result generated by this method, on the basis of the object detection network, an object tracking method and a sub-pixel refinement extraction method are added. The method for analyzing the target displacement from the video is divided into three steps:

[0075] The first step of data analysis is to apply a deep learning object detection network to the original video to identify the target position in the video and segment the target from the video frames. The network used is the YOLO v5s network trained in the bridge target dataset. Using the automatic target recognition based on the YOLO v5s network has more advantages in terms of speed and platform consistency. The pictures in the established dataset come from the target videos of multiple types of bridges actually taken by drones. The pictures contain a large number of target images with different lighting conditions, which helps to improve the universality and practicability of the proposed method. Specifically, by pasting targets on the bridge surface and then measuring the target positions, the center of the target is used as the displacement measurement point. 1336 target images under different lighting conditions are established, manually marked with positions, and the dataset is made in the format of PASCAL VOC. Finally, the k-means clustering method is used to cluster them. Figure 5 Schematic diagram of the training result of the target recognition model based on the YOLO v5s network.

[0076] The second step of data analysis is to apply a feature extraction network to the segmented target images to distinguish between different targets at multiple different positions, so that each target is assigned a fixed ID. The core of the second step of data analysis is the multi-object tracking algorithm based on DeepSORT. This algorithm uses a convolutional neural network to learn the identified targets in advance, so as to judge the differences between different targets through deep learning classification. After obtaining the recognition box or the center coordinates of the target in the current frame, the Kalman filter is used to predict the predicted trajectory of the current coordinates, and the predicted trajectory is input into the target detection of the next frame. If the detection result of the next frame has a good correlation with the predicted trajectory of the next frame, the detection result is considered correct, and this process continues.

[0077] Specifically, see Figure 4, after achieving real-time identification of the target, calculate the center of the target marking box for each frame and connect the centers of the marking boxes in each frame. In fact, a rough displacement trajectory of the target is obtained. Use the multi-object tracking algorithm based on DeepSORT to accurately handle the re-identification problem during target movement and the identity transformation problem under large displacement or occlusion. After obtaining the recognition box or the target center coordinates of the current frame, use Kalman filtering to predict the predicted trajectory of the current coordinates and input the predicted trajectory into the target detection of the next frame. If the detection result of the next frame has a good correlation with the predicted trajectory of the next frame, it is considered that the detection result is correct, and the process continues accordingly. For the situation where the target is not recognized, perform IOU matching on this trajectory and the target detected again later, and continue to depict the trajectory. Figure 6 It is a schematic diagram of the training result of the inter-class recognition network based on the DeepSORT multi-object tracking algorithm.

[0078] The third step of data analysis is to further refine the pixel-level target center displacement obtained in the second step to the sub-pixel level to improve the accuracy. Identify the center point of the target image at the sub-pixel level to obtain the sub-pixel level displacement trajectory of the target center point. Use the rectangular corner point coordinates of the four corners of the target as known points to calculate the scale parameter of the image and perform tilt correction. Adopt the method of graphic detection to screen the coordinate information of the rectangular corner points, use the Hough transform to detect the center coordinates of the circle for the fitted prototype, and take the average value of the five extracted coordinates. The final result is the sub-pixel level target center coordinates.

[0079] The aforementioned data analysis method has the advantages of high measurement accuracy and being unaffected by environmental illumination and the displacement of the UAV itself, and can effectively improve the efficiency of bridge displacement monitoring; in addition, based on the deep learning tracking method, it can achieve sub-pixel level bridge displacement monitoring and can be further used for the performance evaluation of the bridge under long-term working conditions.

[0080] Next, a typical bridge case is used to illustrate the specific implementation steps of the displacement rapid measurement method based on the UAV platform and the deep learning target tracking algorithm of this embodiment.

[0081] Step 1: Image acquisition. Hover the drone at a distance of about 20 m from the bridge, and aim the camera directly at the bridge to continuously shoot videos. Set the camera to free mode and maintain the attitude of always facing the bridge directly. Set the wide-angle lens of the camera to just capture the field of view including the left and right bridge piers. Set the zoom camera to 40x zoom, with a focal length of 240 mm, and adjust the field of view so that the target is clearly visible. The ground-fixed camera aims at the target pasting position for shooting. Set the acquisition frame rates of the ground camera and the drone camera to 30 fps. The size of the targets pasted on the bridge surface is 10 cm, and 3 targets are pasted near the mid-span respectively. For the measurement of the relative displacement of the bridge pier position, no targets are used, and the measurement is carried out by using the matching relationship of the texture of the bridge pier itself. During the entire measurement process, 3 targets near the mid-span position are photographed respectively, and the shooting time for each time is 3 minutes.

[0082] Step 2: Calculate the absolute displacement of the bridge. Adopt the method proposed in this embodiment to process the videos of the telephoto camera and the wide-angle camera respectively. Among them, for the video of the wide-angle camera, the two bridge piers on both sides are used as the measurement positions, and the calculated result is the relative displacement of the drone relative to the fixed point of the bridge pier. Subtract it from the target displacement obtained from the analysis of the telephoto camera video, and then the true displacement of the target relative to the fixed point of the bridge pier can be obtained.

[0083] Compare the method adopted in this embodiment with the traditional DIC method, and calculate the correlation between them by using the Pearson correlation coefficient. For target 1, the correlation between the displacement curves of the proposed method and the DIC method adopted by the fixed camera is 0.885, and the correlation between the displacement curves of the proposed method and the proposed method measured by the fixed camera is 0.917. For target 2, the above correlations are 0.766 and 0.772 respectively. For target 3, the above correlations are 0.761 and 0.792 respectively. For the displacement curves before and after removing the drone displacement, the three targets are 0.374, 0.135 and 0.248 respectively. The calculation results of the correlation also prove the above conclusion.

[0084] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions within the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A method for measuring bridge deformation based on airborne dual cameras and target tracking, characterized in that, The bridge deformation measurement method includes the following steps: S1. Arrange a number of measurement targets at the measurement points to be measured on the side of the bridge, and use a drone to carry a dual-camera system including a telephoto camera and a wide-angle camera to collect bridge videos. Among them, the positions of the telephoto camera and the wide-angle camera are coaxial and fixed, and both are facing the bridge and continuously shooting. The field of view of the wide-angle camera includes the left and right piers of the bridge. The telephoto camera is only used to collect the image videos of the measurement targets at the measurement points to be measured on the side of the bridge. S2. Build a target recognition model based on the YOLO v5s network, import the video frame images collected by the telephoto camera and the wide-angle camera into the target recognition model respectively, use the target recognition model to identify the positions of the measurement targets from the bridge video frame images collected by both, and segment the measurement target images from the video frames. S3. Use a feature extraction network to distinguish between different positions of the segmented measurement target images, assign a fixed ID to each measurement target, obtain the center coordinates of the bounding boxes of the measurement targets in each frame, and draw a rough displacement trajectory of the measurement targets. S4. Based on the multi-object tracking algorithm of DeepSORT, after obtaining the center coordinates of the bounding boxes of the measurement targets in the current frame, use Kalman filtering to predict the predicted trajectory of the current coordinates, calculate the correlation degree between the predicted trajectory and the actual coordinates of the center of the bounding box in the next frame, and correct the rough displacement trajectory of the measurement targets according to the correlation degree to obtain the actual displacement trajectory of the center point of the measurement targets. S5. Identify the center points of the measurement target images at the sub-pixel level to obtain the sub-pixel level displacement trajectories of the center points of the measurement targets corresponding to the images collected by the two cameras. S6. Combine the sub-pixel level displacement trajectories corresponding to the two cameras and the positional relationship between the two cameras to eliminate the displacement errors caused by the shaking of the drone from the measurement results of the moving targets.

2. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 1, characterized in that, In step S1, before measurement using the dual-camera system, the Zhang calibration method is used to independently calibrate the telephoto camera and the wide-angle camera, and the internal parameter matrix \(K\) of the wide-angle camera is obtained w and the internal parameter matrix \(K\) of the telephoto camera t .

3. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 1, characterized in that, For the relative displacement measurement of the pier positions, measurement targets are not used, and the matching relationship of the textures of the piers themselves is used for measurement.

4. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 1, characterized in that, In step S2, the process of building a target recognition model based on the YOLO v5s network includes the following steps: Paste targets on the bridge surface, measure the positions of the targets, and use the centers of the targets as displacement measurement points. Collect a number of target images under different lighting conditions, manually mark the positions, make a target dataset in the format of PASCAL VOC, and use the k-means clustering method to cluster the target images in the dataset. Import the target dataset into the YOLO v5s network for training and verification to obtain a trained target recognition model.

5. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 1, characterized in that, In step S4, the process of correcting the rough displacement trajectory of the measurement targets according to the correlation degree to obtain the actual displacement trajectory of the center point of the measurement targets includes: Calculate the correlation degree between the predicted trajectory and the actual coordinates of the center of the bounding box in the next frame. If the correlation degree reaches the predicted correlation degree threshold, it is determined that the detection result in the next frame is correct; otherwise, perform IOU matching on the predicted trajectory and the re-detected measurement targets to continue depicting the trajectory.

6. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 1, characterized in that, The pattern of the measurement target includes four rectangular marks at the four corners and a circular mark in the middle.

7. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 6, characterized in that, In step S5, the process of identifying and measuring the center point of the target image at the sub-pixel level to obtain the sub-pixel point-level displacement trajectory of the center points of the measurement targets corresponding to the images collected by the two cameras includes the following steps: Taking the coordinates of the four rectangular marks at the four corners of the target as known points, calculating the scale parameter of the measurement target image, and performing skew correction on the measurement target image; Use the method of graphic detection to screen the coordinates P1(x1, y1), P2(x2, y2), P3(x3, y3), P4(x4, y4) of four rectangular markers, fit to obtain the prototype of the measurement target, use the Hough transform to detect the center coordinates P5(x5, y5) of the fitted prototype, and take the average value of these five coordinates as the center coordinates P of the measurement target image c (x c , y c ).

8. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 6, characterized in that, If some of the rectangular marks and circular marks in the measurement target image are missing, taking the average value of the coordinates of the remaining marks as the center coordinates of the measurement target image.

9. The method for measuring bridge deformation based on airborne dual cameras and target tracking according to claim 1, characterized in that, In step S6, the pure displacement of the bridge after removing the base point displacement of the UAV platform is: ΔP b = k ti ΔY t - R t，w k wi ΔY w where R t，w and T t，w are the rotation and translation relationships between the two cameras, where k i is the scale factor of the target at time T1, and ΔY w and ΔY t are the vertical displacements of the target measured by the wide-angle camera and the telephoto camera.

10. A bridge deformation measurement system based on an unmanned aerial vehicle (UAV) airborne dual camera and deep learning tracking, characterized in that, The bridge deformation measurement system includes a number of measurement targets, a UAV carrying a dual-camera system including a long-focus camera and a wide-angle camera, and a processor; The number of measurement targets are distributed at the measurement points to be measured on the side of the bridge; the UAV hovers on the side of the bridge according to the control instructions of the processor, uses the long-focus camera and the wide-angle camera to collect bridge videos, and sends the collected bridge videos to the processor; The processor calculates the vertical displacement of the measurement target at the measurement point to be measured on the side of the bridge by using the bridge deformation measurement method described in any one of claims 1-9.

Images