Bridge displacement and rotation angle synchronous real-time monitoring method and system based on online camera

Abstract

The application discloses a bridge displacement and rotation angle synchronous real-time monitoring method and system based on an online camera, uses an infrared array target multi-feature point as a visual input to construct a geometric relation between a camera and a motion target coordinate system, and in view of the problems that a traditional centroid method cannot realize one-to-one corresponding continuous matching tracking of multiple points and local precision of the target is insufficient due to environmental interference, thereby causing large overall solving error, proposes a motion target relative displacement and rotation angle measurement algorithm based on a centroid array adaptive independent search and a coordinate system conversion matrix purification optimization, and effectively improves result solving precision and robustness. In addition, a monitoring system is developed, edge calculation of an embedded processing platform directly outputs bridge displacement and rotation angle data to terminal users, and thus all-weather, wireless and real-time monitoring of bridge dynamic displacement and rotation angle is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of structural health monitoring, and particularly relates to a bridge displacement and rotation angle synchronous real-time monitoring method and system based on an online camera, which can realize all-weather, wireless and real-time monitoring of bridge dynamic displacement and rotation angle. BACKGROUND

[0002] The structural response measurement of a bridge under the action of vehicle load and environment (wind, temperature) is an important content of bridge safety evaluation. In particular, in the load test of a bridge, the accurate measurement of bridge response can be compared with the bridge deformation condition of finite element analysis, and the deep-level characteristic parameters (such as structural frequency response function and modal flexibility) of the bridge can be calculated according to the structural dynamics theory to provide a basis for bridge damage identification. The most commonly measured bridge responses are displacement, strain, rotation angle, vibration acceleration and the like. In recent years, intelligent sensors such as optical fiber sensors, optical fiber sensors, piezoelectric sensors, electromagnetic expansion material sensors and GPS have been widely used in the measurement of the above bridge responses, and can be measured in a more accurate and effective manner.

[0003] Rotation angle measurement includes beam end rotation angle, beam body rotation angle, support rotation angle, bridge tower rotation angle and the like. Like displacement parameters, rotation angle parameters are sensitive to damage of any part of a bridge. Since displacement is subject to its own properties such as the need for reference measurement points, rotation angle parameters are often easier to measure and have great advantages in structural evaluation. In addition, the beam end rotation angle has a great influence on the vehicle load and unloading action and the impact on the bridge, especially the railway bridge.

[0004] The existing rotation angle measurement means uses information based on an inclinometer. The inclinometer can provide information about the health status of a structural system. Most existing inclinometers are based on gravity / acceleration and can provide accurate static response measurement. However, for dynamic tilt, the acceleration will significantly affect the measured response due to crosstalk and the like. Therefore, dynamic tilt measurement is still a challenging problem. One method is to integrate the output of a gyroscope sensor that measures angular velocity to obtain the tilt. However, the gyroscope measurement has the problem of poor low-frequency sensitivity.

[0005] The visual-based structural response measurement method has been widely applied in the field of structural health monitoring due to its advantages of non-destructive, non-contact, high precision and multi-point synchronous measurement. Machine vision is more commonly used in bridge displacement measurement, which improves the sub-pixel measurement accuracy to better adapt to the dynamic displacement non-contact measurement of bridge structures. The visual-based rotation angle measurement is not widely used at present. There are three main problems in the existing visual-based rotation angle measurement methods: 1) Most of the target markers use chessboard, coded points and other feature markers for measurement, which are easily affected by environmental interference such as light and cannot be measured all-weather; 2) The existing methods mostly use the displacement difference between two points to indirectly calculate the rotation angle information; 3) The existing methods rely on a large amount of recorded offline video data on the workstation for post-processing, and do not directly provide displacement time history to bridge inspectors, which is not suitable for unattended deployment in remote areas for regular long-term monitoring. SUMMARY

[0006] The technical problem solved by the present application is that the existing displacement rotation angle measurement method has bottlenecks. The present application proposes a bridge displacement rotation angle synchronous real-time monitoring method and system based on an online camera, which innovatively uses an infrared array target multi-feature point as a visual input to construct the geometric relationship between the camera and the moving target coordinate system, and directly outputs the bridge displacement and rotation angle that can be sent to the end user through edge computing on an embedded processing platform, thereby realizing all-weather, wireless and real-time monitoring of bridge dynamic displacement and rotation angle.

[0007] TECHNICAL SOLUTION

[0008] A bridge displacement rotation angle synchronous real-time monitoring method based on an online camera, the real-time monitoring method comprising the following steps:

[0009] S1, fixing the online camera device on the bridge reference point through a stable mechanical device, setting multiple target markers, the target markers corresponding to the measured targets on the bridge, calibrating the target markers using a chessboard, adjusting the parameters of the online camera device and the target markers, and then acquiring real-time motion target image sequences;

[0010] S2, estimating the conversion relationship between the measured target world coordinate system and the camera coordinate system corresponding to each frame of motion target image, and calculating the position and attitude information of the measured target before and after the motion through the position change of the space pose of the measured target in the camera coordinate system;

[0011] S3, using a centroid array tracking matching method based on a comprehensive predictor and continuous adaptive search, combining the position, area, gray scale and shape attributes of the measured target, proposing a target multi-attribute decision function to determine whether the measured target is blocked, if the measured target is blocked, using the comprehensive predictor to predict the position parameters of the tracked target until the target reappears, to complete the continuous adaptive matching tracking of the centroid of each light spot in the target marker between image sequences;

[0012] S4, calculating displacement and rotation angle by using a minimum plane pose estimation algorithm based on subset model purification and nonlinear iterative optimization.

[0013] Further, in step S1, the online camera device comprises a GIGE industrial camera and a POE switch (Power Over Ethernet) connected in sequence; the POE switch is used for powering the GIGE industrial camera and sending the motion target image sequence collected by the GIGE industrial camera to the computer, so that the displacement and rotation angle of the bridge are obtained by the computer in real time, and then the displacement and rotation angle of the bridge are sent to the monitoring center.

[0014] Further, the target adopts an infrared array lamp bead target with a wavelength of 850 nm, and a plurality of infrared small lamp beads with known spacing are uniformly arranged on a single target.

[0015] Further, a narrow-band filter that only allows infrared light to pass through is installed on the lens of the industrial camera of the online camera device.

[0016] Further, in step S2, the conversion relationship between the world coordinate system of the measured target corresponding to each frame of motion target image and the camera coordinate system is estimated, and the position and attitude information of the measured target before and after the motion are obtained by the position change of the to-be-measured target space pose in the camera coordinate system, including the following steps:

[0017] The conversion relationship between the world coordinate system of the measured target corresponding to each frame of image and the camera coordinate system is estimated, and the camera coordinate system is used as an intermediary, and P 11 and P 12 are the coordinates of the target in the initial coordinate system and the coordinate system after the motion, P c is the coordinate of the target in the camera coordinate system, is the rotation matrix of the target in the initial coordinate system and the coordinate system after the motion to the camera coordinate system, is the translation vector of the target in the initial coordinate system and the coordinate system after the motion to the camera coordinate system, and the position and attitude information of the target before and after the motion are obtained.

[0018] Let the transformation matrix of the coordinate system before and after the target motion be and the relative relationship between the coordinate systems before and after the motion is obtained as: The above two formulas are combined to obtain

[0019] Let P 1i be the coordinate of the target in the i-th frame coordinate system, and Pc the coordinate of the target in the camera coordinate system, the rotation matrix of the target in the i-th frame coordinate system to the camera coordinate system, the translation vector of the target in the i-th frame coordinate system to the camera coordinate system, the relative relationship of the coordinate of the target in the i-th frame coordinate system to the coordinate at the initial moment is:

[0020]

[0021] Let the rotation matrix of the target in the i-th frame coordinate system to the coordinate at the initial moment be ΔR p , the rotation component of the m-th row and the n-th column of the rotation matrix, then:

[0022]

[0023] Solving the rotation angles α, β, γ around the x, y, z three axes respectively are:

[0024]

[0025] Let the translation vector of the target in the i-th frame coordinate system to the coordinate at the initial moment be Δt p , the translation vector component of the three axes of the target at the initial moment, the translation vector component of the three axes of the target at the i-th frame moment, the the rotation component of the m-th row and the n-th column of the rotation matrix of the inverse matrix of the matrix, then:

[0026]

[0027] By substituting the measured data into the solution, ΔR p , Δt p , the spatial pose of the monocular camera is measured.

[0028] Further, in step S3, the centroid array tracking matching method based on comprehensive predictor and continuous adaptive search is adopted to complete the process of continuous adaptive matching and tracking of the centroids of each light spot in the target in the image sequence, which includes the following steps:

[0029] The x-axis and y-axis coordinates (X l [k+1], Y l [k+1]) of the predicted point of the k+1-th frame based on the linear prediction empirical formula of 3 points are:

[0030]

[0031] Wherein, X l [i], Y l[i] is the x-axis and y-axis coordinates of the centroid point of the i-th frame of linear prediction method, i=k-2, k-1, k, k+1;

[0032] The x-axis and y-axis coordinates (X c [k+1], Y c [k+1]) of the (k+1)-th frame of predicted points are obtained based on the 3-point circular arc trajectory prediction formula:

[0033] Wherein,

[0034]

[0035]

[0036]

[0037] In the formula, X j , Y j are the x-axis and y-axis coordinates of the j-th frame of centroid points of the circular arc prediction method, j=k-2, k-1, k, k+1;

[0038] The motion of the target is regarded as a combination of linear motion and quadratic curve motion, and the formula of the combined predictor F[k+1] after combination is as follows:

[0039] F[k+1]=τf l [k+1]+(1-τ)f c [k+1]

[0040] Wherein, f l [k+1] is a linear predictor function, f c [k+1] is a circular arc predictor function, and τ is a proportionality coefficient, which is in the range of 0-1.

[0041] Further, in step S3, the centroid array tracking and matching method based on the combined predictor and continuous adaptive search is adopted to complete the process of continuous adaptive matching and tracking of the centroids of each light spot in the target in the image sequence, which includes the following steps:

[0042] The size and position of the search window are initialized, and the position and size of the search window are adaptively adjusted according to the results obtained in the last frame, and the size b of the automatic search frame is:

[0043]

[0044] Wherein, A i is the pixel area size of any point A in the i-th frame, B is the point closest to point A in a certain direction, C is the point closest to point A in the normal direction of the line connecting points A and B, r A，i , rB，i , r C，i are the actual radius size of A, B, C points in the i-th frame, D AC，i , D AB，i are the actual distances from A to B and C, and a takes the value of 0.5-0.8.

[0045] Further, in step S4, the process of calculating the displacement and the rotation angle by using the minimal plane pose estimation algorithm based on the subset model purification and the nonlinear iterative optimization includes the following steps:

[0046] S41, a rough pose is calculated by using the IPPE method (Infinitesimal Plane-based Pose Estimation), a minimal subset model containing 7 matching point pairs is randomly selected from all 3D-2D matching point original data of the array target, 4 matching point pairs are used to calculate the rough pose by using the IPPE algorithm, and the remaining 3 matching point pairs are used to verify whether the initial model is suitable;

[0047] S42, inliers are calculated, the three-dimensional points in the remaining matching points in the subset are re-projected as two-dimensional points according to the rough pose calculated in step S41, and the re-projection error is calculated, if the test error is less than the threshold value, the remaining matching points are used to adaptively check the subset model; if the test error of 1 matching point pair exceeds the threshold value, the initial subset model is directly discarded, and the next loop of iterative estimation is returned to step S41 to select 7 matching point pairs; otherwise, the data is divided into outliers and inliers according to the error threshold value according to the re-projection error of the remaining matching points;

[0048] S43, the number of inliers is judged, if it is less than the set threshold value, it is returned to step S41 to reselect the minimum subset of 7 points; if it is greater than the threshold value, the second IPPE calculation is performed on all matching points to obtain the first optimized pose information;

[0049] S44, the motion of the camera is controlled to continuously observe the specific target in the image, and the second optimization of the pose information is performed by minimizing the error between the expected state parameters s * and the actual state parameters s of the image features; specifically, the centroid of the 3D spot on the measured array target is regarded as a feature point, o P i is the 3D coordinate of the i-th centroid point on the measured array target in the motion target coordinate system; a virtual camera is set, the position and attitude of the virtual camera in the target coordinate system are defined as r, and the second optimization of the pose information is performed by minimizing the error between the observed data s i * and the 2D coordinates s iError between the real pose parameter r and the estimated pose parameter r is calculated as follows:

[0050]

[0051] Where Pr ξ (r, o P i ) is the projection model based on camera intrinsic parameters and camera pose r, N is the number of light spot centroid points in the array target; in the process of pose solving, the virtual camera initially located at r position moves following the virtual visual servo control algorithm to minimize the error parameter Δ, under the condition of convergence, the virtual camera reaches r d pose with minimized Δ, r d is the real camera pose parameter to be solved.

[0052] The application also discloses a bridge displacement and rotation angle synchronous real-time monitoring system based on an online camera.

[0053] The image acquisition unit comprises an online camera device and a target, and the target is imaged on an image sensor through a long-focus optical system based on an industrial camera; the image calculation unit calculates a motion target image sequence acquired by the image sensor by using the real-time monitoring method, and obtains displacement and rotation angle time course data of the target; the communication transmission unit forms a local area network through an existing optical fiber network on the bridge by using an industrial-grade router and switch, configures a fixed public network IP address, and transmits the calculation result of the image calculation unit to a remote monitoring center.

[0054] Advantages:

[0055] The bridge displacement and rotation angle synchronous real-time monitoring method and system based on an online camera can more effectively evaluate and maintain and manage the bridge, and improve the bridge monitoring efficiency, and are a very potential new bridge rotation angle monitoring method, and have the following main advantages:

[0056] (1) In view of the bottleneck problem of the existing displacement and rotation angle measurement method, the application innovatively uses an infrared array target multi-feature point as a visual input to construct a geometric relationship between a camera and a motion target coordinate system, and directly outputs bridge displacement and rotation angle which can be sent to a terminal user through edge calculation of an embedded processing platform, so that all-weather, wireless and real-time monitoring of bridge dynamic displacement and rotation angle is realized, and the existing machine vision displacement and rotation angle measurement which depends on a large amount of recorded offline video data on a workstation for post-processing is avoided.

[0057] (2) In view of the limitation that the camera optical axis needs to be perpendicular to the target plane when directly monitoring the support and other components by using a visual odometer, the mathematical relationship between the relative displacement rotation of the moving target coordinate system and the conversion matrix between the camera and target coordinate systems is derived, and this method has the advantages of flexible camera position layout in the engineering field and simultaneous monitoring of multiple targets.

[0058] (3) The active infrared array lamp bead target centroid tracking and matching method based on the comprehensive predictor and continuous adaptive search can overcome the problem that the traditional centroid method cannot realize automatic and continuous tracking and can only realize centroid positioning in the ROI range. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The bridge displacement rotation synchronous real-time monitoring method flow chart based on the online camera of the embodiment of the application;

[0060] Figure 2 The geometric relative relationship principle diagram of the camera and the moving target coordinate system based on the oblique axis photography;

[0061] Figure 3 The particle tracking and matching method flow chart;

[0062] Figure 4 The adaptive region search schematic diagram;

[0063] Figure 5 The minimum plane pose estimation algorithm measurement process diagram based on the subset model purification and nonlinear iterative optimization;

[0064] Figure 6 The laboratory layout diagram;

[0065] Figure 7 The measurement result comparison diagram, wherein (a) represents the L / 4 point rotation, and (b) represents the L / 4 point displacement;

[0066] Figure 8 The measurement result comparison diagram, wherein (a) represents the L / 2 point rotation, and (b) represents the L / 2 point displacement;

[0067] Figure 9 The real bridge test diagram;

[0068] Figure 10 The two-hour rotation measurement result comparison diagram;

[0069] Figure 11 The six-minute rotation measurement result comparison diagram;

[0070] Figure 12 The two-hour displacement measurement result comparison diagram;

[0071] Figure 13 Figure 6 is a six-minute displacement measurement comparison chart. DETAILED DESCRIPTION

[0072] The following examples enable those skilled in the art to more fully understand the present application, but are not intended to limit the present application in any way.

[0073] Reference Figure 1 The present embodiment refers to a bridge displacement and rotation angle synchronous real-time monitoring method based on an online camera, comprising the following steps:

[0074] (1) Install and build the online camera device and multiple target targets and adjust the parameters, calibrate with a checkerboard, obtain the camera intrinsic parameters, and then obtain the real-time image sequence of the moving target;

[0075] (2) Solve the mathematical relationship between the relative displacement and rotation angle of the moving target coordinate system under oblique axis photography and the conversion matrix between the camera and target coordinate systems;

[0076] (3) Use the centroid array tracking matching method based on comprehensive predictor and continuous adaptive search to achieve continuous adaptive matching and tracking of the spot centroid in the target target between image sequences;

[0077] (4) Use the minimum plane pose estimation algorithm based on subset model purification and nonlinear iterative optimization to calculate the displacement and rotation angle.

[0078] The online camera hardware device in step (1) is composed of a GIGE gigabit network port industrial camera, a mini computer, and a POE switch. The use of a GIGE gigabit network port industrial camera enables transmission distance to reach hundreds of meters. The POE switch can not only power the industrial camera but also perform data communication transmission. Only network cables need to be arranged without additional power lines. The mini computer can perform edge computing to obtain the displacement and rotation angle of the bridge in real time, and can wirelessly transmit the displacement and rotation angle data to a big data center to achieve real-time monitoring.

[0079] The multiple target targets in step (1) use multiple wavelength 850nm infrared array type lamp bead targets, and multiple infrared small lamp beads with known spacing are uniformly arranged on a single target. Infrared light can pass through fog, smoke and dust that visible light cannot penetrate, and a narrowband filter is installed on the camera to only allow infrared light to pass through, which can effectively eliminate the interference of natural stray light.

[0080] Preferably, step (2) is dedicated to real bridge applications, and is based on the geometric relative relationship between the camera and the moving target coordinate system under oblique axis photography, and the principle is as follows:

[0081] Monocular vision measurement object motion pose changes by estimating the conversion relationship between the world coordinate system of the measured target and the camera coordinate system corresponding to each frame of image respectively, and the position and attitude information of the target before and after motion can be obtained by taking the camera coordinate system as the medium. Let the relative relationship of the coordinate system pose before and after motion be:

[0082]

[0083] The i-th frame is

[0084]

[0085] Further derivation can obtain:

[0086] Then

[0087]

[0088] Therefore, the rotation matrix and the translation vector can be obtained as:

[0089]

[0090]

[0091]

[0092] In this way, the pose change AR of Pi relative to P0 can be detected through the position P0 and Pi of the space pose of the target to be measured in the camera coordinate system p , Δt p , and the space pose measurement of the monocular camera is realized, as shown in Figure 2 . Because oblique axis photogrammetry can be achieved, the application has stronger engineering applicability.

[0093] Preferably, the centroid array tracking matching method based on the comprehensive predictor and the continuous adaptive search in the step (3) has the following principles and steps:

[0094] According to the fixed distance between the array target light spots and the small change rate of the bridge corner, the change amount of each frame of light spot is much smaller than the distance between the light spots, and the application proposes a continuous adaptive multi-point centroid continuous tracking method. The position, area, gray scale and shape of the tracked target are utilized, a target multi-attribute decision function is proposed to judge whether the current target is blocked, and if the target is blocked, the position parameters of the tracked target are predicted by using the comprehensive predictor until the target reappears.

[0095] Let the optimal estimation equation of the horizontal coordinate of the target centroid be: X l The centroid coordinates X(t of the previous N frames can be calculated from the above equationi If the error sum of squares of N-point estimation is calculated as follows: The circular curve uses the least square method, and has The general solution of the N-point trajectory optimal estimation function is obtained by substituting the above formula into the optimal estimation equation. The prediction points can be obtained based on the 3-point linear prediction empirical formula and the 3-point circular trajectory prediction formula.

[0096] In the range involved in a certain prediction algorithm, the motion of the target can be regarded as a combination of linear motion and quadratic curve motion, so that both the performance of linear prediction for simple linear motion estimation and the prediction performance of quadratic prediction for random motion can be taken into account. The comprehensive predictor formula is as follows: F[k+1] = τf l [k+1] + (1-τ)f c [k+1]. Wherein, f l [k+1] is a linear predictor function, f c [k+1] is a circular arc predictor function.

[0097] In general, the size of the tracking window remains unchanged throughout the tracking process of the method, and when the target has a scale change, it will lead to inaccurate scale positioning. The size and position of the search window are initialized, and the position and size of the search window are adaptively adjusted according to the results obtained in the last frame. The size of the automatic search frame is:

[0098]

[0099] The relative pose solution between the camera coordinate system and the world coordinate system needs to solve the one-to-one matching problem of object points and image points. The existing spot centroid method for bridge deflection response measurement only locates the centroid of each ROI range, and obtains the displacement time history by subtracting the coordinates obtained by positioning the spot centroid in each frame. The distance between the spots is far apart and only single-direction vertical displacement occurs, and the displacement is small. They are independent and do not interfere with each other. However, when facing irregular motion of multiple spot centroids, this method fails to achieve continuous tracking of multiple spot centroids. In addition, if the matching problem is solved by the chessboard grid corner point sorting algorithm, since the unordered point set is sorted according to the principle of "from top to bottom, from left to right", once a large angle rotation occurs, the existing sorting method fails, and the matching problem of object points and image points is not fundamentally solved.

[0100] According to the fixed distance between the array target light spots and the small change rate of the bridge corner, the variation of each frame light spot is far less than the distance between the light spots, and a continuous adaptive multi-point centroid continuous tracking method is provided. The position, area, gray scale and shape of the tracked target are utilized, a target multi-attribute decision function is proposed to determine whether the target is blocked, if the target is blocked, the position parameters of the tracked target are predicted by using a comprehensive predictor until the target reappears, and the flow is shown in Figure 3 .

[0101] The best estimation equation of the centroid horizontal coordinate of the target is X1=a0+a1t, the centroid coordinates X(t i )(i=1, 2,.., N) of the previous N frames can be calculated from the above equation, and the error square sum of N-point estimation is calculated as follows: The circular arc curve uses the least square method, and then The general solution of the N-point trajectory best estimation function when the variance is minimum can be obtained by substituting the above equation into the best estimation equation. The predicted point (X l [k+1], Y l [k+1]) based on the 3-point linear prediction empirical formula is:

[0102]

[0103] The predicted point (X c [k+1], Y c [k+1]) based on the 3-point circular arc trajectory prediction formula is:

[0104] Wherein,

[0105]

[0106] Within the scope involved in a certain prediction algorithm, the motion of the target can be regarded as a combination of linear motion and quadratic curve motion, so that the performance of linear prediction for simple linear motion estimation and the prediction performance of square prediction for random motion can be considered. The comprehensive predictor formula is as follows: F[k+1]=τf l [k+1]+(1-τ)f c [k+1]. Wherein, f l [k+1] is a linear predictor function, and f c [k+1] is a circular arc predictor function.

[0107] Generally, the size of the tracking window remains unchanged during the whole tracking process of the method, and when the target has a scale change, it will cause inaccurate scale positioning. The size and position of a search window are initialized, and the position and size of the search window are adaptively adjusted according to the result obtained in the last frame, and the size of the automatic search frame is: The principle is as shown in Figure 4 FIG. 1. Wherein A i is the pixel area size of point A in the i-th frame, r A，i , r B，i , r C，i are the actual radius sizes of points A, B and C in the i-th frame, D AC，i , D AB，i is the actual distance from point A to points B and C, and generally, a takes a value of 0.5-0.8.

[0108] Preferably, the step (4) calculates the displacement and rotation angle by the minimal plane pose estimation algorithm based on the subset model purification and nonlinear iterative optimization, and the principle and steps are as follows:

[0109] According to the Perspective-n-Point (PnP) method, a rotation matrix R and a translation matrix T can be solved from a plurality of 3D-2D corresponding points, wherein the IPPE (Infinitesimal Plane-based Pose Estimation) is a pose solving algorithm suitable for the coplanar condition of 3D points in the world coordinate system, and based on the coplanar condition of the observation points, the relative pose relationship between the plane where the cooperative target is located and the camera coordinate system is solved by using the unit orthogonality of the rotation matrix. The present application is improved from two aspects of subset model purification optimization and virtual visual servo nonlinear iterative optimization (as shown in Figure 5 ).

[0110] ①The pose measurement algorithm based on subset model purification optimization includes three steps:

[0111] 1) Coarse pose solving: a minimum subset model containing 7 matching point pairs is randomly selected from all 3D-2D matching point original data of the array target, wherein at least 4 points are required for the IPPE algorithm, and a coarse pose is obtained by using the IPPE algorithm with 4 matching point pairs, and whether the remaining 3 matching point pairs are suitable for the initial model is verified.

[0112] 2) Inlier solving: according to the coarse pose obtained in the first step, the 3D points in the remaining 3D-2D points in the subset are re-projected as 2D points, and the re-projection error (unit: pixel) is calculated, if the test error is less than the threshold value, then the remaining matching point pairs are used to adaptively check the subset model; if the test error of 1 pair of matching point pairs exceeds the threshold value, then the initial subset model is directly discarded, and the next loop of iterative estimation is returned to step one to select 7 pairs of matching point pairs.

[0113] 3) First, determine the number of inlier points, if less than the set threshold, return to step one to select the minimum subset of 7 points. If greater than the threshold, then perform a second IPPE calculation on all 3D-2D matched points.

[0114] This method effectively improves the robustness of the PNP method in solving the pose, especially when the array target is affected by light and other factors, causing the accuracy of part of the centroid calculation to be insufficient. In addition, this method can quickly discard unreasonable initial models by selecting a minimum subset of 7 matched points, reducing the detection time required to verify the reasonableness of the initial model.

[0115] ② Virtual visual servoing (VVS algorithm for short) is to gradually and iteratively modify the pose of a "virtual" camera using visual servoing control technology, so that this camera gradually moves from a given reference state to a final state, and in this final state, the 3D model information corresponding to the target (feature points) is seamlessly fused with the current image features when projected onto the image of the "virtual" camera.

[0116] In the process of pose optimization, the VVS target is to continuously observe the specific target in the image by controlling the motion of the camera, which can be achieved by minimizing the error between the expected state parameters s * and the actual state parameters s of the image features. Under ideal conditions, there is a unique camera pose to minimize this error. In this invention, the centroids of many 3D spots on the measured array target can be regarded as feature points, o P is the 3D coordinates of the centroid points on the measured array target in the moving target coordinate system, and then a virtual camera is set, whose position and attitude in the target coordinate system are defined as r. By minimizing the error Δ between the observed data s * (the 2D coordinates of the centroid of the array target spot in the image, at this time s * =p) and the 2D coordinates s obtained by projecting the 3D feature points according to the pose parameters r to the image plane, the true pose parameters can be solved.

[0117]

[0118] where, pr ξ (r, o P i ) is the projection model based on the camera internal parameters ξ and the camera pose r, and N is the number of centroid points of the array target spots. In the process of pose solving, the virtual camera initially located at r position follows the virtual visual servoing control algorithm to move to minimize the error parameter Δ. Under the condition of convergence, the virtual camera reaches r dThe pose is minimized Δ, r d That is, the pose parameters of the real camera are solved.

[0119] Figure 6 A laboratory layout, Figure 7 And Figure 8 A schematic diagram of the measurement results is shown by Figure 7 And Figure 8 It can be seen that the monitoring method of the embodiment is very good. The proposed bridge displacement and angle synchronous real-time monitoring method based on online cameras and the implementation steps of the measurement system will be described below using the actual bridge online camera monitoring displacement and angle case.

[0120] The Pearl River Huangpu Bridge is a super large bridge connecting the urban area of Guangzhou and Panyu District in the Guangzhou East Ring Section of the main line of Tongzhi and Jingzhu National Highway, crossing the Pearl River North Branch of Pololou Waterway between Guangzhou Ocean Ship Repair Yard and Pololou Shipyard, and then crossing the South Branch of Dahoachui Waterway after Dahoachui, and entering the Hualong Town of Panyu District. The Pearl River Huangpu Bridge is composed of a north approach bridge, a north branch bridge (crossing the Pololou Waterway), a middle approach bridge, a south branch bridge (crossing the Dahoachui Waterway), and a south approach bridge. The vertical supports of the suspension bridge are installed at both ends of the steel box girder, two at each end, and four in total for the whole bridge. The vertical support adopts a roller type structure, the support slide plate and the side slide plate are connected with the steel box girder as a whole, and the rollers and the seat are fixed on the lower structure; when the steel box girder is displaced due to temperature change or under the action of live load, the support slide plate can roll on the rollers to complete the expansion and force transmission of the steel box girder. Through the connection of the side slide plate, the rocker shaft, the rocker seat, the seat and the pressing plate and the foundation bolt, the support can also bear a certain negative reaction force to limit the lifting of the steel box girder at the support. The lower part of the seat is provided with rubber elastic elements to allow the support assembly to have a small swing and displacement.

[0121] The main technical parameters of the vertical support are as follows: rated bearing capacity: 4900kN (downward), 1226KN (upward); horizontal movement: ±1050mm (longitudinal), ±10mm (transverse); angle: 0.06 radian (in the vertical plane), about 3.44°; 0.05 radian (in the horizontal plane), about 2.86°; assembly mass: 10564kg.

[0122] The panoramic view of the test bridge is shown in Figure 9 The monitoring results of the angle and displacement of the system of the application are shown in Figures 10 to 13 It can be seen that the application can more effectively evaluate and maintain the safety of the bridge and improve the efficiency of bridge detection, and is a very potential new method for angle monitoring.

[0123] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for online camera-based synchronous real-time monitoring of bridge displacement and rotation angle, characterized in that, The real-time monitoring method comprises the following steps: S1, fixing the online camera device on the bridge reference point through a stable mechanical device, arranging a plurality of target targets, the target targets corresponding to the measured targets on the bridge, calibrating the target targets by using a chessboard, adjusting the parameters of the online camera device and the target targets, and then acquiring a motion target image sequence in real time; S2, estimating the conversion relationship between the world coordinate system of the measured target corresponding to each frame of the motion target image and the camera coordinate system, and obtaining the position and attitude information of the measured target before and after the motion through the position change of the to-be-measured target space pose in the camera coordinate system; S3, adopting a centroid array tracking matching method based on a comprehensive predictor and a continuous adaptive search, combining the position, area, gray scale and shape attributes of the measured target, proposing a target multi-attribute judgment function, judging whether the measured target is blocked or not, if the measured target is blocked, predicting the position parameters of the tracked target by using the comprehensive predictor until the target reappears, so as to complete the continuous adaptive matching tracking of the spot centroid in each target target between the image sequences; S4, adopting a minimum plane pose estimation algorithm based on subset model purification and nonlinear iterative optimization to calculate the displacement and the rotation angle.

2. The online camera-based bridge displacement rotation angle synchronous real-time monitoring method according to claim 1, characterized in that, In step S1, the online camera device comprises a gigabit network port industrial camera and a POE switch connected in sequence; the POE switch is used for supplying power to the gigabit network port industrial camera and sending the motion target image sequence collected by the gigabit network port industrial camera to a computer, the displacement and the rotation angle of the corresponding bridge are obtained by the computer through real-time processing, and then the calculated displacement and rotation angle of the bridge are sent to a monitoring center.

3. The online camera based bridge displacement rotation angle synchronization real time monitoring method according to claim 1, characterized in that, The target target adopts an infrared array type lamp bead target with a wavelength of 850 nm, and a plurality of infrared small lamp beads with known spacings are uniformly arranged on a single target.

4. The online camera-based bridge displacement rotation angle synchronous real-time monitoring method according to claim 3, characterized in that, A narrow-band filter that only allows infrared light to pass through is mounted on the lens of the industrial camera of the online camera device.

5. The online camera based bridge displacement rotation angle synchronization real time monitoring method according to claim 1, characterized in that, In step S2, the process of estimating the conversion relationship between the world coordinate system of the measured target corresponding to each frame of the motion target image and the camera coordinate system, and obtaining the position and attitude information of the measured target before and after the motion through the position change of the to-be-measured target space pose in the camera coordinate system comprises the following steps: The conversion relationship between the world coordinate system of the measured target corresponding to each frame of image and the camera coordinate system is estimated respectively, and the camera coordinate system is used as an intermediary, and P 11 , P 12 are the coordinates of the target in the initial coordinate system and the coordinate system after motion, P c is the coordinate of the target in the camera coordinate system, is the rotation matrix of the target in the initial coordinate system and the coordinate system after motion to the camera coordinate system, is the translation vector of the target in the initial coordinate system and the coordinate system after motion to the camera coordinate system, and the position and attitude information of the target before and after motion is obtained: Let the transformation matrix of the coordinate system before and after the target motion be and The relative relationship between the coordinates before and after the motion is obtained as The above two formulas are combined to obtain Let P 1i be the coordinate of the target in the i-th frame coordinate system, P c be the coordinate of the target in the camera coordinate system, be the rotation matrix of the target in the i-th frame coordinate system to the camera coordinate system, be the translation vector of the target in the i-th frame coordinate system to the camera coordinate system, the relative relationship of the coordinate of the target in the i-th frame coordinate system to the coordinate at the initial moment is: Let the rotation matrix of the coordinate relative to the coordinate at the initial moment in the target i-th frame coordinate system be ΔR p , is the rotation component of the rotation matrix in the m-th row and n-th column, then: The rotation angles α, β and γ around the x, y and z axes are obtained as follows: Let the translation vector of the coordinate in the target coordinate system of the i-th frame relative to the coordinate at the initial moment be Δt p , is the translation vector component of the target three axes at the initial moment, is the translation vector component of the target three axes at the i-th frame moment, is is the rotation component of the rotation matrix of the inverse matrix of the rotation matrix in the m-th row and the n-th column, then: By the measured data into the solution to get ΔR p , Δt p , the spatial pose of monocular camera is measured.

6. The online camera based bridge displacement rotation angle synchronization real time monitoring method according to claim 1, wherein, In step S3, the process of completing the continuous adaptive matching tracking of the spot centroid in each target target between the image sequences by using the centroid array tracking matching method based on the comprehensive predictor and the continuous adaptive search comprises the following steps: The x-axis and y-axis coordinates of the (k+1)th frame prediction point (X l [k+1],Y l [k+1]) are obtained based on a 3-point linear prediction empirical formula as follows: wherein X l [i], Y l [i] are the x and y coordinates of the i-th frame centroid point of the linear prediction method, i = k-2, k-1, k, k+1. The x-axis and y-axis coordinates of the (k+1)th frame prediction point (X c [k+1],Y c [k+1]) are obtained based on a 3-point circular arc trajectory prediction formula: wherein In the formula, X j , Y j are the x-axis and y-axis coordinates of the jth frame of the circle arc prediction method, j=k-2, k-1, k, k+1; The motion of the target is regarded as a combination of linear motion and quadratic curve motion, and the formula of the combined comprehensive predictor F[k+1] is as follows: F[k+1] = τf l [k+1] + (1 - τ)f c [k+1] wherein f l [k+1] is a linear predictor function, f c [k+1] is a circular arc predictor function, and τ is a scale factor, with a value in the range 0-1.

7. The online camera-based bridge displacement rotation angle synchronous real-time monitoring method according to claim 6, characterized in that, In step S3, the process of completing the continuous adaptive matching tracking of the spot centroid in each target target between the image sequences by using the centroid array tracking matching method based on the comprehensive predictor and the continuous adaptive search comprises the following steps: The size and position of the search window are initialized, and the position and size of the search window are adaptively adjusted according to the result obtained in the last frame, and the size b of the automatic search frame is as follows: Wherein, A i is the pixel area size of any point A in the i-th frame, B is the point closest to point A in a certain direction, C is the point closest to point A in the normal direction of the line connecting points A and B, r A,i , r B,i , r C,i are the actual radius sizes of points A, B and C in the i-th frame respectively, D AC,i , D AB,i is the actual distance from point A to points B and C, and α is 0.5-0.

8.

8. The online camera based bridge displacement rotation angle synchronization real time monitoring method according to claim 1, characterized in that, In step S4, the process of calculating displacement and rotation angle by using the minimum plane pose estimation algorithm based on subset model purification and nonlinear iterative optimization includes the following steps: S41, a rough pose is calculated by using the IPPE method, a minimum subset model containing 7 matching point pairs is randomly selected from all three-dimensional-two-dimensional matching point original data of the array target, 4 matching point pairs are used to calculate a rough pose by using the IPPE algorithm, and the remaining 3 matching point pairs are used to verify whether the initial model is suitable; S42, inliers are calculated, the three-dimensional points in the remaining matching points in the subset are re-projected as two-dimensional points according to the rough pose calculated in step S41, and the re-projection error is calculated, if the test error is less than the threshold value, the remaining matching points are used to adaptively check the subset model; if the test error of 1 matching point pair exceeds the threshold value, the subset model is directly discarded, and the next loop iteration is returned to step S41 to select 7 matching point pairs; otherwise, according to the re-projection error of the remaining matching points, the data is divided into outliers and inliers according to the error threshold value; S43, the number of inliers is judged, if it is less than the set threshold value, it is returned to step S41 to select the minimum subset of 7 points; if it is greater than the threshold value, a second IPPE calculation is performed on all matching points to obtain the first optimized pose information; S44, controlling the motion of the camera to continue observing the target in the image by minimizing the error between the expected state parameters s * and the actual state parameters s of the image features, and a second optimization of the pose information; specifically, the centroids of the 3D spots on the measured array target are regarded as feature points, O P i are the 3D coordinates of the i-th centroid on the measured array target in the motion target coordinate system; a virtual camera is set, the position and pose of the virtual camera in the target coordinate system are defined as r, and the real pose parameters are solved by minimizing the error Δ between i * the 2D coordinates s obtained by projecting the 3D feature points onto the image plane according to the pose parameters r i ​ where pr ξ (r, O P i ) is the projection model based on the camera intrinsic parameters ξ and the camera pose r, and N is the number of the light spot centroid points in the array target; in the process of pose solving, the virtual camera initially located at the position r moves following the virtual visual servo control algorithm to minimize the error parameter Δ, and under the condition of convergence, the virtual camera reaches r d pose which minimizes Δ, r d is the real camera pose parameter to be solved.

9. An online camera-based bridge displacement and rotation angle synchronous real-time monitoring system, characterized in that, The real-time monitoring system comprises an image acquisition unit, an image calculation unit and a communication transmission unit; The image acquisition unit comprises an online camera device and a target target, the target target is imaged on an image sensor through a long-focus optical system based on an industrial camera; the image calculation unit calculates the motion target image sequence obtained by the image sensor by using the real-time monitoring method according to any one of claims 1-8, to obtain displacement and rotation angle time course data of the target target; the communication transmission unit uses an industrial-grade router and switch to form a local area network by using the existing optical fiber network on the bridge for the online camera at the field bridge site, and configures a fixed public network IP address, and transmits the calculation result of the image calculation unit to a remote monitoring center.

Images