A device and method for detecting the deflection of long-span bridges based on UAV vision

Abstract

This invention discloses a device and method for detecting the deflection of long-span bridges based on UAV vision, relating to the field of bridge inspection technology. Based on the structural characteristics of the main beam, a target parameter sequence is pre-constructed and targets are installed. Using a reverse estimation method, considering the photographing parameters and the target position of a single target, the shooting area of ​​a single target is pre-determined. A UAV task sequence is planned and generated based on a repetitive merging strategy to improve deflection detection efficiency. Single UAV tasks are executed sequentially according to the UAV task sequence, moving to the task area corresponding to each single UAV task and generating a single deflection sequence corresponding to that single UAV task using a vision-based classification tracking deflection detection method. The total deflection sequence is generated by statistically analyzing the deflection sequences corresponding to all UAV tasks in the UAV task sequence. This achieves automated main beam deflection detection without fixed-point measurement, solving the problems of difficult equipment deployment and low detection efficiency caused by fixed-point measurement in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of bridge inspection technology, and in particular to a device and method for detecting the deflection of long-span bridges based on UAV vision. Background Technology

[0002] The deflection of the main girder refers to the magnitude of the longitudinal linear displacement of the centroid at a certain cross-section of the girder in a direction perpendicular to the axis. If the deflection of the main girder exceeds the allowable range, or if it cannot return to its original position after being subjected to external forces, it can be determined that the main girder has a safety hazard. Therefore, main girder deflection testing is an important task in assessing the operational status of bridges during static load tests, taking into account their structural characteristics and load-bearing capacity.

[0003] Traditional methods for measuring the deflection of main beams typically involve contact measurements using instruments such as dial gauges and displacement meters. However, when bridges span rivers, highways, railways, or canyons, the inability to install support structures for these contact instruments presents installation challenges. In recent years, non-contact methods for measuring bridge deflection have emerged, utilizing laser technology and digital imaging technology.

[0004] A Chinese patent with authorization announcement number CN107588913B proposes a bridge deflection detection system and method, including multiple targets, an image acquisition device, a laser ranging module, a deflection calculation module, and an operation unit. The image acquisition device and the deflection calculation module are both connected to the operation unit, and the image acquisition device and the laser ranging module are both connected to the deflection calculation module. The horizontal rotation of the image acquisition device and the laser ranging module enables the measurement of deflection values ​​at multiple test points on a long-span bridge. This solves the problems of existing technologies that require multiple CCD cameras or laser rangefinders, resulting in low accuracy, high cost, inconvenient operation, and limited practicality.

[0005] However, existing technologies require additional calibration plates to be set up and the equipment to be moved to a fixed location for each test so that the image acquisition device can be calibrated. Furthermore, the orientation of the laser equipment needs to be precisely adjusted during the deflection test to achieve target alignment. For the deflection test of large-span main beams, this leads to problems such as difficult equipment deployment, complex operation, high cost, and reduced efficiency. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a device and method for detecting the deflection of long-span bridges based on UAV vision, providing an efficient non-fixed-point tracking deflection detection method.

[0007] The technical solution to achieve the purpose of this invention is as follows:

[0008] A method for detecting the deflection of long-span bridges based on UAV vision includes the following specific steps:

[0009] Target parameter sequence B is constructed based on the structural characteristics of the main beam. t And install the target;

[0010] The reverse estimation method, combined with the target installation location and imaging parameters, is used to determine the imaging area of ​​a single target to generate an imaging area sequence A. t Planning the shooting area sequence A based on the repeated merging strategy t Generate UAV mission sequence Ta u ;

[0011] Based on the UAV mission sequence Ta u The system sequentially executes individual UAV missions, proceeds to the mission area corresponding to each individual UAV mission, and generates a single deflection sequence corresponding to each individual UAV mission using a visual classification-based tracking deflection detection method.

[0012] Statistical analysis of unmanned aerial vehicle (UAV) mission sequences Ta u The total deflection sequence de is generated from the single deflection sequence corresponding to all individual UAV missions, thus completing the main beam deflection detection.

[0013] Furthermore, the structural characteristics of the main girder include the bridge's structural form, span, and support method, which are used to construct the target parameter sequence B. t Target parameter sequence B t Including K t The target parameter pairs of each target, K t For the total number of targets, target parameter sequence B t The specific format is as follows:

[0014]

[0015] Among them, b t (k t ) is the kth t The target parameter pairs for each target, k t For target numbering, p t (k t ) represents the k-th element in the world coordinate system. t The target installation location of each target.

[0016] Furthermore, the reverse estimation method is used to pre-determine and record the shooting area of ​​a single target, which includes the following specific steps:

[0017] Get the kth t The target installation position p of each target t (k t And the drone's photographic parameters;

[0018] Based on the horizontal width w of the camera sensor c and vertical height h c Determine the horizontal field of view θ of the camerac,w and vertical field of view θ c,h ;

[0019] Based on the camera's maximum horizontal adjustment angle and maximum vertical adjustment angle Expand the camera's horizontal and vertical fields of view to obtain the camera's actual horizontal field of view range. and actual vertical field of view in, and These are the leftmost and rightmost horizontal fields of vision, respectively. and These are the lowest vertical field of view and the highest vertical field of view, respectively.

[0020] With the target installation position p t (k t ( ) is the center of the sphere, at the kth position t Each target is located on the right side of the plane with the minimum shooting distance. and maximum shooting distance Construct a minimum and maximum hemisphere with a radius, and use the spatial region inside the maximum hemisphere but outside the minimum hemisphere as the initial shooting region A0(k). t );

[0021] Obtain the minimum movement distance of the drone. With minimum movement distance Generate a mesh to divide the initial shooting area To obtain a number of grid points;

[0022] Calculate the initial shooting area Internal single grid point and target installation position p t (k t The azimuth and elevation angles are used to determine whether a single grid point meets the view conditions. The spatial region encompassed by all grid points that meet the view conditions is considered as the k-th grid point. t Feasible shooting area for each target

[0023] Obtain the inscribed area within the feasible shooting region The largest cuboid region, as the k-th t The shooting area A of the target t (k t Recorded in the shooting area sequence A t middle.

[0024] Furthermore, based on the repetition merging strategy, the unmanned aerial vehicle (UAV) mission sequence Ta is generated. u The specific steps include the following:

[0025] Obtain the shooting area sequence A t ;

[0026] Perform an empty sequence decision to determine the UAV mission sequence Ta. u If the sequence is empty, change the maximum task number and maximum target number from 0 to 1, and proceed to the component region set expansion. If the sequence is not empty, confirm the UAV task sequence Ta. u The highest task number in;

[0027] Perform a target exhaustive test to obtain the maximum target number corresponding to the maximum task number, and determine whether the maximum target number is equal to the total number of targets K. t If equal to the total number of targets K t If the number of targets is less than the total number of targets K, then the repeated merging strategy is stopped. t Increment the maximum mission number and the maximum target number by 1.

[0028] Execute component region set expansion, placing the shooting region corresponding to the largest target number into the component region set corresponding to the largest task number;

[0029] Perform extended valid decision-making to determine whether the ratio of the volume of the overlapping region of all shooting regions in the component region set to the volume of any single shooting region is greater than or equal to the overlap ratio λ. A ;

[0030] If it is less than the overlap ratio λ A Remove the shooting area corresponding to the largest target number in the component area set, and based on the removed component area set, confirm the task area and task target number set corresponding to the largest task number, generate the UAV task corresponding to the largest task number, increment the largest task number by 1 and jump to execute the target exhaustive decision.

[0031] If it is greater than or equal to the overlap ratio λ A Increment the maximum target number by 1 and jump to the component region set expansion.

[0032] Furthermore, the method of generating a single deflection sequence corresponding to a single UAV mission using a vision-based classification-based tracking deflection detection method includes the following specific steps:

[0033] Suppose we are preparing to execute the nth drone mission Ta u (n), n = 1, 2, ..., N, where N is the total number of drone missions;

[0034] Confirm drone mission Ta u Task region A in (n) Ta (n) and the target number set k Ta (n), assuming the target number set k Ta (n)={k t ,…,k t +K n-1},n≤k t ≤k t +K n -1≤K t K t K represents the total number of targets. n For task area A Ta (n) is the total number of mission targets;

[0035] The location of the camera mounted on the drone is obtained in real time based on GPS. c When the camera position p is detected c Located in mission area A Ta (n) A hovering drone can acquire an enhanced image set I′ based on a progressive adjustment strategy. n And shooting external references E n ;

[0036] A tracking-transformation deflection algorithm is used for coordinate tracking transformation to obtain the deflection corresponding to each task target, and the deflection of the UAV task Ta is statistically generated. u (n) corresponds to the deflection sequence de n .

[0037] Furthermore, an enhanced image set I′ is obtained based on a stepwise adjustment strategy. n And shooting external references E n The specific steps include the following:

[0038] Obtain the actual horizontal field of view of the camera and actual vertical field of view Set the initial shooting angle of the camera, and set the horizontal unit adjustment angle based on the number of adjustments M for the horizontal field of view and the number of adjustments M for the vertical field of view. Adjusting the angle with vertical units

[0039] Adjust the angle according to the horizontal unit Adjusting the angle with vertical units Delineating the actual horizontal field of vision and actual vertical field of view Generate the shooting angle matrix X, where X i,j Let represent the (i-1)·(M+1)+j-th shooting angle in the shooting angle matrix X, with the specific formula as follows:

[0040]

[0041] Among them, X i,j The camera shooting angle is determined by adjusting the camera i-1 times in the horizontal field of view and j-1 times in the vertical field of view;

[0042] Images are acquired from each shooting angle and placed into the initial image set I. n,0 The extrinsic parameters for each shooting angle are determined using Eulerian geometry and then set into the initial extrinsic parameter set E. n,0 ;

[0043] A visual classification network combined with the optimal matching principle is used to classify the initial image set I. n,0 and initial shooting external parameters E n,0 The image set I′ was filtered to generate an enhanced image set. n And shooting external references E n .

[0044] Furthermore, the extrinsic parameters e of the m-th shooting angle are determined using Eulerian geometry. n,0 (m) includes the following specific steps:

[0045] The camera's X-axis deflection angle relative to the world coordinate system at the m-th shooting angle is measured using an inertial sensor. Y-axis deflection angle and Z-axis deflection angle

[0046] Based on X-axis deflection angle Y-axis deflection angle and Z-axis deflection angle Obtain the rotation matrix of the camera at the m-th shooting angle. Rotation matrix The rotation matrix represents the rotation of the camera coordinate system relative to the world coordinate system. The specific calculation formula is as follows:

[0047]

[0048] in, and These represent the X-axis rotation matrix, Y-axis rotation matrix, and Z-axis rotation matrix of the camera coordinate system about the world coordinate system, respectively.

[0049] Get camera position p c Position the camera p c The transpose of is used as a translation vector Translation vector This indicates the translation of the camera coordinate system origin about the world coordinate system origin;

[0050] Combined rotation matrix Translation vector To construct shooting external parameters Shooting external references e n,0 (m) determines the transformation relationship from the world coordinate system to the camera coordinate system.

[0051] Furthermore, a visual classification network combined with the optimal matching principle is used to generate an enhanced image set I′. n And shooting external references E n The specific steps include the following:

[0052] The dimension to be obtained is L c,w ×L c,h Image i n,0 (m), m=1,2,…,(M+1) 2 L c,w and L c,h These are the camera's pixel width and pixel height, respectively.

[0053] Image i is processed by two repeated deep mining groups. n,0 (m) Generate the feature matrix z n,0 (m), the deep mining group includes a convolutional layer with a kernel size of 3 and a stride of 1, ReLU, and a pooling layer with a kernel size of 2 and a stride of 1.

[0054] The characteristic matrix z n,0 (m) is convolved with L1 convolution kernels respectively to generate a multi-channel feature matrix Z. n,0 (m), the multi-channel feature matrix Z n,0 Each pixel in (m) is proportionally expanded to become an anchor point, generating a multi-channel anchor point matrix Z′. n,0 (m);

[0055] The multi-channel anchor matrix Z′ n,0 (m) After convolution and reshape, the dimensions are reconstructed, and the positive score of each anchor point is generated using Sigmoid;

[0056] The L2 anchors with the highest positive scores are selected and fixed. Non-maximum suppression is applied to the remaining anchors. A linear layer is used to generate an image i. n,0 (m) Enhanced image i′ of the same dimension n,0 (m) and replace the initial image set I n,0 Image i in n,0 (m);

[0057] Image i′ is enhanced using two fully connected layers and Softmax processing. n,0 (m) Generate the classification score vector g n,m ;

[0058] When generating all enhanced images and their corresponding classification score vectors, the initial enhanced image set I′ is obtained. n,0 And classification score matrix G n From the target number set k Ta Extract the individual task target numbers sequentially from (n), and query the classification score matrix G.n The enhanced image with the highest probability and its corresponding shooting extrinsic parameters are labeled as the single-task target number;

[0059] Remove the initial enhanced image set I′ n,0 and initial shooting external parameters E n,0 The unlabeled enhanced images and imaging extrinsic parameters are renumbered according to the mission target number to generate an enhanced image set I′. n And shooting external references E n .

[0060] Furthermore, a tracking-conversion deflection algorithm is used to statistically generate a deflection sequence. n The specific steps include the following:

[0061] Based on the kth n Target number k t +k n -1, from the enhanced image set I′ n And shooting external references E n Obtain the corresponding enhanced image i′ from each. n (k t +k n -1) and shooting external parameters e n (k t +k n -1), 1≤k n ≤K n K n For task area A Ta (n) is the total number of mission targets;

[0062] In enhanced image i′ n (k t +k n -1) Establish L c,w ×L c,h A pixel coordinate system of 3D is established, and the pixel coordinates p of the target are determined by Hough transform. c,t,p (k t +k n -1), where L c,w and L c,h These are the camera's pixel width and pixel height, respectively.

[0063] Determine the camera's shooting parameters in c Shooting internal references in c This determines the transformation relationship from the camera coordinate system to the two-dimensional pixel coordinate system, and the shooting intrinsic parameters. c The specific formula is as follows:

[0064]

[0065] Among them, f c The focal length of the camera;

[0066] Based on the camera tracking and calibration principle, the target pixel coordinates p c,t,p (k t +k n -1) Relative to the world coordinates of the mission target The following transformation relationship exists:

[0067]

[0068] The camera tracking calibration principle shows that any world coordinate is converted into camera coordinates through shooting extrinsic parameters, and then converted into pixel coordinates through shooting intrinsic parameters. This can be combined with the target pixel coordinates p. c,t,p (k t +k n -1) Reverse the world coordinates of the mission target

[0069] Obtain the target installation location p t (k t +k n -1), calculate and obtain the target number k t +k n -1 corresponds to the deflection de(k) of the target. t +k n -1);

[0070] Statistical task target number set k Ta (n) in K n The deflections corresponding to each task target are arranged in order according to the target number, generating a deflection sequence. n .

[0071] A device for detecting the deflection of long-span bridges based on UAV vision is used to perform a method for detecting the deflection of long-span bridges based on UAV vision. The device includes a UAV, a base plate fixed to the bottom of the UAV, a rotating shaft connected to the lower end of the base plate, a connecting shaft fixed to the bottom of the rotating shaft, the rotating shaft being used to control the horizontal field of view rotation of the bottom connecting shaft, and a camera fixed to the top of the connecting shaft via a rotatable bearing, the bearing being used to control the vertical field of view rotation of the camera.

[0072] It is worth noting that the target is a passive marker, which is fixed to the main beam with special glue or bolts. It is used to mark the points on the main beam where deflection testing is required. The material and marking style can be determined according to the external environment of the main beam.

[0073] Compared with the prior art, the significant advantages of this invention are:

[0074] 1. The reverse estimation method combined with the repetition merging strategy is designed to pre-plan and generate the UAV mission sequence. Considering the actual horizontal and vertical field of view of the camera, the shooting area is reversed based on the target installation position. The original fixed-point shooting is changed to shooting at any point within the shooting area. The repetition merging strategy is used to merge highly overlapping shooting areas, which greatly improves the shooting efficiency of UAV.

[0075] 2. Design a step-by-step adjustment strategy. After the UAV arrives at the mission area, it can hover at any time and acquire multiple images by gradually adjusting the camera shooting angle. A visual classification network is used to adaptively filter images without mission targets and generate enhanced images that highlight mission targets. At the same time, the shooting extrinsic parameters corresponding to the shooting angle are acquired synchronously based on the Eulerian geometry method.

[0076] 3. A tracking-to-deflection algorithm was designed. Combining the camera tracking calibration principle, the pixel coordinates of the target in the enhanced image were adaptively determined. Based on the shooting extrinsic parameters and fixed shooting intrinsic parameters, the pixel coordinates of the target were converted into the target world coordinates. The deflection was obtained by comparing the target installation coordinates. This enabled batch deflection detection at any position and angle within the shooting area, greatly improving detection efficiency and reducing the difficulty of equipment deployment. Attached Figure Description

[0077] Figure 1 A flowchart of a method for detecting the deflection of a large-span steel structure main beam based on UAV vision;

[0078] Figure 2 This is a flowchart of the duplicate merging strategy in this invention;

[0079] Figure 3 This is a flowchart of the visual classification network combined with the optimal matching principle in this invention;

[0080] Figure 4 This is a diagram of a long-span bridge deflection detection device based on UAV vision, as described in this invention. Detailed Implementation

[0081] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0082] Example 1

[0083] like Figure 1 As shown in the figure, a specific embodiment of the present invention discloses a method for detecting the deflection of long-span bridges based on UAV vision, comprising the following specific steps:

[0084] Based on the structural characteristics of the main beam, a target parameter sequence B is pre-constructed. t And install the target;

[0085] Using a reverse estimation method, considering the imaging parameters and the target installation position of a single target, the imaging area of ​​a single target is pre-determined and recorded in the imaging area sequence A. t In the middle, the shooting area sequence A is planned based on the repeated merging strategy. t Generate UAV mission sequence Ta u ;

[0086] Based on the UAV mission sequence Ta u The system executes a single UAV mission sequentially, moves to the mission area corresponding to the single UAV mission, and uses a visual classification-based tracking deflection detection method to generate a single deflection sequence corresponding to the single UAV mission.

[0087] Statistical analysis of unmanned aerial vehicle (UAV) mission sequences Ta u The total deflection sequence de = {de1,…,de} is generated from the single deflection sequence corresponding to each individual UAV mission. N}, where N is the total number of drone missions, and the main beam deflection detection is now complete.

[0088] Furthermore, the structural characteristics of the main girder form the basis for constructing the target parameter sequence. Due to the large span and complex stress of large-span main girders, professional bridge engineers are required to determine the total number of targets K based on the bridge's structural form, span, and support method when constructing the target parameter sequence. t Target installation position p of a single target t Target parameter sequence B t It includes K t Target parameter pairs for each target, target parameter sequence B t The specific format is as follows:

[0089]

[0090] Among them, b t (k t ) is the kth t The target parameter pairs for each target, k t p is the target number. t (k t ) represents the k-th element in the world coordinate system. t The target installation position of each target, the world coordinate system is the three-dimensional rectangular coordinate system of the real world, which has been established in advance before constructing the target parameter sequence.

[0091] Furthermore, the reverse estimation method is used to pre-determine and record the shooting area of ​​a single target, which includes the following specific steps:

[0092] Get the kth t The target installation position p of each target t (k t )=[x t (kt ),y t (k t ),z t (k t [)] and the drone's photography parameters, including the focal length f of the camera mounted on the drone. c Minimum shooting distance of the camera Maximum shooting distance of the camera Camera maximum horizontal adjustment angle Maximum vertical adjustment angle of the camera The horizontal width of the camera sensor is w c Vertical height h of the camera sensor c ;

[0093] Determine the horizontal field of view θ of the camera c,w and vertical field of view θ c,h Horizontal field of view θ c,w and vertical field of view θ c,h These determine the maximum shooting angles in the horizontal and vertical planes when the camera is fixed in one direction, respectively; the horizontal field of view angle θ c,w and vertical field of view θ c,h The specific calculation formula is as follows:

[0094]

[0095] Among them, w c and h c These are the horizontal width and vertical height of the camera sensor, respectively.

[0096] Based on the camera's maximum horizontal adjustment angle and maximum vertical adjustment angle This can further expand the camera's horizontal and vertical field of view. After expansion, the camera's actual horizontal field of view range is: The actual vertical field of view is in, and These are the leftmost and rightmost horizontal fields of vision, respectively. and These are the lowest vertical field of view and the highest vertical field of view, respectively.

[0097] With the kth t The target installation position p of each target t (k t )=[x t (k t ),y t (k t ),z t (k t With )] as the center of the sphere, and the kth sphere as the center of the sphere. tThe plane containing each target serves as the interface. On the right side of the interface, at the minimum shooting distance... and maximum shooting distance Construct a minimum and maximum hemispheres within the world coordinate system with radius , then the spatial region inside the maximum hemisphere and outside the minimum hemisphere is considered as the k-th region. t The initial shooting area A0(k) of the target t );

[0098] Obtain the minimum movement distance of the drone With minimum movement distance Generate a mesh to divide the initial shooting area To obtain the initial shooting area Several grid points within;

[0099] Calculate the initial shooting area Each grid point within the target installation position p t (k t The azimuth and elevation angles are used to determine whether the grid points meet the field of view conditions. Specifically, the field of view conditions refer to the azimuth angle of the grid points being within the actual horizontal field of view of the camera. Within this range, the pitch angle of the grid points is also within the actual vertical field of view. Inside;

[0100] The spatial region encompassed by all grid points that satisfy the view conditions is taken as the k-th grid point. t Feasible shooting area for each target The feasible shooting area is cut off by planes parallel to the three coordinate axes in the world coordinate system. The irregular parts are obtained by intruding within the feasible shooting area. The largest cuboid region, and as the kth... t The shooting area A of the target t (k t Recorded in the shooting area sequence A t middle.

[0101] like Figure 2 As shown, further, the UAV mission sequence Ta is generated based on the repetition merging strategy. u The specific steps include the following:

[0102] Obtain the shooting area sequence A t ={A t (1),…,A t (K t )}, Shooting area sequence A t K was recorded t The shooting area of ​​the target, K t The total number of targets;

[0103] Perform an empty sequence decision to determine the UAV mission sequence Ta. u Is it an empty sequence? If the UAV mission sequence Ta u If the sequence is empty, then both the maximum task number and the maximum target number are 0. Increment both the maximum task number and the maximum target number by 1, and then proceed to the component region set expansion. If the UAV task sequence Ta... u Not an empty sequence, confirm the UAV mission sequence Ta u The highest task number in;

[0104] Perform a target exhaustive test to obtain the maximum target number in the UAV mission corresponding to the maximum mission number, and determine whether the maximum target number is equal to the total number of targets K. t If the maximum target number is equal to the total number of targets K t If the repeated merging strategy is immediately stopped, the drone mission sequence Ta... u Planning complete; if the maximum target number is less than the total number of targets K. t Increment both the maximum mission number and the maximum target number by 1.

[0105] Execute component region set expansion, placing the shooting region corresponding to the largest target number into the component region set corresponding to the largest task number;

[0106] Perform extended valid decision-making to determine whether the ratio of the volume of the overlapping region of all shooting regions in the component region set to the volume of any single shooting region is greater than or equal to the overlap ratio λ. A In this embodiment, the overlap ratio λ A =0.95;

[0107] If the ratio of the volume of the overlapping region to the volume of any single shooting region is less than the overlap ratio λ A If the shooting area corresponding to the largest target number in the component area set is removed, the overlapping area of ​​all shooting areas in the component area set is obtained again, the largest inscribed cuboid area in the overlapping area is taken as the task area corresponding to the largest task number, all target numbers in the component area set are extracted, the task target number set corresponding to the largest task number is constructed, the task area and the task target number set are packaged to generate the UAV task corresponding to the largest task number, the largest task number is incremented by 1 and the target exhaustive judgment is executed.

[0108] If the ratio of the volume of the overlapping region to the volume of any single shooting region is greater than or equal to the overlap ratio λ A Increment the maximum target number by 1 and jump to the component region set expansion.

[0109] Furthermore, the method of generating a single deflection sequence corresponding to a single UAV mission using a vision-based classification-based tracking deflection detection method includes the following specific steps:

[0110] Assume the drone mission sequence Ta u The total number of UAV missions is N, which is the number of UAV mission sequences Ta. u ={Ta u (1),…,Ta u (N)}, and based on the UAV mission sequence Ta u The order in which the nth drone mission Ta should be executed at this point is as follows. u (n), n = 1, 2, ..., N;

[0111] Confirm the nth drone mission Ta u Task region A in (n) Ta (n) and the target number set k Ta (n), assuming the target number set k Ta (n)={k t ,…,k t +K n -1},n≤k t ≤k t +K n -1≤K t K t K represents the total number of targets. n For task area A Ta If the total number of task targets is (n), then the task area A Ta (n) represents the shooting area A. t (k t To shooting area A t (k t +K n The largest inscribed cuboid region of the overlapping region of -1);

[0112] The location of the camera mounted on the drone is obtained in real time based on GPS. c =[x c ,y c ,z c When the camera position p is detected c Located in mission area A Ta (n) The drone can hover at any time within the specified range, and acquire the mission area A based on a gradually adjusted strategy. Ta Enhanced image set I′ of (n) n And shooting external references E n ;

[0113] The tracking-conversion deflection algorithm is used to consider the enhancement image set I′ n The pixel coordinates of the target in the enhanced image, combined with the imaging extrinsic set E n The coordinate tracking transformation is performed on the extrinsic parameters of the corresponding mission target to obtain the deflection corresponding to each mission target, and the nth UAV mission Ta is generated statistically. u(n) corresponds to the deflection sequence de n .

[0114] Furthermore, based on the gradual adjustment strategy, the task area A is obtained. Ta Enhanced image set I′ of (n) n And shooting external references E n The specific steps include the following:

[0115] Obtain the actual horizontal field of view of the camera and actual vertical field of view set up Set the camera's horizontal adjustment unit angle to the initial shooting angle when the camera is not adjusted. Adjusting the angle with vertical units The specific settings are as follows:

[0116]

[0117] Where M represents the number of times the horizontal field of view is adjusted and the number of times the vertical field of view is adjusted;

[0118] Adjust the angle according to the horizontal unit Adjusting the angle with vertical units The actual horizontal field of vision can be increased and actual vertical field of view Each of the M+1 horizontal shooting angles and M+1 vertical shooting angles is divided, and these are combined to generate a (M+1)×(M+1) dimensional shooting angle matrix X, where X... i,j Let represent the (i-1)·(M+1)+j-th shooting angle in the shooting angle matrix X, with the specific formula as follows:

[0119]

[0120] In essence, X i,j This refers to the camera shooting angle determined by adjusting the camera i-1 times in the horizontal field of view and j-1 times in the vertical field of view;

[0121] At each shooting angle of the shooting angle matrix X, images are acquired by the camera and placed into the initial image set I. n,0 ={i n,0 (1),…,i n,0 ((M+1) 2 The extrinsic parameters for each shooting angle are determined using Eulerian geometry and then set into the initial extrinsic parameter set E. n,0 ={e n,0 (1),…,e n,0 ((M+1) 2 )}, where i n,0 (m) and en,0 (m) represents the image acquired from the m-th shooting angle and the corresponding shooting parameters;

[0122] A visual classification network combined with the optimal matching principle is used to classify the initial image set I. n,0 and initial shooting external parameters E n,0 Filter and generate task area A Ta Enhanced image set I′ of (n) n And shooting external references E n .

[0123] Furthermore, the task region A was confirmed using the Eulerian geometry method. Ta The shooting extrinsic parameter e of the m-th shooting angle in (n) n,0 (m) includes the following specific steps:

[0124] The camera's X-axis deflection angle relative to the world coordinate system at the m-th shooting angle is measured using the camera's built-in inertial sensor. Y-axis deflection angle and Z-axis deflection angle

[0125] Obtain the rotation matrix of the camera at the m-th shooting angle. Rotation matrix The rotation matrix represents the rotation of the camera coordinate system relative to the world coordinate system. The specific calculation formula is as follows:

[0126]

[0127] in, and Let X, Y, and Z represent the rotation matrices of the camera coordinate system about the world coordinate system, respectively, and C and S represent the cosine and sine functions, respectively.

[0128] Obtain the camera position p in the world coordinate system when the drone is hovering. c =[x c ,y c ,z c ], position the camera p c The transpose of A is used as task region A Ta Translation vector of (n) Translation vector This indicates the translation of the camera coordinate system origin about the world coordinate system origin;

[0129] Combined rotation matrix Translation vector Construct the shooting extrinsic parameters e for the m-th shooting angle n,0 (m), shooting external parameters en,0 (m) determines the transformation relationship from the world coordinate system to the camera coordinate system. The camera coordinate system refers to a three-dimensional spatial coordinate system constructed with the camera as the origin, the vertical axis perpendicular to the camera lens as the Z-axis, the horizontal axis parallel to the camera lens as the X-axis, and the vertical axis parallel to the camera lens as the Y-axis. (e) represents the external parameters of the image. n,0 The specific formula for (m) is:

[0130] like Figure 3 As shown, further, a visual classification network combined with the optimal matching principle is used to generate task region A. Ta Enhanced image set I′ of (n) n And shooting external references E n The specific steps include the following:

[0131] Get the initial image set I n,0 Image i obtained from the m-th shooting angle n,0 (m), m=1,2,…,(M+1) 2 Image i n,0 (m) dimension is fixed as L c,w ×L c,h , where L c,w and L c,h These are the camera's pixel width and pixel height, respectively.

[0132] Image i was extracted through two repeated deep mining groups. n,0 The characteristic matrix z corresponding to (m) n,0 (m), where the deep mining group includes a convolutional layer with a kernel size of 3 and a stride of 1, a ReLU, and a pooling layer with a kernel size of 2 and a stride of 1. Since the convolution dimension remains unchanged and the pooling dimension is halved, the feature matrix z n,0 The dimension of (m) is (L) c,w / 4)×(L c,h / 4);

[0133] Feature matrix z n,0 (m) contains a large number of background features and a small number of target features, in order to fully cover the feature matrix z n,0 The target features in (m) will be used to transform the feature matrix z. n,0 (m) is convolved with L1 convolutional kernels to generate a dimension of (L... c,w / 4)×(L c,h The multi-channel feature matrix Z of ( / 4)×L1 n,0 (m), and the multi-channel feature matrix Z n,0 Each pixel in (m) is expanded into 9 anchor points, generating a dimension of (L). c,w / 4)×(L c,hThe multi-channel anchor point matrix Z′ is 4)×(L1×9) n,0 (m);

[0134] The multi-channel anchor matrix Z′ n,0 (m) after convolution transformation has a dimension of (L) c,w / 4)×(L c,h / 4)×9, and reshape the dimensions to (3·L) c,w / 4)×(3·L c,h / 4), use Sigmoid to generate a positive score for each anchor point, where the positive score represents the probability that the anchor point belongs to the target feature;

[0135] The L2 anchors with the highest positive scores are selected and fixed. Non-maximum suppression is applied to the remaining anchors, and the dimensions are further transformed to match those of image i through a linear layer. n,0 (m) Same L c,w ×L c,h Generate enhanced image i′ n,0 (m) and replace the initial image set I n,0 Image i in n,0 (m);

[0136] Image i′ is enhanced using two fully connected layers and Softmax processing. n,0 (m) Generates (K) t +1)×1 dimensional classification score vector g n,m =[g n,m (0),…,g n,m (K t )], where g n,m (0) is to enhance image i′ n,0 (m) excludes the probability of any target, g n,m (k t To enhance image i′ n,0 (m) includes the kth t The probability of a target, 1≤k t ≤K t K t The total number of targets;

[0137] Get (M+1) 2 The classification score vectors corresponding to each enhanced image are integrated into a classification score matrix G. n At this time, the initial image set I n,0 All images in the initial image set I have been replaced with their corresponding enhanced images. n,0 Transformed into the initial enhanced image set I′ n,0 From the target number set k Ta Extract the individual task target numbers sequentially from (n), and query the classification score matrix G.n The enhanced image with the highest probability and its corresponding shooting extrinsic parameters are labeled with a single task target number, until the task target number set k is reached. Ta All task target numbers in (n) were extracted;

[0138] Remove the initial enhanced image set I′ n,0 and initial shooting external parameters E n,0 The unlabeled enhanced images and imaging extrinsic parameters in the initial enhanced image set I′ are used to determine the target number of the target. n,0 and initial shooting external parameters E n,0 The remaining enhanced images and captured extrinsic parameters are renumbered to generate task region A. Ta Enhanced image set I′ of (n) n ={i′ n (k t ),…,i′ n (k t +K n -1)} and shooting external parameter set E n ={e n (k t ),…,e n (k t +K n -1)}.

[0139] Furthermore, the tracking-conversion deflection algorithm is used to statistically generate the nth UAV mission Ta. u (n) corresponds to the deflection sequence de n The specific steps include the following:

[0140] For the task target number set k Ta (n) of the kth n Target number k t +k n -1, from the enhanced image set I′ n And shooting external references E n Obtain the corresponding enhanced image i′ from each. n (k t +k n -1) and shooting external parameters e n (k t +k n -1), 1≤k n ≤K n K n For task area A Ta (n) is the total number of mission targets;

[0141] In pixels, in the enhanced image i′ n (k t +k n-1) Establish L c,w ×L c,h A 3D pixel coordinate system, where L c,w and L c,h These are the camera's pixel width and pixel height, respectively.

[0142] Determining the enhanced image i′ using Hough transform n (k t +k n -1) The target number is k t +k n -1 target pixel coordinate p c,t,p (k t +k n -1)=[x c,t,p (k t +k n -1),y c,t,p (k t +k n -1)];

[0143] Determine the camera's shooting parameters in c Shooting internal references in c This determines the transformation relationship from the camera coordinate system to the two-dimensional pixel coordinate system, and the shooting intrinsic parameters. c The specific formula is as follows:

[0144]

[0145] Among them, f c This refers to the camera's focal length; therefore, when the camera model is fixed, the camera's shooting intrinsics are... c Also fixed;

[0146] Based on the camera tracking calibration principle, the target pixel coordinates p in the pixel coordinate system are... c,t,p (k t +k n -1) Relative to the world coordinates of the mission target in the world coordinate system The following transformation relationship exists:

[0147]

[0148] The camera tracking calibration principle states that any world coordinate in the world coordinate system is transformed into camera coordinates in the camera coordinate system through shooting extrinsic parameters, and then transformed into pixel coordinates in the pixel coordinate system through shooting intrinsic parameters. Therefore, it can be combined with the target pixel coordinates p. c,t,p (k t +k n -1)=[x c,t,p (k t +k n -1),yc,t,p (k t +k n -1)] Deducing the world coordinates of the mission target

[0149] Based on target number k t +k n -1 Obtain the corresponding target installation location p t (k t +k n -1), calculate and obtain the target number k t +k n -1 deflection corresponding to the target Among them, y t (k t +k n -1) and The target installation positions are p respectively. t (k t +k n -1) Y-axis value and world coordinates of the mission target Y-axis value;

[0150] Statistical task target number set k Ta (n) K n The deflections corresponding to each task target are arranged in order according to the target number, generating a deflection sequence. n ={de(k t ),…,de(k t +K n -1)}.

[0151] Example 2

[0152] like Figure 4 As shown, the present invention also discloses a deflection detection device for long-span bridges based on UAV vision, used to perform a deflection detection method for long-span steel main beams based on UAV vision proposed in Embodiment 1. The device includes a UAV 1, a UAV base plate 2 fixed to the bottom of the UAV 1, a rotating shaft 3 connected to the lower end of the UAV base plate 2, a connecting shaft 4 fixed to the bottom of the rotating shaft 3, the rotating shaft 3 being used to control the horizontal field of view rotation of the bottom connecting shaft 4, and a camera 6 fixed to the top of the connecting shaft 4 via a rotatable bearing 5, the bearing 5 being used to control the vertical field of view rotation of the camera 6.

[0153] It is worth noting that the target is a passive marker, fixed to the main beam with special glue or bolts. It is used to mark the points on the main beam where deflection testing is required. The target does not require any electronic components, and the manufacturing material and marking style can be determined according to the external environment of the main beam to ensure the target's durability and clear visibility. Therefore, the target does not need to be replaced frequently and can be used for a long time after a single installation. Moreover, the passive design makes the target more adaptable to various external environments. In this embodiment, the target used is a square iron sheet with a white paint coating. The target number is marked in black paint in the center of the target.

[0154] This invention discloses a device and method for detecting the deflection of long-span bridges based on UAV vision. Based on the structural characteristics of the main beam, a target parameter sequence is pre-constructed and targets are installed. Using a reverse estimation method, considering the photographing parameters and the target position of a single target, the shooting area of ​​a single target is pre-determined. A UAV task sequence is planned and generated based on a repetitive merging strategy to improve deflection detection efficiency. Single UAV tasks are executed sequentially according to the UAV task sequence, moving to the task area corresponding to each single UAV task and generating a single deflection sequence corresponding to that single UAV task using a vision-based classification tracking deflection detection method. The total deflection sequence is generated by statistically analyzing the deflection sequences corresponding to all UAV tasks in the UAV task sequence. This achieves automated main beam deflection detection without fixed-point measurement, solving the problems of difficult equipment deployment and low detection efficiency caused by fixed-point measurement in existing technologies.

[0155] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting the deflection of long-span bridges based on UAV vision, characterized in that, The specific steps include the following: Based on the structural characteristics of the main beam, a target parameter sequence was constructed and the target was installed. The reverse estimation method is used to determine the shooting area of ​​a single target to generate a shooting area sequence, and the UAV mission sequence is planned and generated based on the repetition merging strategy. According to the UAV mission sequence, a single UAV mission is executed sequentially. The mission area is moved to the mission area and the mission target number set in the single UAV mission is confirmed by the tracking deflection detection method based on visual classification. When the camera position is in the mission area, the UAV is hovered. The enhanced image set and shooting extrinsic parameter set are obtained based on the stepwise adjustment strategy. The coordinate tracking transformation algorithm is used to perform coordinate tracking transformation to obtain the deflection corresponding to each mission target in the single UAV mission. The single deflection sequence corresponding to the single UAV mission is statistically generated. The total deflection sequence is generated by statistically analyzing the single deflection sequences corresponding to all individual UAV missions in the aforementioned UAV mission sequence, thus completing the main beam deflection detection. The reverse estimation method includes: obtaining the target installation position of a single target and the UAV's photography parameters; determining the horizontal and vertical field of view angles of the camera based on the horizontal width and vertical height of the camera sensor; expanding the horizontal and vertical field of view of the camera based on the maximum horizontal and vertical adjustment angles of the camera; obtaining the actual horizontal and vertical field of view ranges of the camera; constructing a minimum and maximum hemisphere on the right side of the plane where the single target is located, with the target installation position as the center, to determine the initial shooting area; dividing the initial shooting area based on the minimum movement distance of the UAV to obtain grid points; and obtaining the shooting area of ​​the single target based on the field of view conditions and recording it in the shooting area sequence. The repeated merging strategy includes: obtaining the shooting area sequence and performing an empty sequence judgment; determining whether the UAV mission sequence is an empty sequence; if it is an empty sequence, setting the maximum mission number and the maximum target number to 1, and jumping to execute the component region set expansion; if it is not an empty sequence, confirming the maximum mission number in the UAV mission sequence, performing a target exhaustion judgment, obtaining the maximum target number corresponding to the maximum mission number, and determining whether the maximum target number is equal to the total number of targets; if it is equal, stopping the repeated merging strategy; if it is less, incrementing the maximum mission number and the maximum target number by 1; performing component region set expansion, placing the shooting area corresponding to the maximum target number into the component region set corresponding to the maximum mission number; performing an expansion validity judgment, determining whether the ratio of the overlapping area volume of all shooting areas in the component region set to the volume of any single shooting area is greater than or equal to the overlap ratio; if it is less than the overlap ratio, removing the shooting area corresponding to the maximum target number in the component region set, generating the UAV mission corresponding to the maximum mission number, incrementing the maximum mission number by 1, and jumping to execute the target exhaustion judgment; if it is greater than or equal to the overlap ratio, incrementing the maximum target number by 1 and jumping to component region set expansion; The stepwise adjustment strategy includes: acquiring the actual horizontal field of view and the actual vertical field of view; setting an initial shooting angle, a horizontal unit adjustment angle, and a vertical unit adjustment angle; dividing the actual horizontal field of view and the actual vertical field of view into several shooting angles based on the horizontal unit adjustment angle and the vertical unit adjustment angle; acquiring an image at each shooting angle and placing it into an initial image set; confirming the shooting extrinsic parameters of each shooting angle using Eulerian geometry and placing them into an initial shooting extrinsic parameter set; using a visual classification network combined with the optimal matching principle to filter the initial image set and the initial shooting extrinsic parameter set; and generating an enhanced image set and shooting extrinsic parameter set.

2. The method for detecting the deflection of long-span bridges based on UAV vision as described in claim 1, characterized in that, The Euler geometry method for determining the extrinsic parameters of a single shooting angle includes the following specific steps: Obtain the camera's position at a single shooting angle. Axis deflection angle, Shaft deflection angle and Axis deflection angle; Based on the above Shaft deflection angle, the aforementioned Shaft deflection angle and the The rotation matrix of a single shooting angle is obtained by the axis deflection angle, and the rotation matrix represents the rotation of the camera coordinate system relative to the world coordinate system; The transpose of the camera position is used as the translation vector, which indicates the translation of the origin of the camera coordinate system about the origin of the world coordinate system. The rotation matrix and the translation vector are combined to construct the extrinsic parameters for shooting at a single shooting angle, which determine the transformation relationship from the world coordinate system to the camera coordinate system.

3. The method for detecting the deflection of long-span bridges based on UAV vision as described in claim 1, characterized in that, The visual classification network generates enhanced image sets and imaging extrinsic parameter sets by combining the optimal matching principle, including the following specific steps: A single image is acquired and processed through multiple deep mining groups to generate a corresponding feature matrix. The deep mining groups include convolutional layers, ReLU, and pooling layers. The feature matrix is ​​convolved with several convolution kernels, and each pixel in the convolution result is expanded into an anchor point according to a fixed ratio to generate a multi-channel anchor point matrix. The multi-channel anchor point matrix is ​​reconstructed in dimension through convolution and reshape, and a positive score for each anchor point is generated based on Sigmoid. Select several anchor points with the highest positive scores and fix them, perform non-maximum suppression on the remaining anchor points, and generate a single enhanced image of the same dimension as the single image through a linear layer and replace the single image. The single enhanced image is processed by two fully connected layers and Softmax to generate a corresponding classification score vector. All individual enhanced images and their corresponding classification score vectors are obtained to generate an initial enhanced image set and a classification score matrix. The classification score matrix is ​​queried according to the single task target number, and the enhanced image with the highest probability and its corresponding shooting extrinsic parameters are marked. Unlabeled enhanced images and imaging extrinsic parameters are removed, and the remaining enhanced images and imaging extrinsic parameters are renumbered to generate enhanced image sets and imaging extrinsic parameter sets.

4. The method for detecting the deflection of long-span bridges based on UAV vision as described in claim 1, characterized in that, The tracking conversion deflection algorithm statistically generates a single deflection sequence corresponding to a single UAV mission, including the following specific steps: A single enhanced image and a single imaging extrinsic parameter are obtained from the enhanced image set and the imaging extrinsic parameter set, respectively, based on a single target number; A pixel coordinate system is established in the single enhanced image, and the pixel coordinates of the target are determined by Hough transform. The shooting intrinsic parameters are determined based on the camera's focal length, and these shooting intrinsic parameters determine the transformation relationship from the camera coordinate system to the pixel coordinate system. The world coordinates of the target are derived based on the camera tracking calibration principle and the pixel coordinates of the target. The camera tracking calibration principle shows that the world coordinates are converted into camera coordinates through shooting extrinsic parameters, and then converted into pixel coordinates through shooting intrinsic parameters. Obtain the target installation location, calculate the deflection corresponding to a single mission target, and generate a single deflection sequence corresponding to a single UAV mission by statistically analyzing the deflections corresponding to all mission targets in the mission target number set.

5. The method for detecting the deflection of long-span bridges based on UAV vision as described in claim 1, characterized in that, The process of obtaining the shooting area of ​​a single target based on field of view conditions includes the following specific steps: Calculate the azimuth and elevation angles of a single grid point within the initial shooting area relative to the target installation position; Determine whether a single grid point meets the field of view conditions, wherein the field of view conditions refer to the azimuth angle of the single grid point being within the actual horizontal field of view and the pitch angle of the single grid point being within the actual vertical field of view. The spatial region encompassed by all grid points that meet the aforementioned field of view conditions is considered as the feasible shooting area for a single target. Obtain the largest cuboid region inscribed within the feasible shooting area, and use it as the shooting area for a single target.

6. A deflection detection device for long-span bridges based on UAV vision, used to execute a deflection detection method for long-span bridges based on UAV vision as described in any one of claims 1-5, comprising a UAV, a UAV base plate fixed to the bottom of the UAV, a rotating shaft connected to the lower end of the UAV base plate, a connecting shaft fixed to the bottom of the rotating shaft, the rotating shaft being used to control the horizontal field of view rotation of the bottom connecting shaft, and a camera fixed to the top of the connecting shaft via a rotatable bearing, the bearing being used to control the vertical field of view rotation of the camera.

Images