Large-span bridge deflection detection device and method based on unmanned aerial vehicle vision

Abstract

The invention discloses a large-span bridge deflection detection device and method based on unmanned aerial vehicle vision, and relates to the technical field of bridge detection, a target parameter sequence is pre-constructed based on the structural characteristics of a main beam, targets are installed, a reverse presumption method is adopted, photographing parameters and the target position of a single target are considered, and the deflection of a large-span bridge is detected. The method comprises the following steps: pre-determining a shooting area of a single target, planning and generating an unmanned aerial vehicle task sequence based on a repetition combination strategy, improving deflection detection efficiency, and sequentially executing a single unmanned aerial vehicle task according to the unmanned aerial vehicle task sequence. And going to a task area corresponding to a single unmanned aerial vehicle task, generating a single deflection sequence corresponding to the single unmanned aerial vehicle task by adopting a tracking deflection detection method based on visual classification, and counting deflection sequences corresponding to all unmanned aerial vehicle tasks in the unmanned aerial vehicle task sequence to generate a total deflection sequence, so that non-fixed-point automatic main beam deflection detection is realized. The problems of difficulty in equipment layout, low detection efficiency and the like caused by fixed-point measurement in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and in particular to a long-span bridge deflection detection device and method based on unmanned aerial vehicle vision. Background Art

[0002] The deflection of the main beam refers to the magnitude of the longitudinal linear displacement of the centroid at a certain cross section of the beam in the direction perpendicular to the axis. Once the deflection of the main beam exceeds the allowable range, or cannot return to its original position after being subjected to external force, it can be judged that there is a safety hazard in the main beam. Therefore, the main beam deflection detection is an important task to evaluate the operating status of the bridge based on its structural characteristics and bearing capacity during the static load test of the bridge.

[0003] Traditional methods for measuring the deflection of main beams usually use instruments such as dial indicators and displacement meters for contact measurement. When a bridge crosses a river, highway, railway, or canyon, it is impossible to lay out brackets, so contact instruments will face the problem of being unable to be installed. In recent years, non-contact methods for measuring bridge deflection using laser technology and digital imaging technology have emerged.

[0004] The existing Chinese patent with authorization announcement number CN107588913B proposes a bridge deflection detection system and detection method, including multiple targets, an image acquisition device, a laser ranging module, a deflection calculation module and an operating unit. The image acquisition device and the deflection calculation module are both connected to the operating unit, and the image acquisition device and the laser ranging module are both connected to the deflection calculation module. The deflection values ​​of multiple points to be measured on a large-span bridge are measured through the horizontal rotation of the image acquisition device and the laser ranging module, which solves the problems of the prior art requiring multiple CCD cameras or laser rangefinders, low accuracy, high cost, inconvenient operation, and low practicality.

[0005] However, the prior art requires the additional setting of a calibration plate and the movement of the equipment to a fixed location each time an inspection is carried out to facilitate the calibration of the image acquisition device. In addition, the direction of the laser equipment needs to be precisely controlled during the deflection detection process to achieve target alignment. For the deflection detection of large-span main beams, this will bring about problems such as difficulty in equipment deployment, complex operation, high cost and reduced efficiency. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a long-span bridge deflection detection device and method based on drone vision, so as to provide an efficient non-fixed-point tracking deflection detection method.

[0007] The technical solution to achieve the purpose of the present invention is:

[0008] A long-span bridge deflection detection method based on drone vision includes the following specific steps:

[0009] Constructing target parameter sequence B based on the structural characteristics of the main beam t and installing the target;

[0010] The reverse inference method is used to combine the target installation position and shooting parameters to determine the shooting area of ​​a single target to generate the shooting area sequence A. t , based on the repeated merging strategy to plan the shooting area sequence A t Generate UAV mission sequence Ta u ;

[0011] According to the UAV mission sequence Ta u Execute a single UAV task sequentially, go to the task area corresponding to the single UAV task, and use the tracking deflection detection method based on visual classification to generate a single deflection sequence corresponding to the single UAV task;

[0012] Statistical UAV mission sequence Ta u The single deflection sequences corresponding to all single UAV tasks in generate the total deflection sequence de to complete the main beam deflection detection.

[0013] Furthermore, the structural characteristics of the main beam include the structural form, span and support method of the bridge, which are used to construct the target parameter sequence B t , target parameter sequence B t Including K t The target parameter pair of targets, K t is the total number of targets, target parameter sequence B t The specific form is as follows:

[0014]

[0015] Among them, b t (k t ) is the kth t The target parameter pairs of k targets, t is the target number, p t (k t ) is the kth t Target installation position for each target.

[0016] Furthermore, the reverse inference method is used to predetermine and record the shooting area of ​​a single target, including the following specific steps:

[0017] Get the kth t The target installation position p of the target t (k t ) and the drone’s photography parameters;

[0018] Based on the horizontal width w of the camera sensor c and vertical height h c Determine the camera's horizontal field of view angle θc,w and vertical viewing angle θ c,h ;

[0019] Based on the maximum horizontal adjustment angle of the camera and maximum vertical adjustment angle Expand the camera's horizontal and vertical fields of view to obtain the camera's actual horizontal field of view and actual vertical field of view in, and are the leftmost horizontal field of view and the rightmost horizontal field of view respectively. and They are the lowest vertical field of view and the highest vertical field of view respectively;

[0020] With target installation position p t (k t ) is the center of the sphere, at the kth t The right side of the plane where the targets are located is the minimum shooting distance and maximum shooting distance Construct the minimum hemisphere and the maximum hemisphere for the radius, and take the space area within the maximum hemisphere and outside the minimum hemisphere as the initial shooting area A0(k t );

[0021] Get the minimum moving distance of the drone Minimum moving distance Generate a grid to divide the initial shooting area To obtain several grid points;

[0022] Calculate the initial capture area Internal single grid point and target installation position p t (k t ) to determine whether a single grid point meets the field of view condition, and the spatial area contained in all grid points that meet the field of view condition is taken as the kth grid point. t The feasible shooting area of ​​​​the target

[0023] Get the inscribed area within the feasible shooting area The maximum rectangular area of ​​​​the kth t Shooting area A of targets t (k t ) recorded in the shooting area sequence A t middle.

[0024] Furthermore, the UAV task sequence Ta is generated based on repeated merging strategy planning u The specific steps include:

[0025] Get the shooting area sequence A t ;

[0026] Execute empty sequence judgment to determine the drone mission sequence Ta u Is it an empty sequence? If it is an empty sequence, change the maximum task number and the maximum target number from 0 to 1, jump to execute the component area set expansion, if it is not an empty sequence, confirm the drone task sequence Ta u The maximum task number in ;

[0027] Execute target exhaustion judgment, obtain the maximum target number corresponding to the maximum task number, and determine whether the maximum target number is equal to the total number of targets K t , if it is equal to the total number of targets K t , then stop the repeated merging strategy. If it is less than the total number of targets K t , let the maximum task number and the maximum target number increase by 1;

[0028] Execute component area set expansion, and place the shooting area corresponding to the maximum target number into the component area set corresponding to the maximum task number;

[0029] Perform extended validity judgment to determine whether the ratio of the overlapped area volume of all shooting areas in the component area set to the volume of any single shooting area is greater than or equal to the overlap ratio λ A ;

[0030] If it is less than the overlap ratio λ A , remove the shooting area corresponding to the maximum target number in the component area set, confirm the task area and task target number set corresponding to the maximum task number based on the removed component area set, generate the drone task corresponding to the maximum task number, add 1 to the maximum task number and jump to execute the target exhaustion judgment;

[0031] If it is greater than or equal to the overlap ratio λ A , increase the maximum target number by 1 and jump to the component area set expansion.

[0032] Furthermore, the tracking deflection detection method based on visual classification is used to generate a single deflection sequence corresponding to a single UAV task, which includes the following specific steps:

[0033] Assume that the nth drone mission Ta is about to be executed u (n), n = 1, 2, ..., N, N is the total number of UAV missions;

[0034] Confirm drone mission Ta u Task area A in (n) Ta (n) and the mission target number set k Ta (n), assuming that the task target number set k Ta (n) = {k t ,…,k t +K n-1}, n≤k t ≤k t +K n -1≤K t , K t is the total number of targets, K n For mission area A Ta (n) Total number of mission targets;

[0035] Get the camera position on the drone in real time based on GPS c , when the camera position p is detected c Located in Mission Area A Ta (n) The drone can hover within the range and obtain the enhanced image set I′ based on the step-by-step adjustment strategy. n and the shooting external parameter set E n ;

[0036] The tracking conversion deflection algorithm is used to perform coordinate tracking conversion, obtain the deflection corresponding to each mission target, and statistically generate the UAV mission Ta u (n) The corresponding deflection sequence de n .

[0037] Furthermore, based on the stepwise adjustment strategy, the enhanced image set I′ is obtained n and the shooting external parameter set E n The specific steps include:

[0038] Get the actual horizontal field of view of the camera and actual vertical field of view Set the initial shooting angle of the camera, and set the horizontal unit adjustment angle according to the horizontal field of view adjustment times M and the vertical field of view adjustment times M. and vertical units to adjust the angle

[0039] Adjust angle according to horizontal units and vertical units to adjust the angle Divide the actual horizontal field of view and actual vertical field of view Generate the shooting angle matrix X, where X i,j represents the (i-1)·(M+1)+jth shooting angle in the shooting angle matrix X. The specific formula is as follows:

[0040]

[0041] Among them, X i,j The camera shooting angle is determined by adjusting the camera i-1 times in horizontal field of view and j-1 times in vertical field of view;

[0042] At each shooting angle, images are acquired and placed into the initial image set I n,0 , the shooting external parameters of each shooting angle are confirmed by Euler geometry method and placed into the initial shooting external parameter set E n,0 ;

[0043] The visual classification network is combined with the optimal matching principle to classify the initial image set I n,0 and the initial shooting external parameter set E n,0 Screening is performed to generate enhanced image set I′ n and the shooting external parameter set E n .

[0044] Furthermore, the shooting external parameter e of the mth shooting angle is determined by the Euler geometry method. n,0 (m) comprising the following specific steps:

[0045] The inertial sensor is used to measure the X-axis deflection angle of the camera with respect to the world coordinate system at the mth shooting angle. Y-axis deflection angle and Z-axis deflection angle

[0046] Based on the X-axis deflection angle Y-axis deflection angle and Z-axis deflection angle Get the rotation matrix of the camera at the mth shooting angle Rotation Matrix Represents the rotation of the camera coordinate system relative to the world coordinate system, the rotation matrix The specific calculation formula is as follows:

[0047]

[0048] in, and They represent the X-axis rotation matrix, Y-axis rotation matrix, and Z-axis rotation matrix of the camera coordinate system with respect to the world coordinate system respectively;

[0049] Get the camera position p c , the camera position p c The transpose of Translation Vector Indicates the translation of the origin of the camera coordinate system with respect to the origin of the world coordinate system;

[0050] Combined rotation matrix and translation vector To build shooting external reference Shooting external reference n,0 (m) determines the transformation relationship from the world coordinate system to the camera coordinate system.

[0051] Furthermore, the visual classification network is combined with the optimal matching principle to generate the enhanced image set I′ n and the shooting external parameter set E n The specific steps include:

[0052] Get the dimension L c,w ×L c,h Image i n,0 (m), m=1,2,…,(M+1) 2 , L c,w and L c,h are the pixel width and pixel height of the camera respectively;

[0053] Process image i through 2 repeated deep mining groups n,0 (m) Generate feature matrix z n,0 (m), the deep mining group includes a convolution layer with a kernel size of 3 and a convolution stride of 1, ReLU, and a pooling layer with a pooling kernel size of 2 and a pooling stride of 1;

[0054] The feature matrix z n,0 (m) Convolve with L1 convolution kernels to generate a multi-channel feature matrix Z n,0 (m), the multi-channel feature matrix Z n,0 Each pixel in (m) is expanded into an anchor point according to the ratio, generating a multi-channel anchor point matrix Z′ n,0 (m);

[0055] The multi-channel anchor matrix Z′ n,0 (m) After convolution and reshape, the dimension is reconstructed and Sigmoid is used to generate the positive score of each anchor point;

[0056] The L2 anchor points with the highest positive scores are selected and fixed, and the remaining anchor points are subjected to non-maximum suppression. The linear layer generates n,0 (m) Enhanced image i′ of the same dimension n,0 (m) and replace the initial image set I n,0 The image in n,0 (m);

[0057] The image i′ is enhanced by two fully connected layers and Softmax processing n,0 (m) Generate classification score vector g n,m ;

[0058] When all enhanced images and corresponding classification score vectors are generated, the initial enhanced image set I′ is obtained. n,0 And the classification score matrix G n , from the task target number set k Ta (n) extracts the single task target number in sequence and queries the classification score matrix Gn , mark the enhanced image with the highest probability and the corresponding shooting external parameter with a single task target number;

[0059] Eliminate the initial enhanced image set I′ n,0 and the initial shooting external parameter set E n,0 The unlabeled enhanced images and external references in the image set are renumbered according to the task target number to generate the enhanced image set I′. n and the shooting external parameter set E n .

[0060] Furthermore, the tracking conversion deflection algorithm is used to statistically generate the deflection sequence de n The specific steps include:

[0061] Based on the kth n Target number k t +k n -1, from the enhanced image set I′ n and the shooting external parameter set E n Get the corresponding enhanced image i′ n (k t +k n -1) and shoot external reference e n (k t +k n -1), 1≤k n ≤K n , K n For mission area A Ta (n) Total number of mission targets;

[0062] In the enhanced image i′ n (k t +k n -1) Establish L c,w ×L c,h dimensional pixel coordinate system, and determine the task target pixel coordinate p through Hough transform c,t,p (k t +k n -1), where L c,w and L c,h are the pixel width and pixel height of the camera respectively;

[0063] Determine the camera's internal parameters c , shoot internal reference in c Determines the conversion relationship from the camera coordinate system to the two-dimensional pixel coordinate system, and shoots the internal reference in c The specific formula is as follows:

[0064]

[0065] Among them, f c is the focal length of the camera;

[0066] Based on the camera tracking calibration principle, the task target pixel coordinates p c,t,p (k t +k n -1) and the world coordinates of the mission target The following conversion relationship exists:

[0067]

[0068] The camera tracking calibration principle shows that any world coordinate is converted into camera coordinates through shooting external parameters, and then converted into pixel coordinates through shooting internal parameters. c,t,p (k t +k n -1) Reverse the world coordinates of the mission target

[0069] Get the target installation position p t (k t +k n -1), calculate and obtain the target number as k t +k n The deflection de(k t +k n -1);

[0070] Statistical task target number set k Ta (n) K n The deflections corresponding to the task targets are arranged in order of target numbers to generate a deflection sequence de n .

[0071] A large-span bridge deflection detection device based on drone vision is used to perform a large-span bridge deflection detection method based on drone vision, including a drone, a drone base plate is fixed at the bottom of the drone, a rotating shaft is connected to the lower end of the drone base plate, a connecting shaft is fixed at the bottom of the rotating shaft, the rotating shaft is used to control the horizontal field of view rotation of the bottom connecting shaft, and a camera is fixed to the top of the connecting shaft through a rotatable bearing, and the bearing is used to control the vertical field of view rotation of the camera.

[0072] It is worth noting that the target is a passive mark, which is fixed to the main beam by special glue or bolt structure. It is used to mark the points on the main beam where deflection detection is required. The manufacturing material and marking style can be determined according to the external environment in which the main beam is located.

[0073] Compared with the prior art, the present invention has the following significant advantages:

[0074] 1. Design a reverse inference method combined with a repeated merging strategy to pre-plan and generate a UAV task sequence. Consider the actual horizontal field of view and actual vertical field of view of the camera, reversely infer the shooting area based on the target installation position, change the original fixed-point shooting into shooting at any point in the shooting area, and use a repeated merging strategy to merge the highly overlapping shooting areas, greatly improving the UAV shooting efficiency;

[0075] 2. Design a gradual adjustment strategy. After the UAV reaches the mission area, it can hover at any time and obtain multiple images by gradually adjusting the camera shooting angle. The visual classification network is used to adaptively filter images without mission targets and generate enhanced images that highlight the mission targets. At the same time, the shooting extrinsic parameters corresponding to the shooting angle are synchronously obtained based on the Euler geometry method.

[0076] 3. Design a tracking and conversion deflection algorithm, combine it with the camera tracking calibration principle, adaptively determine the pixel coordinates of the task target in the enhanced image, convert the task target pixel coordinates into the task target world coordinates based on the shooting external parameters and fixed shooting internal parameters, and obtain the deflection by comparing it with the target installation coordinates. This realizes batch deflection detection at any position and angle in the shooting area, greatly improving the detection efficiency and reducing the difficulty of equipment deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a flow chart of a method for detecting the deflection of the main beam of a large-span steel structure based on UAV vision;

[0078] Figure 2 It is a flow chart of the repeated merging strategy in the present invention;

[0079] Figure 3 This is a flow chart of the visual classification network combined with the optimal matching principle in the present invention;

[0080] Figure 4 This is a diagram of a large-span bridge deflection detection device based on drone vision in the present invention. DETAILED DESCRIPTION

[0081] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0082] Example 1

[0083] like Figure 1 As shown, a specific embodiment of the present invention discloses a long-span bridge deflection detection method based on drone vision, comprising the following specific steps:

[0084] Pre-construct the target parameter sequence B based on the structural characteristics of the main beam t and installing the target;

[0085] The reverse inference method is used to consider the shooting parameters and the target installation position of a single target, and the shooting area of ​​a single target is determined in advance and recorded in the shooting area sequence A. t In the above example, the shooting area sequence A is planned based on the repeated merging strategy. t Generate UAV mission sequence Ta u ;

[0086] According to the UAV mission sequence Ta u The single UAV task is executed in the order of the single UAV task, the single UAV task is headed to the task area corresponding to the single UAV task, and the tracking deflection detection method based on visual classification is used to generate a single deflection sequence corresponding to the single UAV task;

[0087] Statistical UAV mission sequence Ta u The single deflection sequence corresponding to all single UAV tasks generates the total deflection sequence de = {de1,…,de N}, N is the total number of UAV tasks, at which time the main beam deflection detection is completed.

[0088] Furthermore, the structural characteristics of the main beam are the basis for constructing the target parameter sequence. Since the long-span main beam has a large span and complex forces, when constructing the target parameter sequence, professional bridge engineers are required to determine the total number of targets K based on the structural form, span and support method of the bridge. t and the target installation position p of a single target t , target parameter sequence B t Including K t The target parameter pair of a target, target parameter sequence B t The specific form is as follows:

[0089]

[0090] Among them, b t (k t ) is the kth t The target parameter pairs of k targets t is the target number, p t (k t ) is the kth t The target installation position of each target, the world coordinate system is the three-dimensional rectangular coordinate system of the real world, which has been established in advance before constructing the target parameter sequence.

[0091] Furthermore, the reverse inference method is used to predetermine and record the shooting area of ​​a single target, including the following specific steps:

[0092] Get the kth t The target installation position p of the target t (k t )=[x t (kt ),y t (k t ),z t (k t )] and the camera parameters of the drone, which include the focal length f of the camera on the drone c , the minimum shooting distance of the camera Maximum shooting distance of the camera Maximum horizontal adjustment angle of the camera Maximum vertical adjustment angle of the camera The horizontal width w of the camera sensor c and the vertical height h of the camera sensor c ;

[0093] Determine the camera's horizontal field of view angle θ c,w and vertical viewing angle θ c,h , horizontal viewing angle θ c,w and vertical viewing angle θ c,h The horizontal field of view angle θ determines the maximum angle that can be captured in the horizontal and vertical planes when the camera is fixed in a fixed direction. c,w and vertical viewing angle θ c,h The specific calculation formula is as follows:

[0094]

[0095] Among them, w c and h c are the horizontal width and vertical height of the camera sensor, respectively;

[0096] Based on the maximum horizontal adjustment angle of the camera and maximum vertical adjustment angle The camera's horizontal and vertical fields of view can be further expanded. After expansion, the actual horizontal field of view of the camera is The actual vertical field of view is in, and are the leftmost horizontal field of view and the rightmost horizontal field of view respectively. and They are the lowest vertical field of view and the highest vertical field of view respectively;

[0097] Take the kth t The target installation position p of the target t (k t )=[x t (k t ),y t (k t ),z t (k t )] as the center of the sphere, with the kth tThe plane where the targets are located is the interface, and the minimum shooting distance is and maximum shooting distance Construct the minimum hemisphere and the maximum hemisphere in the world coordinate system with radius , and the space area inside the maximum hemisphere and outside the minimum hemisphere is taken as the kth t The initial shooting area A0(k t );

[0098] Get the minimum moving distance of the drone Minimum moving distance Generate a grid to divide the initial shooting area To obtain the initial shooting area Several grid points within ;

[0099] Calculate the initial capture area Each grid point within the target installation position p t (k t ) to determine whether the grid point meets the field of view condition. The field of view condition specifically refers to the azimuth of the grid point within the actual horizontal field of view of the camera. The grid point’s elevation angle is also within the actual vertical field of view. Inside;

[0100] The spatial area contained by all grid points that meet the field of view condition is taken as the kth t The feasible shooting area of ​​​​the target Use planes parallel to the three coordinate axes in the world coordinate system to cut off the feasible shooting area Irregular parts in the image are obtained by inscribing them in a feasible shooting area. The maximum rectangular area of ​​​​the kth t Shooting area A of targets t (k t ) recorded in the shooting area sequence A t middle.

[0101] like Figure 2 As shown, further, the UAV task sequence Ta is generated based on repeated merging strategy planning u The specific steps include:

[0102] Get the shooting area sequence A t ={A t (1),…,A t (K t )}, shooting area sequence A t Recorded K t The shooting area of ​​​​the target, K t is the total number of targets;

[0103] Execute empty sequence judgment to determine the drone mission sequence Ta u Is it an empty sequence? If the drone mission sequence Ta u If the UAV task sequence Ta is empty, then the maximum task number and the maximum target number are both 0. Increase the maximum task number and the maximum target number by 1, and jump to the component area set expansion. u Not an empty sequence, confirm the drone mission sequence Ta u The maximum task number in ;

[0104] Execute target exhaustion judgment, obtain the maximum target number in the UAV task corresponding to the maximum task number, and determine whether the maximum target number is equal to the total number of targets K t , if the maximum target number is equal to the total number of targets K t , then the repeated merging strategy is stopped immediately, and the UAV task sequence Ta u Planning is completed, if the maximum target number is less than the total number of targets K t , let the maximum task number and the maximum target number both increase by 1;

[0105] Execute component area set expansion, and place the shooting area corresponding to the maximum target number into the component area set corresponding to the maximum task number;

[0106] Perform extended validity judgment to determine whether the ratio of the overlapped area volume of all shooting areas in the component area set to the volume of any single shooting area is greater than or equal to the overlap ratio λ A , in this embodiment, the overlap ratio λ A =0.95;

[0107] If the ratio of the volume of the overlapped area to the volume of any single shot area is less than the overlap ratio λ A , then remove the shooting area corresponding to the maximum target number in the component area set, re-acquire the overlapping area of ​​all shooting areas in the component area set, take the largest inscribed cuboid area in the overlapping area as the task area corresponding to the maximum task number, extract all target numbers in the component area set, construct the task target number set corresponding to the maximum task number, package the task area and the task target number set to generate the drone task corresponding to the maximum task number, add 1 to the maximum task number and jump to execute the target exhaustion judgment;

[0108] If the ratio of the overlap area volume to the volume of any single shooting area is greater than or equal to the overlap ratio λ A , increase the maximum target number by 1 and jump to the component area set expansion.

[0109] Furthermore, the tracking deflection detection method based on visual classification is used to generate a single deflection sequence corresponding to a single UAV task, which includes the following specific steps:

[0110] Assume that the UAV mission sequence Ta u The total number of UAV tasks is N, that is, the UAV task sequence Ta u ={Ta u (1),…,Ta u (N)}, and according to the UAV mission sequence Ta u The nth drone task Ta should be executed in the order u (n), n=1,2,…,N;

[0111] Confirm the nth drone mission Ta u Task area A in (n) Ta (n) and the mission target number set k Ta (n), assuming that the task target number set k Ta (n) = {k t ,…,k t +K n -1}, n≤k t ≤k t +K n -1≤K t , K t is the total number of targets, K n For mission area A Ta (n) The total number of mission targets, then the mission area A Ta (n) is the shooting area A t (k t ) to shooting area A t (k t +K n -1) the maximum inscribed cuboid area of ​​the overlapping area;

[0112] Get the camera position on the drone in real time based on GPS c =[x c ,y c ,z c ], when the camera position p is detected c Located in Mission Area A Ta (n) The drone can be hovered at any time, and the mission area A can be obtained based on the gradual adjustment strategy. Ta (n) The enhanced image set I′ n and the shooting external parameter set E n ;

[0113] The tracking transformation deflection algorithm is used to consider the enhanced image set I′ n The pixel coordinates of the target in the enhanced image are combined with the shooting external parameter set E n The coordinate tracking conversion of the shooting external parameters of the corresponding task target is performed to obtain the deflection corresponding to each task target, and the nth UAV task Ta is statistically generated u(n) The corresponding deflection sequence de n .

[0114] Furthermore, based on the step-by-step adjustment strategy, task area A is obtained. Ta (n) The enhanced image set I′ n and the shooting external parameter set E n The specific steps include:

[0115] Get the actual horizontal field of view of the camera and actual vertical field of view set up The initial shooting angle when the camera is not adjusted. Set the horizontal unit adjustment angle of the camera. and vertical units to adjust the angle The specific settings are as follows:

[0116]

[0117] Wherein, M is the number of horizontal field of view adjustment and the number of vertical field of view adjustment;

[0118] Adjust angle according to horizontal units and vertical units to adjust the angle The actual horizontal field of view and actual vertical field of view Each of the two is divided into M+1 horizontal shooting angles and M+1 vertical shooting angles, and combined to generate a (M+1)×(M+1)-dimensional shooting angle matrix X, where X i,j represents the (i-1)·(M+1)+jth shooting angle in the shooting angle matrix X. The specific formula is as follows:

[0119]

[0120] In essence, X i,j That is, the camera shooting angle determined by adjusting the camera i-1 times in the horizontal field of view and j-1 times in the vertical field of view;

[0121] At each shooting angle of the shooting angle matrix X, the camera acquires an image and places it into the initial image set I n,0 ={i n,0 (1),…,i n,0 ((M+1) 2 )}, the shooting external parameters of each shooting angle are confirmed by Euler geometry method and placed into the initial shooting external parameter set E n,0 ={e n,0 (1),…,e n,0 ((M+1) 2 )}, where i n,0 (m) and en,0 (m) are the images obtained from the mth shooting angle and the corresponding shooting external parameters;

[0122] The visual classification network is combined with the optimal matching principle to classify the initial image set I n,0 and the initial shooting external parameter set E n,0 Filter and generate task area A Ta (n) The enhanced image set I′ n and the shooting external parameter set E n .

[0123] Furthermore, the task area A is confirmed by Euler geometry method. Ta The shooting external parameter e of the mth shooting angle in (n) n,0 (m) comprising the following specific steps:

[0124] The camera's built-in inertial sensor measures the camera's X-axis deflection angle with respect to the world coordinate system at the mth shooting angle. Y-axis deflection angle and Z-axis deflection angle

[0125] Get the rotation matrix of the camera at the mth shooting angle Rotation Matrix Represents the rotation of the camera coordinate system relative to the world coordinate system, the rotation matrix The specific calculation formula is as follows:

[0126]

[0127] in, and They represent the X-axis rotation matrix, Y-axis rotation matrix, and Z-axis rotation matrix of the camera coordinate system with respect to the world coordinate system, respectively. C and S represent the cosine function and sine function, respectively.

[0128] Get the camera position p in the world coordinate system when the drone is hovering c =[x c ,y c ,z c ], change the camera position p c The transpose of the task area A Ta Translation vector of (n) Translation Vector Indicates the translation of the origin of the camera coordinate system with respect to the origin of the world coordinate system;

[0129] Combined rotation matrix and translation vector Construct the shooting external parameter e of the mth shooting angle n,0 (m), shoot external reference en,0 (m) determines the conversion relationship from the world coordinate system to the camera coordinate system. The camera coordinate system refers to the three-dimensional coordinate system constructed with the camera as the origin, the vertical axis perpendicular to the camera lens as the Z axis, the horizontal axis parallel to the camera lens as the X axis, and the longitudinal axis parallel to the camera lens as the Y axis. n,0 The specific formula of (m) is

[0130] like Figure 3 As shown, further, the visual classification network is combined with the optimal matching principle to generate the task area A Ta (n) The enhanced image set I′ n and the shooting external parameter set E n The specific steps include:

[0131] Get the initial image set I n,0 Image i obtained from the mth shooting angle n,0 (m), m=1,2,…,(M+1) 2 , image i n,0 (m) dimension is fixed to L c,w ×L c,h , where L c,w and L c,h are the pixel width and pixel height of the camera respectively;

[0132] Extract image i through 2 repeated deep mining groups n,0 (m) The corresponding feature matrix z n,0 (m), where the deep mining group includes a convolution layer with a convolution kernel size of 3 and a convolution step of 1, ReLU, and a pooling layer with a pooling kernel size of 2 and a pooling step of 1. Since the convolution dimension remains unchanged and the pooling dimension is halved, the feature matrix z n,0 The dimension of (m) is (L c,w / 4)×(L c,h / 4);

[0133] Feature matrix z n,0 (m) contains a large number of background features and a small number of target features. n,0 (m) is the target feature, and the feature matrix z n,0 (m) are convolved with L1 convolution kernels respectively, and the generated dimension is (L c,w / 4)×(L c,h / 4)×L1 multi-channel feature matrix Z n,0 (m), and the multi-channel feature matrix Z n,0 Each pixel in (m) is expanded to 9 anchor points, and the generated dimension is (L c,w / 4)×(L c,h / 4)×(L1×9) multi-channel anchor matrix Z′ n,0 (m);

[0134] The multi-channel anchor matrix Z′ n,0 (m) after convolution transformation dimension is (L c,w / 4)×(L c,h / 4)×9, and reshape the dimension to (3·L c,w / 4)×(3·L c,h / 4), use Sigmoid to generate a positive score for each anchor point, which represents the probability that the anchor point belongs to the target feature;

[0135] The L2 anchor points with the highest positive scores are selected and fixed, and the remaining anchor points are subjected to non-maximum suppression and further transformed into the same dimension as image i through a linear layer. n,0 (m)Same L c,w ×L c,h , generate enhanced image i′ n,0 (m) and replace the initial image set I n,0 The image in n,0 (m);

[0136] The image i′ is enhanced by two fully connected layers and Softmax processing n,0 (m)Generate(K t +1)×1-dimensional classification score vector g n,m =[g n,m (0),…,g n,m (K t )], where g n,m (0) is the enhanced image i′ n,0 (m) The probability of not including any target, g n,m (k t ) is the enhanced image i′ n,0 (m) includes kth t The probability of a target, 1≤k t ≤K t , K t is the total number of targets;

[0137] Get (M+1) 2 The classification score vectors corresponding to the enhanced images are integrated into the classification score matrix G n , at this time the initial image set I n,0 All images in the image set have been replaced by the corresponding enhanced images. n,0 Transformed into the initial enhanced image set I′ n,0 , from the task target number set k Ta (n) extracts the single task target number in sequence and queries the classification score matrix Gn , mark the enhanced image with the highest probability and the corresponding shooting external parameter with a single task target number until the task target number set k Ta All mission target numbers in (n) are extracted;

[0138] Eliminate the initial enhanced image set I′ n,0 and the initial shooting external parameter set E n,0 The unlabeled enhanced images and shooting external references in the image set are quantized according to the marked task target number. n,0 and the initial shooting external parameter set E n,0 The remaining enhanced images and the captured external references are renumbered to generate the task area A Ta (n) The enhanced image set I′ n ={i′ n (k t ),…,i′ n (k t +K n -1)} and the shooting external parameter set E n ={e n (k t ),…,e n (k t +K n -1)}.

[0139] Furthermore, the tracking conversion deflection algorithm is used to statistically generate the nth UAV mission Ta u (n) The corresponding deflection sequence de n The specific steps include:

[0140] For the task target number set k Ta (n) n Target number k t +k n -1, from the enhanced image set I′ n and the shooting external parameter set E n Get the corresponding enhanced image i′ n (k t +k n -1) and shoot external reference e n (k t +k n -1), 1≤k n ≤K n , K n For mission area A Ta (n) Total number of mission targets;

[0141] In pixels, in the enhanced image i′ n (k t +k n-1) Establish L c,w ×L c,h dimensional pixel coordinate system, where L c,w and L c,h are the pixel width and pixel height of the camera respectively;

[0142] Determine the enhanced image i′ by Hough transform n (k t +k n -1) The target number is k t +k n -1 mission target pixel coordinate p c,t,p (k t +k n -1)=[x c,t,p (k t +k n -1),y c,t,p (k t +k n -1)];

[0143] Determine the camera's internal parameters c , shoot internal reference in c Determines the conversion relationship from the camera coordinate system to the two-dimensional pixel coordinate system, and shoots the internal reference in c The specific formula is as follows:

[0144]

[0145] Among them, f c is the focal length of the camera. Therefore, when the camera model is fixed, the camera's internal parameters in c Also for fixation;

[0146] Based on the camera tracking calibration principle, the task target pixel coordinates p in the pixel coordinate system c,t,p (k t +k n -1) and the world coordinates of the mission target in the world coordinate system The following conversion relationship exists:

[0147]

[0148] The camera tracking calibration principle shows that any world coordinate in the world coordinate system is converted into the camera coordinate in the camera coordinate system through the shooting external parameter, and then converted into the pixel coordinate in the pixel coordinate system through the shooting internal parameter. c,t,p (k t +k n -1)=[x c,t,p (k t +k n -1),yc,t,p (k t +k n -1)] Reverse the world coordinates of the mission target

[0149] Based on target number k t +k n -1 Get the corresponding target installation position p t (k t +k n -1), calculate and obtain the target number as k t +k n -1 mission target corresponding to the deflection Among them, y t (k t +k n -1) and are the target installation positions p t (k t +k n -1) and the world coordinates of the mission target Y-axis value of

[0150] Statistical task target number set k Ta (n) K n The deflections corresponding to the task targets are arranged in order of target numbers to generate a deflection sequence de n ={de(k t ),…,de(k t +K n -1)}.

[0151] Example 2

[0152] like Figure 4 As shown, the present invention also discloses a large-span bridge deflection detection device based on UAV vision, which is used to execute a large-span steel structure main beam deflection detection method based on UAV vision proposed in Example 1, including a UAV 1, a UAV base plate 2 is fixed at the bottom of the UAV 1, a rotating shaft 3 is connected to the lower end of the UAV base plate 2, a connecting shaft 4 is fixed at the bottom of the rotating shaft 3, and the rotating shaft 3 is used to control the horizontal field of view rotation of the bottom connecting shaft 4, and a camera 6 is fixed to the top of the connecting shaft 4 through a rotatable bearing 5, and the bearing 5 is used to control the vertical field of view rotation of the camera 6.

[0153] It is worth noting that the target is a passive mark, which is fixed to the main beam by special glue or bolt structure. It is used to mark the points on the main beam where deflection detection is required. The target does not require any electronic components. The manufacturing material and marking style can be determined according to the external environment in which the main beam is located to ensure that the target is durable and eye-catching. Therefore, the target does not need to be replaced frequently and can be used for a long time after a single installation. The passive design makes the target more adaptable to various external environments. The target used in this embodiment is a square iron sheet with white paint on the surface and the target number is marked with black paint in the middle of the target.

[0154] The present invention discloses a large-span bridge deflection detection device and method based on unmanned aerial vehicle vision. A target parameter sequence is pre-constructed based on the structural characteristics of the main beam and the target is installed. A reverse inference method is adopted to consider the shooting parameters and the target position of a single target, and a shooting area of ​​the single target is pre-determined. A UAV task sequence is generated based on repeated merging strategy planning to improve the deflection detection efficiency. A single UAV task is executed in sequence according to the UAV task sequence. The task area corresponding to the single UAV task is reached and a tracking deflection detection method based on visual classification is adopted to generate a single deflection sequence corresponding to the single UAV task. The deflection sequences corresponding to all UAV tasks in the UAV task sequence are counted to generate a total deflection sequence, thereby realizing non-fixed-point automated main beam deflection detection and solving the problems of difficult equipment layout and low detection efficiency caused by fixed-point measurement in the prior art.

[0155] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A long-span bridge deflection detection method based on drone vision, characterized in that: The specific steps include: Construct the target parameter sequence based on the structural characteristics of the main beam and install the target; The reverse inference method is used to determine the shooting area of ​​a single target to generate a shooting area sequence, and the UAV task sequence is generated based on repeated merging strategy planning; Execute a single UAV task according to the UAV task sequence order, go to the task area and use a tracking deflection detection method based on visual classification to generate a single deflection sequence corresponding to the single UAV task; The single deflection sequences corresponding to all single UAV tasks in the UAV task sequence are counted to generate a total deflection sequence, thereby completing the main beam deflection detection.

2. A long-span bridge deflection detection method based on drone vision as claimed in claim 1, characterized in that: The method of generating a single deflection sequence by using the tracking deflection detection method based on visual classification comprises the following specific steps: Confirm the mission area and mission target number set in a single UAV mission; When the camera position is located in the mission area, the drone is hovered, and a reinforced image set and a shooting external parameter set are acquired based on a step-by-step adjustment strategy; A tracking conversion deflection algorithm is used to perform coordinate tracking conversion, the deflection corresponding to each task target in the single UAV task is obtained, and a single deflection sequence corresponding to the single UAV task is statistically generated.

3. A long-span bridge deflection detection method based on drone vision as claimed in claim 2, characterized in that: The step-by-step adjustment strategy based on which the enhanced image set and the captured external parameter set are obtained comprises the following specific steps: Get the actual horizontal field of view and the actual vertical field of view, set the initial shooting angle, horizontal unit adjustment angle and vertical unit adjustment angle; Dividing the actual horizontal field of view range and the actual vertical field of view range into a plurality of shooting angles according to the horizontal unit adjustment angle and the vertical unit adjustment angle; Acquire images at each shooting angle and place them into the initial image set, confirm the shooting extrinsic parameters of each shooting angle by Euler geometry method and place them into the initial shooting extrinsic parameter set; The visual classification network is combined with the optimal matching principle to screen the initial image set and the initial shooting extrinsic parameter set to generate an enhanced image set and a shooting extrinsic parameter set.

4. A long-span bridge deflection detection method based on drone vision as claimed in claim 3, characterized in that: The Euler geometry method for determining the shooting extrinsic parameters of a single shooting angle includes the following specific steps: Get the X-axis deflection angle, Y-axis deflection angle and Z-axis deflection angle of the camera at a single shooting angle; Acquire a rotation matrix of a single shooting angle based on the X-axis deflection angle, the Y-axis deflection angle, and the Z-axis deflection angle, wherein the rotation matrix represents a rotation of a camera coordinate system relative to a world coordinate system; The transpose of the camera position is used as a translation vector, wherein the translation vector indicates the translation of the origin of the camera coordinate system with respect to the origin of the world coordinate system; The rotation matrix and the translation vector are combined to construct the single shooting angle shooting extrinsic parameter, and the shooting extrinsic parameter determines the conversion relationship from the world coordinate system to the camera coordinate system.

5. The long-span bridge deflection detection method based on drone vision as claimed in claim 3 is characterized in that: The visual classification network generates a reinforced image set and a captured external parameter set in combination with the optimal matching principle, including the following specific steps: Acquire a single image, and process the single image through multiple deep mining groups to generate a corresponding feature matrix, wherein the deep mining group includes a convolution layer, a ReLU layer, and a pooling layer; Convolving the feature matrix with several convolution kernels respectively, and expanding each pixel point in the convolution result into an anchor point according to a fixed ratio to generate a multi-channel anchor point matrix; The multi-channel anchor point matrix is ​​reconstructed through convolution and reshape, and a positive score for each anchor point is generated based on Sigmoid; Select several anchor points with the highest positive scores and fix them, perform non-maximum suppression on the remaining anchor points, and generate a single enhanced image with the same dimension as the single image through a linear layer and replace the single image; Processing the single enhanced image through two fully connected layers and Softmax to generate a corresponding classification score vector; Obtain all single enhanced images and corresponding classification score vectors to generate an initial enhanced image set and a classification score matrix, query the classification score matrix according to the single task target number, and mark the enhanced image with the highest probability and the corresponding shooting external parameter; Unlabeled enhanced images and shot extrinsic parameters are removed, and the remaining enhanced images and shot extrinsic parameters are renumbered to generate enhanced image sets and shot extrinsic parameter sets.

6. The long-span bridge deflection detection method based on drone vision as claimed in claim 2 is characterized in that: The tracking conversion deflection algorithm statistically generates a single deflection sequence corresponding to a single UAV mission, including the following specific steps: Based on a single target number, a single enhanced image and a single shot extrinsic parameter are obtained from the enhanced image set and the shot extrinsic parameter set respectively; Establishing a pixel coordinate system in the single enhanced image, and determining the task target pixel coordinates by Hough transform; Determine the shooting internal parameters based on the focal length of the camera. The shooting internal parameters determine the conversion relationship from the camera coordinate system to the pixel coordinate system. The world coordinates of the task target are obtained based on the camera tracking calibration principle combined with the task target pixel coordinates. The camera tracking calibration principle indicates that the world coordinates are converted into camera coordinates by shooting external parameters, and then converted into pixel coordinates by shooting internal parameters. The target installation position is obtained, the deflection corresponding to a single mission target is calculated, and the deflections corresponding to all mission targets in the mission target number set are counted to generate a single deflection sequence corresponding to a single UAV mission.

7. The long-span bridge deflection detection method based on drone vision as claimed in claim 1 is characterized in that: The generation of the UAV task sequence based on the repeated merging strategy planning includes the following specific steps: Get the shooting area sequence; Execute the empty sequence judgment to determine whether the UAV task sequence is an empty sequence. If it is an empty sequence, set the maximum task number and the maximum target number to 1, jump to execute the component area set expansion, and if it is not an empty sequence, confirm the maximum task number in the UAV task sequence; Execute target exhaustion judgment, obtain the maximum target number corresponding to the maximum task number, and determine whether the maximum target number is equal to the total number of targets. If it is equal, stop the repeated merging strategy. If it is less than, increase the maximum task number and the maximum target number by 1. Execute component area set expansion, and place the shooting area corresponding to the maximum target number into the component area set corresponding to the maximum task number; Perform extended validity judgment to determine whether the ratio of the overlapping area volume of all shooting areas in the component area set to the volume of any single shooting area is greater than or equal to the overlapping ratio; If it is less than the overlap ratio, the shooting area corresponding to the largest target number in the component area set is eliminated, and the drone task corresponding to the largest task number is generated. The maximum task number is increased by 1 and the target exhaustion judgment is executed. If it is greater than or equal to the overlap ratio, increase the maximum target number by 1 and jump to the component area set expansion.

8. The long-span bridge deflection detection method based on drone vision as claimed in claim 1 is characterized in that: The method of predetermining and recording the shooting area of ​​a single target by using the reverse inference method includes the following specific steps: Get the target installation position of a single target and the drone's photography parameters; Determine the horizontal and vertical viewing angles of the camera according to the horizontal width and vertical height of the camera sensor; Expand the horizontal field of view and the vertical field of view of the camera based on the maximum horizontal adjustment angle and the maximum vertical adjustment angle of the camera, and obtain the actual horizontal field of view and the actual vertical field of view of the camera; Taking the target installation position as the sphere center, constructing a minimum hemisphere and a maximum hemisphere on the right side of the plane where the single target is located to determine the initial shooting area; The initial shooting area is divided based on the minimum moving distance of the drone to obtain grid points, and the shooting area of ​​a single target is obtained based on the field of view conditions and recorded in the shooting area sequence.

9. A long-span bridge deflection detection method based on drone vision as claimed in claim 8, characterized in that: The method of acquiring the shooting area of ​​a single target based on the field of view condition includes the following specific steps: Calculate the azimuth and elevation angles between a single grid point in the initial shooting area and the target installation position; Determine whether a single grid point meets a field of view condition, wherein the field of view condition refers to that the azimuth angle of the single grid point is within an actual horizontal field of view and the elevation angle of the single grid point is within an actual vertical field of view; The spatial area contained by all grid points that meet the field of view condition is taken as a feasible shooting area of ​​a single target; The largest rectangular area inscribed in the feasible shooting area is obtained as the shooting area of ​​the single target.

10. A large-span bridge deflection detection device based on drone vision, used to execute a large-span bridge deflection detection method based on drone vision as described in any one of claims 1-9, comprising a drone, a drone base plate fixed at the bottom of the drone, a rotating shaft connected to the lower end of the drone base plate, a connecting shaft fixed at the bottom of the rotating shaft, the rotating shaft is used to control the horizontal field of view rotation of the bottom connecting shaft, the top of the connecting shaft fixes a camera through a rotatable bearing, and the bearing is used to control the vertical field of view rotation of the camera.

Images