Wide-range deformation displacement measurement method, device and application compensated by virtual camera array

Abstract

The present application relates to a wide-range deformation displacement measurement method, device and application of virtual camera array compensation. The device comprises: a UAV platform; a platform relative motion measurement camera group mounted on the upper part of the UAV body, used to form at least two equivalent observation angles to observe the static control points in space; a global field of view camera module mounted on the upper part of the UAV body, used to obtain the wide-range image of the region to be measured; a holder mounted on the lower part of the UAV body; a deformation displacement measurement camera mounted on the holder, used to obtain the high-resolution image of the target point; a marker block mounted on the holder or a component rigidly connected with the deformation displacement measurement camera; a holder angle measurement camera mounted on the lower part of the UAV body platform and rigidly connected with the UAV body, used to observe the marker block to solve the real pose of the holder; a data processing module connected with the platform relative motion measurement camera group, the global field of view camera module, the holder angle measurement camera and the deformation displacement measurement camera respectively.

Description

Technical Field

[0001] This invention belongs to the fields of visual measurement, UAV remote sensing, structural health monitoring, precision optical measurement, and multi-camera pose calculation. Specifically, it relates to a method and apparatus for precisely measuring the deformation and displacement of a target point under UAV conditions by mounting a camera array, a platform relative motion measurement camera, a global field-of-view camera, a gimbal deformation and displacement measurement camera, and a gimbal rotation angle measurement camera on a UAV platform, through platform motion compensation, global map guidance, precise measurement of the gimbal's true pointing angle, and inverse calculation of the target point image displacement.

[0002] This invention can be used for non-contact measurement of deformation, displacement, deflection, vibration and attitude changes of large-scale structures such as bridges, dams, high slopes, power transmission towers, wind turbine towers, tunnel entrances, ship structures, port facilities, and the outer surface of major equipment. Background Technology

[0003] Large-scale engineering structures and major equipment typically require deformation, deflection, and vibration monitoring during service. Currently, total stations, GNSS, laser rangefinders, fixed visual stations, and photogrammetric systems are widely used for structural deformation monitoring. However, in complex scenarios such as bridge mid-spans, tower tops, mountain slopes, tunnel entrances, and ship exteriors, traditional fixed stations face challenges such as difficult station deployment, limited observation angles, high costs of repeated deployments, and insufficient measurement coverage.

[0004] Unmanned aerial vehicle (UAV) platforms offer advantages such as high mobility, flexible viewing angles, rapid deployment, and access to hazardous areas, making them suitable for large-scale structural inspections and emergency measurements. However, directly using UAVs for high-precision displacement measurement still faces the following technical bottlenecks:

[0005] First, during hovering or low-speed flight, UAVs are affected by wind disturbances, rotor vibrations, flight control errors, and structural flexibility, inevitably causing translational and rotational disturbances. The displacement of the target point in the image comes not only from the actual structural deformation but also from changes in the line of sight and image drift caused by the movement of the UAV platform itself. Relying solely on the UAV's own IMU, GNSS, or flight control attitude information is usually insufficient to meet the requirements for precise deformation measurement, especially in environments such as under bridges, urban canyons, strong electromagnetic interference, or GNSS obstruction, where the accuracy of inertial navigation and satellite positioning will further decrease.

[0006] Secondly, while conventional gimbals can achieve camera stabilization and target pointing, issues such as encoder accuracy, control hysteresis, installation deviations, structural deformation, and vibration coupling can lead to errors between the actual pointing angle and the control angle. For long-distance telephoto measurements, even minute angular errors can be significantly amplified, directly affecting the accuracy of target displacement calculations.

[0007] Third, there is a contradiction between large-area target search and high-resolution displacement measurement. Wide field-of-view cameras can cover a large area, but their object surface resolution is low, making it difficult to meet the requirements of millimeter-level or sub-pixel-level displacement measurement; telephoto cameras have high object surface resolution, but their field of view is narrow, making it difficult to quickly locate the target to be measured.

[0008] Therefore, there is an urgent need for a visual measurement method and device that can still accurately calculate the deformation and displacement of target points even under conditions where the UAV platform itself is moving, the gimbal pointing error is incorrect, and the target search is difficult. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to overcome the difficulty in accurately separating and measuring the true deformation displacement of the target point due to the motion disturbance of the UAV platform itself, the gimbal pointing control error, and the contradiction between large field of view and high resolution in the existing UAV visual measurement technology, and to provide a method, device and application that can achieve high precision, large range, non-contact deformation displacement measurement.

[0010] The technical solution of the present invention is a method for measuring large-range deformation displacement of virtual camera array compensation, which is characterized by including the following steps:

[0011] S1. Install a platform relative motion measurement camera group and a global field of view camera above the UAV body, wherein the platform relative motion measurement camera group is used to form at least two equivalent observation angles; S2. Install a gimbal under the drone body, and install a deformation displacement measuring camera and a marker block on the gimbal; install a gimbal rotation measuring camera on the platform under the drone body; calibrate the internal parameters of each camera, and calibrate the external parameter relationships between each component; S3. Measure the equivalent observation angle of the camera group based on the relative motion of the platform, observe the stationary control point in the observation space, and calculate the pose of the UAV platform at the initial moment; collect data from each camera at the initial moment and record the initial state; S4. Acquire a large-scale image of the area to be measured through the global field-of-view camera, and guide the deformation displacement measuring camera on the gimbal to point to the target point according to the position of the target point in the global image; S5. By observing the marker block through the gimbal angle measurement camera, the true six-degree-of-freedom pose of the gimbal relative to the UAV platform is calculated to obtain the true pointing attitude of the deformation displacement measurement camera. S6. Based on the observation results of the platform relative motion measurement camera group, combined with the calibration parameters and initial observation results, calculate the six-degree-of-freedom relative motion of the UAV platform at the current moment relative to the initial moment; S7. The deformation displacement measurement camera acquires the image of the target point at the current time and matches it with the image at the initial time to extract the displacement of the target point image; S8. Substitute the relative motion of the UAV platform, the true pointing attitude of the gimbal, and the image displacement of the target point into the imaging inverse calculation model, calculate and eliminate the false displacement caused by the motion of the UAV platform and the pointing error of the gimbal, and obtain the true deformation displacement of the target point to be measured.

[0012] The core method steps of this invention are as follows:

[0013] The method of the present invention includes the following steps:

[0014] S1: System installation and mechanical connection; Install each component according to the above structure, ensuring that all measuring cameras are rigidly connected to the machine body and that the deformation displacement measuring camera is rigidly connected to the gimbal.

[0015] S2: Multi-camera system calibration; complete the precise calibration of all intrinsic parameters and extrinsic parameters between system components;

[0016] S3: Initial observation; After the UAV arrives at the measurement position, it synchronously acquires initial images from all cameras, including fixed-point images, overall structural images, gimbal marker block images, and local images of the target point, and records the initial baseline;

[0017] S4: Global field of view target recognition and gimbal guidance; The global field of view camera recognizes the target point, calculates its line of sight direction in the body coordinate system, and generates the desired gimbal rotation angle so that the deformation displacement measurement camera points to the area to be measured.

[0018] S5: Gimbal Rotation Measurement; The gimbal rotation measurement camera observes the marker block on the gimbal and obtains the true six-degree-of-freedom pose of the gimbal relative to the UAV body through pose calculation methods such as PnP. This allows us to obtain the true pointing direction of the deformation displacement measurement camera. ;

[0019] S6: Platform relative motion calculation; The platform relative motion measurement camera group observes the fixed point through multiple equivalent virtual viewpoints, and combines the minimized reprojection error model shown in formula (3) to calculate the six-degree-of-freedom relative motion of the UAV body at the current moment relative to the initial moment. ;

[0020] S7: Target point image displacement extraction; The deformation displacement measurement camera acquires the current image of the target point, and the sub-pixel matching algorithm is used to obtain the image displacement of the target point;

[0021] S8: Target True Displacement Inverse Calculation; Substitute the platform relative motion and gimbal true pose obtained in the previous steps into a unified imaging inverse calculation model. By minimizing the reprojection error, accurately remove the false displacement components caused by platform motion and gimbal pointing error from the observed image displacement, and finally solve for the true deformation displacement of the target point. .

[0022] The core concept of this invention is to construct a multi-level, functionally decoupled and mutually collaborative visual measurement system. Through a triple mechanism of "platform motion visual self-compensation + global field of view visual guidance + gimbal pointing visual precision measurement", various error sources in the measurement process are separated and compensated from the source.

[0023] Preferably, the platform relative motion measurement camera group includes a physical camera and an optical unit disposed at its front end. The optical unit is used to guide light paths from different directions to the same imaging sensor, and after optical path calibration, image separation, parameter demixing or feature decoupling, it is equivalent to multiple virtual camera observation results with independent equivalent intrinsic parameters and fixed extrinsic parameters. The platform relative motion measurement camera group observes more than three non-collinear stationary control points through each equivalent virtual field of view, and uses a multi-camera pose calculation method to obtain the six-degree-of-freedom relative motion of the UAV platform. The imaging inverse calculation model constructs an optimization objective function based on the projection relationship, and solves the true deformation displacement of the target point by minimizing the reprojection error.

[0024] As a preferred option: the platform relative motion measurement camera group at the initial moment and the current moment Observe stationary control points or immobile feature points in space respectively; the control points are in the world coordinate system. The coordinates in the image remain unchanged; the changes in the image coordinates are mainly due to the UAV's body coordinate system. The translation and rotation cause this; multiple virtual perspectives are formed by the camera group measuring the relative motion of the platform. Treating the UAV as multiple equivalent cameras, the initial pose of the UAV body is first calculated from the initial observations at the initial moment, and then the coordinate system of the UAV body is calculated from... arrive The six-degree-of-freedom relative motion, let the first... A stationary control point in the world coordinate system The homogeneous coordinates in the middle are In its first A virtual perspective At the moment The homogeneous coordinates of the image are Then the camera projection relationship can be written as:

[0025] ;

[0026] In the formula, As a scale factor, For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Represents the image coordinate system;

[0027] Combining the transformation relationship between the virtual camera coordinate system and the UAV body coordinate system, we can obtain:

[0028] ;

[0029] In the formula, It is a virtual camera coordinate system With UAV body coordinate system The fixed extrinsic parameters between them It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix;

[0030] Get the current time of the first The projection relationship of stationary control points under a virtual viewpoint is calculated by minimizing the reprojection error. :

[0031] ;

[0032] In the formula, This refers to the six-degree-of-freedom relative motion matrix of the unmanned aerial vehicle platform relative to the initial moment. For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in It is a virtual camera coordinate system With UAV body coordinate system The fixed extrinsic parameters between them It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix.

[0033] Preferably, the step of guiding the deformation displacement measurement camera through the global field-of-view camera specifically includes: determining the pixel position of the target point in the global field-of-view camera image. and the intrinsic parameter matrix of the global field of view camera Calculate the line-of-sight vector of the target point in the global field-of-view camera coordinate system. In the formula, For the target point at The downward line of sight vector, To fix external parameters Extract the rotation matrix;

[0034] Using fixed extrinsic parameters between the global field-of-view camera and the UAV body The line-of-sight vector is transformed into the UAV body coordinate system to obtain... ;

[0035] according to Calculate the desired azimuth and pitch angles of the gimbal so that the deformation displacement measuring camera is pointed at the target point to be measured;

[0036] The global field of view camera mentioned in step S1 Rigidly mounted on top of the UAV, it is used to photograph a large area to be tested and identify structural contours, candidate test points, artificial targets, or natural features in the images; the system determines the target point's position in the global field of view image based on the global field of view camera. The calibration parameters are used to calculate the target point relative to the UAV body coordinate system. The spatial line of sight, combined with the gimbal coordinate system Coordinate system with deformation displacement measurement camera Based on the installation relationship, the desired rotation angle of the gimbal is generated, so that the deformation displacement measurement camera points to the target area;

[0037] Let the homogeneous coordinates of the target point in the global field-of-view camera image be... The intrinsic parameter matrix of the global field-of-view camera is The target point is in the global field-of-view camera coordinate system. The downward line of sight can be written as:

[0038] ;

[0039] In the formula, For the target point at The downward line of sight vector, Let be the homogeneous coordinates of the target point in the global field-of-view camera image. The intrinsic parameter matrix of the global field-of-view camera; composed of fixed extrinsic parameters. Extracting the rotation matrix This line of sight can be converted to the drone's body coordinate system. :

[0040] ;

[0041] In the formula, For the target point at The downward line-of-sight vector; For the target point at The downward line of sight vector, It is a fixed external parameter The extracted rotation matrix is ​​based on Calculate the desired azimuth and pitch angles of the gimbal; after the gimbal completes coarse pointing, the deformation displacement measurement camera acquires local high-resolution images and achieves precise locking through target feature localization or template matching. Step S3 further includes: after the UAV arrives at the measurement position, the following data are collected at the initial moment: the top platform relative motion measurement camera group takes a fixed point image; the top global field of view camera takes an overall image of the structure under test; the lower gimbal angle measurement camera takes a gimbal marker block image; the deformation displacement measurement camera on the gimbal takes a local image of the target point under test; the system records the platform pose, the gimbal's true attitude, the target point image coordinates, and the control point image coordinates at the initial moment.

[0042] As a preferred embodiment, the steps for calculating the true six-degree-of-freedom pose of the gimbal relative to the UAV platform specifically include:

[0043] Using pre-calibrated gimbal rotation angle to measure fixed external parameters between the camera and the drone body And the fixed external parameters between the gimbal marker block coordinate system and the gimbal coordinate system. ;

[0044] Gimbal angle measurement camera fixed under the drone body Continuous observation of the marker block fixed to the gimbal. ;

[0045] The extrinsic parameters of the camera measuring the gimbal angle relative to the gimbal marker block at the current moment are solved using a visual pose calculation method. Then, the true pose of the gimbal coordinate system relative to the UAV body coordinate system is calculated. ;

[0046] because UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters For the gimbal coordinate system To the coordinate system of the gimbal marker block Given fixed extrinsic parameters, the gimbal's current pose relative to the drone body can be written as:

[0047] ;

[0048] In the formula, UAV body coordinate system gimbal coordinate system at the current moment The extrinsic matrix, For the gimbal coordinate system To the coordinate system of the gimbal marker block Fixed extrinsic parameters UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters The external parameter of the camera is the angle measurement of the gimbal marker block relative to the gimbal rotation at the current moment. The solution can be obtained using existing PnP methods;

[0049] Deformation displacement measurement camera coordinate system With the gimbal coordinate system The fixed extrinsic parameters between them are The current pose of the deformation displacement measurement camera relative to the drone body is:

[0050] ;

[0051] In the formula, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment extrinsic parameter matrix; visual measurement results It can directly replace the angle of the gimbal encoder, or be used to correct the actual pointing error caused by the gimbal encoder, control hysteresis, installation error, and vibration coupling of the gimbal structure below.

[0052] As a preferred approach, the unified imaging inverse calculation model solves for the true deformation and displacement of the target point by minimizing the reprojection error. :

[0053] ;

[0054] In the formula, The image coordinates of the target point at the current time. For perspective projection functions, The intrinsic parameter matrix of the deformation displacement measurement camera. Let be the world coordinates of the target point at the initial moment. For the actual deformation displacement of the target point, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment The extrinsic parameter matrix; This is the extrinsic parameter matrix of the camera coordinate system relative to the world coordinate system at the current moment, which is derived from the relative motion matrix of the UAV platform. and the true position of the gimbal Combined structure:

[0055] ;

[0056] In the formula, The extrinsic parameter matrix of the camera coordinate system relative to the world coordinate system at the current moment for measuring deformation and displacement. UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix;

[0057] When measuring multiple target points, the imaging inverse calculation model is extended to a multi-point joint least squares model to jointly solve the true deformation displacement of all target points, thereby improving the stability of the inverse calculation.

[0058] As a preferred option: In the process of back-calculating the actual deformation displacement of the target point in the target displacement back-calculation model described in step S8, structural geometric constraints, known displacement direction constraints, or structural three-dimensional model constraints are introduced.

[0059] Let the world coordinates of the target point at the initial moment be... The actual displacement at the current moment is Then the world coordinates of the target point at the current moment are:

[0060] ;

[0061] In the formula, The true three-dimensional displacement of the target point to be determined; Let the target point be its world coordinates at the initial moment. The world coordinates of the target point at the current moment; the coordinate system of the deformation displacement measurement camera at the current moment, based on the platform's relative motion and the gimbal's true pose. Relative to the world coordinate system The extrinsic parameter matrix can be written as:

[0062] ;

[0063] In the formula, The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic matrix, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix;

[0064] The current image coordinates of the target point satisfy the perspective projection relationship:

[0065] ;

[0066] In the formula, As a scale factor, For deformation displacement measurement camera The intrinsic parameter matrix, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. Let the target point be the world coordinates. The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic parameter matrix.

[0067] By uniformly incorporating the platform's relative motion, the gimbal's actual attitude, and the target point image observations, a target displacement inverse calculation optimization model can be constructed:

[0068] ;

[0069] In the formula, For perspective projection functions, The true three-dimensional displacement of the target point is to be determined. Let the target point be its world coordinates at the initial moment. For deformation displacement measurement camera The intrinsic parameter matrix, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. Let the target point be the world coordinates. The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic parameter matrix.

[0070] Another technical solution of the present invention is the application of the virtual camera array compensation method for large-scale deformation displacement measurement in bridge deflection measurement. Its unique feature is the introduction of structural geometric constraints, displacement direction constraints, or structural model constraints. For bridge deflection measurement, the target point displacement is constrained to the world coordinate system. Vertical direction in:

[0071] ;

[0072] In the formula, The true three-dimensional displacement of the target point is to be determined. Let be the vertical unit vector in the world coordinate system W. The vertical displacement to be determined is denoted as .

[0073] The optimization model can now be simplified to:

[0074] ;

[0075] In the formula, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. For perspective projection functions, Let the target point be its world coordinates at the initial moment. For deformation displacement measurement camera The intrinsic parameter matrix, The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic matrix, This is the vertical displacement. World coordinate system The vertical unit vector in the vector.

[0076] Another technical solution of the present invention is the large-range deformation displacement measuring device for virtual camera array compensation for implementing any of the foregoing methods, characterized in that it includes:

[0077] Unmanned aerial vehicle (UAV) platform;

[0078] The platform relative motion measurement camera group is rigidly mounted on the top of the UAV body to form at least two equivalent observation angles to observe stationary control points in space;

[0079] A global field-of-view camera module is rigidly mounted on top of the UAV body to acquire large-scale images of the area to be tested;

[0080] The gimbal is installed below the drone body;

[0081] A deformation displacement measuring camera is fixedly mounted on the pan-tilt unit to acquire high-resolution images of the target point;

[0082] The marker block is fixedly installed on the pan-tilt unit or on a component rigidly connected to the deformation displacement measuring camera;

[0083] A gimbal angle measurement camera is installed on the platform below the UAV body and rigidly connected to the UAV body. It is used to observe the marker block to calculate the true pose of the gimbal.

[0084] The data processing module is connected to the platform relative motion measurement camera group, the global field of view camera module, the gimbal angle measurement camera, and the deformation displacement measurement camera, respectively, and is used to execute the steps of the method.

[0085] Preferably, the platform relative motion measurement camera group consists of a physical camera and an optical beam splitter set at its front end. The optical beam splitter is a semi-reflective semi-transparent film, a beam splitter prism, a reflector, or a multifaceted mirror, used to simultaneously image the optical paths of stationary control points from different observation directions onto the same imaging sensor of the physical camera.

[0086] The platform relative motion measurement camera group at the initial moment At time t, stationary control points or immobile feature points are observed in space. Since the coordinates of these control points in the world coordinate system W do not change, the changes in their image coordinates are mainly caused by the translation and rotation of the UAV body coordinate system B. Multiple virtual viewpoints formed by the platform's relative motion measurement camera group are used. Treating the UAV as multiple equivalent cameras, the initial pose of the UAV body is first calculated from the initial observations at the initial moment, and then the coordinate system of the UAV body is calculated from... arrive Six degrees of freedom relative motion;

[0087] Let the first A stationary control point in the world coordinate system The homogeneous coordinates in the middle are In its first A virtual perspective At the moment The homogeneous coordinates of the image are Then the camera projection relationship can be written as:

[0088] ;

[0089] In the formula, As a scale factor, For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Represents the image coordinate system;

[0090] Combining the transformation relationship between the virtual camera coordinate system and the UAV body coordinate system, we can obtain:

[0091] ;

[0092] In the formula, World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Coordinate systems of each virtual camera in the platform relative motion measurement camera group With UAV body coordinate system The fixed extrinsic parameters between them Let be the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix.

[0093] Get the current time of the first The projection relationship of stationary control points under a virtual viewpoint is calculated by minimizing the reprojection error. :

[0094] ;

[0095] In the formula, This refers to the six-degree-of-freedom relative motion matrix of the unmanned aerial vehicle platform relative to the initial moment. For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in It is a virtual camera coordinate system With UAV body coordinate system The fixed extrinsic parameters between them It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix.

[0096] As a preferred embodiment, the global field of view camera module is composed of multiple wide-angle cameras spliced ​​together. After calibration, the multiple cameras are unified to the global field of view coordinate system or the UAV body coordinate system to form a larger field of view coverage.

[0097] The system determines the target point's position based on its pixel location in the global field of view image and the global field of view camera. The calibration parameters are used to calculate the target point relative to the UAV body coordinate system. The spatial line of sight, combined with the gimbal coordinate system Coordinate system with deformation displacement measurement camera Based on the installation relationship, the desired rotation angle of the gimbal is generated, so that the deformation displacement measurement camera points to the target area;

[0098] Let the homogeneous coordinates of the target point in the global field-of-view camera image be... The intrinsic parameter matrix of the global field-of-view camera is The target point is in the global field-of-view camera coordinate system. The downward line of sight can be written as:

[0099] ;

[0100] In the formula, For the target point at The downward line of sight vector, Let be the homogeneous coordinates of the target point in the global field-of-view camera image. The intrinsic parameter matrix of the global field-of-view camera; composed of fixed extrinsic parameters. Extracting the rotation matrix This line of sight can be converted to the drone's body coordinate system. :

[0101] ;

[0102] In the formula, For the target point at The line-of-sight vector below; according to It can calculate the desired azimuth and pitch angles of the gimbal. After the gimbal completes coarse pointing, the deformation displacement measurement camera acquires local high-resolution images and achieves precise locking through target feature localization or template matching.

[0103] Preferably, the marker block is provided with cooperative marker points, coded points, dots, checkerboard patterns, AprilTag, ArUco markers, or actively emitting markers; the actively emitting markers are infrared markers or coded LED arrays, used to improve the recognition stability under low light or complex background conditions.

[0104] As a preferred embodiment, the step of obtaining the true attitude of the gimbal coordinate system Y relative to the UAV body coordinate system B through visual pose calculation includes:

[0105] set up UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters For the gimbal coordinate system To the coordinate system of the gimbal marker block The fixed extrinsic parameters are the gimbal's current pose relative to the drone body.

[0106] ;

[0107] In the formula, UAV body coordinate system gimbal coordinate system at the current moment The extrinsic matrix, For the gimbal coordinate system To the coordinate system of the gimbal marker block Fixed extrinsic parameters UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters It is the extrinsic parameter of the camera relative to the gimbal angle measurement block at the current moment. The problem can be solved using existing PnP methods:

[0108] Deformation displacement measurement camera coordinate system With the gimbal coordinate system The fixed extrinsic parameters between them are The current pose of the deformation displacement measurement camera relative to the drone body is:

[0109] ;

[0110] In the formula, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment extrinsic parameter matrix; visual measurement results It can directly replace the angle of the gimbal encoder, or be used to correct the actual pointing error caused by the gimbal encoder, control hysteresis, installation error and vibration coupling of the gimbal structure below;

[0111] The device also includes a synchronization triggering module, which is used to synchronously trigger the acquisition of the platform relative motion measurement camera group, global field of view camera module, gimbal angle measurement camera and deformation displacement measurement camera to eliminate time synchronization errors between multiple cameras.

[0112] Overall structure of the system of this invention:

[0113] The device of this invention mainly includes: an unmanned aerial vehicle (UAV) platform, a platform relative motion measurement camera group, a global field-of-view camera module, a gimbal, a deformation displacement measurement camera, a gimbal fixed marker block, and a gimbal rotation angle measurement camera; the system also includes a synchronization trigger module and a data processing module;

[0114] The drone platform is a multi-rotor, fixed-wing, compound-wing, or tethered drone, used to carry measurement payloads;

[0115] The platform relative motion measurement camera group is rigidly mounted on the top of the UAV body to observe stationary control points within the field of view. The camera group consists of a single physical camera and optical elements (semi-reflective and semi-transparent film, beam splitter, reflector or multifaceted mirror) to form multiple equivalent virtual viewpoints in different observation directions, thereby solving the six degrees of freedom relative motion of the UAV platform.

[0116] The global field-of-view camera module is rigidly mounted on the top of the UAV body to capture large-scale images of the structure under test, and to complete target area recognition and gimbal coarse pointing control;

[0117] Gimbal and Deformation Displacement Measurement Camera: The gimbal is mounted under the drone body, and the deformation displacement measurement camera is fixed to the gimbal to acquire local high-resolution images of the target point.

[0118] Gimbal marker block and gimbal angle measurement camera: The marker block is fixed to the gimbal; the gimbal angle measurement camera is installed under the UAV body and rigidly connected to the body, used to observe the marker block, and obtain the true angle and attitude of the gimbal through visual pose calculation.

[0119] Coordinate system definition and calibration relationship To accurately describe the measurement model, this invention defines the following coordinate systems: world coordinate system W, UAV body coordinate system B, and platform relative motion measurement virtual camera coordinate system. Global field of view camera coordinate system The coordinate system of the gimbal is Y, and the coordinate system of the deformation displacement measurement camera is Y. Pan-tilt angle measurement camera coordinate system The coordinate system Q of the gimbal marker block; before measurement, the intrinsic parameters of all cameras and the extrinsic parameters between each coordinate system must be calibrated, such as... , , , , wait.

[0120] Key technology implementation:

[0121] Single-camera equivalent multi-field structure:

[0122] To reduce the load weight, the platform's relative motion measurement camera group can use a single physical camera; a semi-reflective plate, a beam splitter, etc., can be placed in front of it to allow light from different directions to enter the same sensor; for example... Figure 2 In the dual-field-of-view structure shown, one path of light is transmitted through a semi-reflective sheet, and the other path is reflected, corresponding to two virtual viewing angles respectively; after calibration, each virtual viewing angle has an independent equivalent intrinsic parameter. and fixed external parameters relative to the body It achieves the function of "single camera, multiple perspectives, and simultaneous imaging";

[0123] Visual measurement of the true pointing angle of the gimbal:

[0124] This invention does not rely on a gimbal encoder, but instead directly measures the actual pointing of the gimbal using a visual method; for example... Figure 5 As shown, the gimbal angle measurement camera is located beneath the drone's fuselage. Continuously observe the marker blocks on the gimbal. ; Using the formula:

[0125] ; In the formula, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment The extrinsic parameter matrix. Visual measurement results. It can directly replace the angle of the gimbal encoder, or be used to correct the actual pointing error caused by the gimbal encoder, control hysteresis, installation error, and vibration coupling of the gimbal structure below.

[0126] Inverse model for target displacement of platform motion compensation:

[0127] The core of this invention lies in modeling the target displacement back-calculation as a unified optimization problem; the real-world coordinates of the target point... Its imaging process is affected by the movement of the drone platform. And the true pointing of the gimbal The combined effects, namely, ;

[0128] By minimizing the reprojection error (Equation 11), this model can obtain the image displacement of the target point. In this process, the motion of the image point caused by three factors—platform motion, gimbal pointing change, and target actual displacement—is simultaneously separated, thereby obtaining high-precision actual deformation displacement. When structural constraints are known (such as the vertical deflection of a bridge), the prior constraints shown in formula (13) can be introduced to further improve the robustness and accuracy of the solution.

[0129] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0130] (1) High-precision displacement measurement capability: Through a dual compensation mechanism of "platform motion visual self-compensation" and "gimbal pointing visual precision measurement", the influence of UAV's own disturbance and gimbal control error on target imaging is eliminated from the source. Compared with methods that rely solely on UAV flight control data and gimbal encoders, this invention can significantly improve displacement measurement accuracy under long-distance and long-focal-length measurement conditions, reaching millimeter-level or even sub-pixel-level accuracy.

[0131] (2) Unity of wide coverage and high resolution: By employing a hierarchical working mode of "global field-of-view visual guidance" and "local telephoto measurement," the system cleverly resolves the contradiction between wide-area search and high-resolution measurement. The system first uses a wide-field-of-view camera to quickly lock onto the target area, and then guides a telephoto camera to perform precise measurements, combining wide-area coverage with detail resolution capabilities, greatly improving on-site operational efficiency.

[0132] (3) Reduced dependence on hardware performance: This invention reduces reliance on high-precision, high-cost inertial navigation systems (IMU / GNSS) and gimbal encoders through algorithmic and system integration innovations. By utilizing visual measurement, a "soft measurement" method, key motion compensation and pointing parameters can be obtained, which helps to reduce system hardware costs and improve system environmental adaptability (especially in GNSS-denied environments) while ensuring accuracy.

[0133] (4) Lightweight and highly integrated system: The proposed "single-camera equivalent multi-field of view" structure replaces multiple independent cameras with a set of optical components and a physical camera. While achieving multi-directional observation, it significantly reduces the weight, size and power consumption of the payload, and completely eliminates the synchronization triggering error between multiple cameras. It is particularly suitable for small rotorcraft UAV platforms with limited payload capacity.

[0134] (5) Strong engineering adaptability: This method is compatible with both artificial markers and natural feature points, and can be flexibly applied to various engineering scenarios such as bridges, slopes, and towers. By introducing structural model constraints and prior knowledge of displacement direction, the algorithm can still maintain high stability and robustness under complex backgrounds and partial occlusion conditions.

[0135] (6) It can compensate for the impact of UAV hovering disturbances, attitude changes and small-range drift on target image displacement, and improve the displacement measurement accuracy of UAV platform.

[0136] (7) It can reduce the reliance on high-precision flight control, GNSS and high-precision gimbal encoders, and obtain key motion parameters and the true attitude of the gimbal through visual self-measurement.

[0137] (8) It can simultaneously perform large-scale target search and local high-resolution deformation measurement, solving the problems of small field of view and difficulty in target acquisition of telephoto cameras.

[0138] (9) It is applicable to long-distance, non-contact measurement scenarios and has strong engineering adaptability in dangerous, complex or difficult-to-deploy areas.

[0139] (10) The system has a compact structure and can adopt a single-camera equivalent multi-field-of-view scheme to reduce load weight and synchronization error. It can also be expanded into a multi-independent camera scheme according to measurement accuracy and engineering cost.

[0140] 11. Combining artificial markers, natural features, structural models, and control point coordinates can improve the feasibility of complex engineering projects on-site. Attached Figure Description

[0141] Figure 1 This is a schematic diagram of the overall system structure of the present invention;

[0142] Figure 2 This is a schematic diagram illustrating the principle of the single-camera equivalent multi-field structure in this invention;

[0143] Figure 3 This is a schematic diagram illustrating the relative motion measurement principle of the platform of this invention.

[0144] Figure 4 This is a schematic diagram illustrating the principle of the global field-of-view guiding gimbal telephoto camera pointing towards the target in this invention.

[0145] Figure 5 This is a schematic diagram illustrating the principle of visual measurement of the true rotation angle of the gimbal in this invention.

[0146] Figure 6 This is a flowchart illustrating the back-calculation process of the target displacement for motion compensation on the platform of this invention. Detailed Implementation

[0147] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0148] Example 1: Overall system structure and coordinate system definition.

[0149] like Figure 1 As shown, the device of the present invention mainly includes: a UAV platform, platform relative motion, a measurement camera group, a global field-of-view camera module, a gimbal, a deformation displacement measurement camera, a gimbal fixed marker block, and a gimbal angle measurement camera; the system also includes a synchronization trigger module and a data processing module (not shown in the figure); the UAV platform is used to carry all payloads; the platform relative motion measurement camera group and the global field-of-view camera module are rigidly installed on the top of the UAV body; the gimbal angle measurement camera and the gimbal are installed on the platform below the UAV body; the deformation displacement measurement camera and the marker block are installed on the gimbal.

[0150] To establish an accurate measurement model, this invention defines the following coordinate system:

[0151] World coordinate system The coordinate system of the field control points, the structural coordinate system, or the initial measurement coordinate system shall be used as the reference.

[0152] UAV body coordinate system : A load frame that is fixed to the drone platform or rigidly connected to the drone body;

[0153] Platform relative motion measurement virtual camera coordinate system : Indicates the first [camera group] formed by the platform's relative motion measurement. A virtual viewpoint coordinate system.

[0154] Global field of view camera coordinate system : The coordinate system of the global field-of-view camera or global stitching camera installed on top of the drone body;

[0155] gimbal coordinate system : Fixed to the gimbal rotation mechanism;

[0156] Deformation displacement measurement camera coordinate system Long-focal-length measuring camera fixed to the pan-tilt head;

[0157] Pan-tilt angle measurement camera coordinate system It is installed on the platform below the drone body and is rigidly connected to the drone body;

[0158] Gimbal marker block coordinate system : Fixed to the gimbal marker block;

[0159] Before measurement, it is necessary to complete the calibration of all camera intrinsic parameters (focal length, principal point, distortion parameters, etc.) and fixed extrinsic parameters between each coordinate system, for example... (External parameters between virtual camera and machine body) (External parameters between the angle measurement camera and the machine body) (External parameters between the deformation measurement camera and the gimbal) etc.

[0160] Example 2: Measurement of relative motion of the platform.

[0161] like Figure 3 As shown, the platform's relative motion measurement camera group consists of a single physical camera and a front-facing optical element, forming multiple virtual viewpoints. These virtual perspective observations show stationary control points in the observation space. ;

[0162] Let the first A stationary control point in the world coordinate system The homogeneous coordinates in the middle are In its first A virtual perspective At the moment The homogeneous coordinates of the image are Then the camera projection relationship can be written as:

[0163] ;

[0164] In the formula, As a scale factor, For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Represents the image coordinate system;

[0165] Combining the transformation relationship between the virtual camera coordinate system and the UAV body coordinate system, we can obtain:

[0166] ;

[0167] In the formula, It is a virtual camera coordinate system With UAV body coordinate system The fixed extrinsic parameters between them It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix;

[0168] We can get the current time of the first... The projection relationship of stationary control points under a virtual viewpoint is calculated by minimizing the reprojection error. :

[0169] ;

[0170] In the formula, This refers to the six-degree-of-freedom relative motion matrix of the unmanned aerial vehicle platform relative to the initial moment. For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in It is a virtual camera coordinate system With UAV body coordinate system The fixed extrinsic parameters between them It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix;

[0171] Let the homogeneous coordinates of the target point in the global field-of-view camera image be... The intrinsic parameter matrix of the global field-of-view camera is The target point is in the global field-of-view camera coordinate system. The downward line of sight can be written as:

[0172] ;

[0173] In the formula, For the target point at The downward line of sight vector, Let be the homogeneous coordinates of the target point in the global field-of-view camera image. This is the intrinsic parameter matrix of the global field-of-view camera; By fixed external parameters Extracting the rotation matrix This line of sight can be converted to the drone's body coordinate system. :

[0174] ;

[0175] In the formula, For the target point at The downward line of sight vector, To fix external parameters Extract the rotation matrix; based on It can calculate the desired azimuth and pitch angles of the gimbal; after the gimbal completes coarse pointing, the deformation displacement measurement camera acquires local high-resolution images and achieves precise locking through target feature localization or template matching. This refers to the six-degree-of-freedom relative motion matrix of the unmanned aerial vehicle platform relative to the initial moment.

[0176] because UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters For the gimbal coordinate system To the coordinate system of the gimbal marker block Given fixed extrinsic parameters, the gimbal's current pose relative to the drone body can be written as:

[0177] ;

[0178] In the formula, UAV body coordinate system gimbal coordinate system at the current moment The extrinsic matrix, For the gimbal coordinate system To the coordinate system of the gimbal marker block Fixed extrinsic parameters UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters The external parameter of the camera is the angle measurement of the gimbal marker block relative to the gimbal rotation at the current moment. The solution can be obtained using existing PnP methods.

[0179] Furthermore, the coordinate system of the deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them are Therefore, the true pose of the deformation displacement measurement camera relative to the drone body at the current moment is:

[0180] ;

[0181] In the formula, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment extrinsic parameter matrix; visual measurement results It can directly replace the angle of the gimbal encoder, or be used to correct the actual pointing error caused by the gimbal encoder, control hysteresis, installation error, and vibration coupling of the gimbal structure below.

[0182] Let the world coordinates of the target point at the initial moment be... The actual displacement at the current moment is Then the world coordinates of the target point at the current moment are:

[0183] ;

[0184] In the formula, The true 3D displacement of the target point is to be determined; based on the relative motion of the platform and the true pose of the gimbal, the deformation displacement is measured in the camera coordinate system at the current moment. Relative to the world coordinate system The extrinsic parameter matrix can be written as:

[0185] ;

[0186] In the formula, The extrinsic parameter matrix of the camera coordinate system relative to the world coordinate system at the current moment for measuring deformation and displacement. UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix;

[0187] The current image coordinates of the target point satisfy the perspective projection relationship:

[0188] ;

[0189] In the formula, As a scale factor, For deformation displacement measurement camera The intrinsic parameter matrix, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. Let the target point be the world coordinates. The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic parameter matrix;

[0190] By uniformly incorporating the platform's relative motion, the gimbal's actual attitude, and the target point image observations, a target displacement inverse calculation optimization model can be constructed:

[0191] ;

[0192] in, For perspective projection functions, The true three-dimensional displacement of the target point is to be determined. Let the target point be its world coordinates at the initial moment. For deformation displacement measurement camera The intrinsic parameter matrix, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. Let the target point be the world coordinates. The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic parameter matrix; if multiple target points are tracked simultaneously, equation (11) can be extended to a multi-point joint least squares model to improve the stability of the inverse calculation.

[0193] In engineering applications, target points are typically located on known structural surfaces, known measurement planes, or known components. In such cases, structural geometric constraints, displacement direction constraints, or structural model constraints can be introduced to improve the stability of the inverse calculation. For example, in bridge deflection measurement, the displacement of the target point can be constrained to the world coordinate system. Vertical direction in:

[0194] ;

[0195] In the formula, The true three-dimensional displacement of the target point is to be determined. World coordinate system The vertical unit vector in the middle, Let the vertical displacement be the unknown quantity; the optimization model can then be simplified to:

[0196] ;

[0197] In the formula, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. For perspective projection functions, Let the target point be its world coordinates at the initial moment. For deformation displacement measurement camera The intrinsic parameter matrix, The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic matrix, This is the vertical displacement. World coordinate system The vertical unit vector in the vector.

[0198] Example 3: Typical application scenario.

[0199] Vertical deflection measurement of bridge: The UAV flies to the side of the bridge, and the platform relative motion measurement camera group observes the fixed points on the shore or pier to calculate the platform disturbance; the global field of view camera identifies the mid-span position and guides the telephoto camera to point to the point to be measured; the gimbal rotation measurement camera corrects the gimbal pointing error in real time; finally, the vertical deflection of the bridge is calculated by combining the target point image displacement and platform motion compensation.

[0200] High slope surface displacement measurement: The UAV cruises along the slope, and the platform's relative motion measurement camera group observes the stationary point on the distant stable rock mass; the global field of view camera identifies cracks or unstable rock masses; the known slip direction constraint of the slope is introduced into the displacement back calculation, so as to obtain the accurate displacement component of the target point along the slope surface.

[0201] Tower lateral sway measurement: The UAV hovers at a safe distance from the tower, and the global field-of-view camera guides the telephoto camera to lock onto key points on the tower. By compensating for the UAV's own motion and gimbal pointing error, the system can accurately measure the lateral sway amplitude and dynamic vibration curve of the tower top under wind load and other effects.

[0202] Example 4: Measurement of vertical deflection of bridge.

[0203] This embodiment uses the measurement of the mid-span vertical deflection of a long-span bridge as an example to illustrate the specific application of the method and device of the present invention.

[0204] 1. Measurement preparation and system calibration:

[0205] Three or more infrared LED active emitting markers are deployed on the stable bank or piers near the bridge as stationary control points; the device of this invention is mounted on a hexacopter UAV; before takeoff, the intrinsic parameters of all cameras are calibrated according to step S2; using a high-precision three-dimensional control field, the equivalent intrinsic parameters of each virtual viewpoint in the platform relative motion measurement camera group are calibrated. and its external parameters with organism B ; Calibrate the external parameters of the global field-of-view camera and the aircraft B ; Calibrate the gimbal angle measurement camera and body B external parameters ; Calibrate the external parameters of the deformation displacement measuring camera and the gimbal Y ; Calibrate the external parameters of the gimbal marker block coordinate system Q and the gimbal Y. ;

[0206] 2. Flight and Initial Alignment:

[0207] The drone hovers approximately 50 meters to the side of the bridge; at this point, step S3 is executed; the relative motion measurement camera group on the top platform (using a single camera + three-part prism structure to form three virtual perspectives) captures three infrared control points, and the initial image coordinates are recorded; the global field-of-view camera (equipped with a 20mm lens) captures an overall image of the bridge, and the operator (or algorithm) selects a natural texture feature at the lower edge of the middle span of the bridge as the target point to be measured in the image, and records its initial coordinates; the gimbal angle measurement camera below captures the AprilTag marker block on the gimbal, and calculates the initial gimbal attitude; the telephoto deformation displacement measurement camera (equipped with a 120mm lens) acquires the initial high-resolution image of the target point.

[0208] 3. Dynamic measurement and compensation:

[0209] When the bridge undergoes deflection deformation under vehicle load, the system begins continuous measurement (steps S4-S8):

[0210] S4 (Guidance): Although the target point may move slightly, it remains within the field of view of the global field-of-view camera. The system calculates the new position of the target point in the global map in real time and generates a fine-tuned gimbal expectation angle to ensure that the telephoto camera is always "locked" on the target;

[0211] S5 (Gimbal Precision Measurement): The gimbal rotation measurement camera photographs the gimbal marker block at a frequency of 100Hz; the true attitude of the gimbal at each moment is calculated using formula (6). This step effectively compensates for the minor vibrations and control hysteresis of the gimbal in the wind; for example, if the gimbal encoder shows a rotation angle of 0°, but visual calculations reveal an actual pitch error of 0.05°, this error will be accurately captured.

[0212] S6 (Platform Motion Compensation): The platform relative motion measurement camera group also photographs three stationary control points at a frequency of 100Hz; the relative motion of the UAV platform is calculated using formula (3). Assuming the drone is affected by a gust of wind, shifting 2cm to the right and producing a 0.1° roll, this six-degree-of-freedom motion will be precisely calculated.

[0213] S7 (Image Displacement): A telephoto camera acquires images at 100Hz; through subpixel template matching, the displacement sequence of the target point in the image is obtained. The displacement includes three components: the actual deflection of the bridge, the motion of the UAV platform, and the change in the gimbal's pointing direction.

[0214] S8 (Displacement Inverse Calculation): The data processing module will calculate the displacement at each moment. , and Substituting into the optimization model of formula (13); the model knows that the direction of bridge deflection is vertical. The quantity to be determined is the vertical displacement. By minimizing reprojection error, the algorithm accurately separates the bridge's true vertical deflection curve from complex image displacements; the final measurement results are compared with those of a high-precision laser displacement meter installed under the bridge, with an error of less than 0.5 mm.

[0215] Synergistic effect analysis: In this embodiment, no single compensation method can complete the task independently; platform motion compensation alone cannot eliminate the gimbal's pointing error; gimbal pointing precision measurement alone cannot eliminate the influence of the overall motion of the UAV; however, this invention generates a synergistic effect of "1+1+1>3" by fusing the data of the two together with the target image displacement, thus achieving accurate measurement of micron-level (millimeter-level on the object side) deformation on a mobile platform.

[0216] Example 5: Surface displacement measurement of high slope.

[0217] This embodiment is basically the same as Embodiment 1, except that the measurement scenario and the introduced constraints are different.

[0218] In a high slope monitoring scenario, a drone hovers along the front of the slope. A global field-of-view camera identifies multiple pre-painted artificial markers on the slope as targets to be measured. Since the surface geometry of the slope is known (e.g., obtained through pre-scanning by a lidar-equipped drone), surface model constraints can be introduced in step S8. The optimization model no longer assumes free displacement but instead constrains the displacement of the target to be measured to the slope surface. This means the algorithm can calculate the three-dimensional displacement components of the target point sliding down the slope and deviating from the normal direction using only monocular images. The introduction of this model constraint greatly improves the feasibility and accuracy of monocular vision in measuring displacement on complex curved surfaces.

[0219] Alternative implementation methods:

[0220] Multi-independent camera platform motion measurement: When the load allows, the platform relative motion measurement camera group can consist of two or more independent physical cameras, each observing the stationary point in different directions. This approach has a more straightforward structure, but requires higher synchronization and increases cost and weight.

[0221] Multi-camera stitching global field of view: For ultra-large structures (such as dams), an array of 3 to 5 wide-angle cameras can be used to form a virtual global camera with an ultra-large field of view after stitching and calibration, which can be used for guidance.

[0222] Active light-emitting marker blocks: In low-light environments such as at night or in tunnels, the marker blocks on the pan-tilt unit can use an infrared LED array, and the pan-tilt unit angle measurement camera is equipped with a corresponding filter to stably obtain high-contrast marker images, ensuring the reliability of pan-tilt unit pointing measurement.

[0223] Structural model constraints: For structures with known CAD models (such as wind turbine towers), the model can be used as a constraint. The model is aligned with the physical object using a global field-of-view image (registration), and then the displacement of local images measured by a telephoto camera is projected onto the model coordinate system to obtain physically meaningful displacements (such as the bending deformation curve of the tower).

[0224] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be covered by the claims of the present invention.

Claims

1. A method for measuring large-range deformation displacement with virtual camera array compensation, characterized in that, Includes the following steps: S1. Install a platform relative motion measurement camera group and a global field of view camera above the UAV body, wherein the platform relative motion measurement camera group is used to form at least two equivalent observation angles; S2. Install a gimbal under the drone body, and install a deformation displacement measuring camera and a marker block on the gimbal; install a gimbal rotation measuring camera on the platform under the drone body; calibrate the internal parameters of each camera, and calibrate the external parameter relationships between each component; S3. Observe the stationary control point in the space through the equivalent observation angle of the platform relative motion measurement camera group, and calculate the pose of the UAV platform at the initial moment; Data from each camera is collected at the initial moment, and the initial state is recorded; S4. Acquire a large-scale image of the area to be measured through the global field-of-view camera, and guide the deformation displacement measuring camera on the gimbal to point to the target point according to the position of the target point in the global image; S5. By observing the marker block through the gimbal angle measurement camera, the true six-degree-of-freedom pose of the gimbal relative to the UAV platform is calculated to obtain the true pointing attitude of the deformation displacement measurement camera. The steps for calculating the true six-degree-of-freedom pose of the gimbal relative to the UAV platform specifically include: Using pre-calibrated gimbal rotation angle to measure fixed external parameters between the camera and the drone body And the fixed external parameters between the gimbal marker block coordinate system and the gimbal coordinate system. ; Gimbal angle measurement camera fixed to the underside of the drone Continuous observation of the marker block fixed to the gimbal. ; The extrinsic parameters of the camera measuring the gimbal angle relative to the gimbal marker block at the current moment are solved using a visual pose calculation method. Then, the coordinate system Y of the gimbal is calculated relative to the coordinate system of the UAV body. True position ; because UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters For the gimbal coordinate system To the coordinate system of the gimbal marker block Given fixed extrinsic parameters, the gimbal's current pose relative to the drone body can be written as: ； In the formula, UAV body coordinate system gimbal coordinate system at the current moment The extrinsic matrix, For the gimbal coordinate system To the coordinate system of the gimbal marker block Fixed extrinsic parameters UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters The external parameter of the camera is the angle measurement of the gimbal marker block relative to the gimbal rotation at the current moment. The solution can be obtained using existing PnP methods; Deformation displacement measurement camera coordinate system With the gimbal coordinate system The fixed extrinsic parameters between them are The current pose of the deformation displacement measurement camera relative to the drone body is: ； In the formula, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment The extrinsic parameter matrix; Visual measurement results It can directly replace the angle of the gimbal encoder, or be used to correct the actual pointing error caused by the gimbal encoder, control hysteresis, installation error and vibration coupling of the gimbal structure below; The unified imaging inverse calculation model solves for the true deformation and displacement of the target point by minimizing the reprojection error. : ； In the formula, The image coordinates of the target point at the current time. For perspective projection functions, The intrinsic parameter matrix of the deformation displacement measurement camera. Let be the world coordinates of the target point at the initial moment. For the actual deformation displacement of the target point, For the corresponding homogeneous coordinates, This is the extrinsic parameter matrix of the camera coordinate system relative to the world coordinate system at the current moment, which is derived from the relative motion matrix of the UAV platform. The deformation displacement measurement camera's current pose relative to the drone's body. Combined structure: ； In the formula, The extrinsic parameter matrix of the camera coordinate system relative to the world coordinate system at the current moment for measuring deformation and displacement. UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix; When measuring multiple target points, the imaging inverse calculation model is extended into a multi-point joint least squares model to jointly solve the true deformation displacement of all target points and improve the stability of the inverse calculation; S6. Based on the observation results of the platform relative motion measurement camera group, combined with the calibration parameters and the initial observation results, the six-degree-of-freedom relative motion of the UAV platform at the current moment relative to the initial moment is calculated; S7. The deformation displacement measurement camera acquires the image of the target point at the current time and matches it with the image at the initial time to extract the displacement of the target point image; S8. Substitute the relative motion of the UAV platform, the true pointing attitude of the gimbal, and the image displacement of the target point into the imaging inverse calculation model, calculate and eliminate the false displacements caused by the motion of the UAV platform and the pointing error of the gimbal, and obtain the true deformation displacement of the target point to be measured; In step S8, during the inverse calculation of the actual deformation displacement of the target point in the target displacement inverse calculation model, structural geometric constraints, known displacement direction constraints, or structural three-dimensional model constraints are introduced. Let the world coordinates of the target point at the initial moment be... The actual displacement at the current moment is Then the world coordinates of the target point at the current moment are: ； In the formula, The true three-dimensional displacement of the target point is to be determined. Let the target point be its world coordinates at the initial moment. The world coordinates of the target point at the current moment; the coordinate system of the deformation displacement measurement camera at the current moment, based on the platform's relative motion and the gimbal's true pose. Relative to the world coordinate system The extrinsic parameter matrix can be written as: ； In the formula, The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic matrix, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. World coordinate system UAV body coordinate system at the initial moment The transformation matrix; The current image coordinates of the target point satisfy the perspective projection relationship: ； In the formula, As a scale factor, For deformation displacement measurement camera The intrinsic parameter matrix, The homogeneous pixel coordinates of the target point in the deformation displacement measurement camera image are given. Let the target point be in homogeneous world coordinates. The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic parameter matrix; By uniformly incorporating the platform's relative motion, the gimbal's actual attitude, and the target point image observations, a target displacement inverse calculation optimization model can be constructed: ； In the formula, For perspective projection functions, The true three-dimensional displacement of the target point is to be determined. Let the target point be its world coordinates at the initial moment. For the corresponding homogeneous coordinates, For deformation displacement measurement camera The intrinsic parameter matrix, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic parameter matrix; If multiple target points are tracked simultaneously, the target displacement back-calculation optimization model can be extended to a multi-point joint least squares model to improve the stability of the back-calculation.

2. The method for measuring large-range deformation displacement with virtual camera array compensation according to claim 1, characterized in that, The platform relative motion measurement camera group includes a physical camera and an optical unit disposed at its front end. The optical unit is used to guide light paths from different directions to the same imaging sensor, and after optical path calibration, image separation, parameter demixing or feature decoupling, it is equivalent to the observation results of multiple virtual cameras with independent equivalent intrinsic parameters and fixed extrinsic parameters. The platform relative motion measurement camera group observes multiple non-collinear stationary control points through each equivalent virtual field of view, and uses a multi-camera pose calculation method to obtain the six degrees of freedom relative motion of the UAV platform. The imaging inverse calculation model constructs an optimization objective function based on the projection relationship, and solves the true deformation displacement of the target point by minimizing the reprojection error.

3. The method for measuring large-range deformation displacement with virtual camera array compensation according to claim 2, characterized in that, Platform relative motion measurement camera group at the initial moment and the current moment Observe stationary control points or immobile feature points in space respectively; control points are in the world coordinate system. The coordinates in the image remain unchanged, while the changes in the image coordinates are determined by the UAV's body coordinate system. The translation and rotation cause this; multiple virtual perspectives are formed by the camera group measuring the relative motion of the platform. Treating the UAV as multiple equivalent cameras, the initial pose of the UAV body is first calculated from the initial observations at the initial moment, and then the coordinate system of the UAV body is calculated from... arrive The six-degree-of-freedom relative motion, let the first... A stationary control point in the world coordinate system The homogeneous coordinates in the middle are In its first A virtual perspective At the moment The homogeneous coordinates of the image are Then the camera projection relationship can be written as: ； In the formula, As a scale factor, For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Represents the image coordinate system; Combining the transformation relationship between the virtual camera coordinate system and the UAV body coordinate system, we can obtain: ； In the formula, It is a virtual camera coordinate system With the UAV body coordinate system The fixed extrinsic parameters between them It is the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the transformation matrix from the world coordinate system W to the UAV body coordinate system B at the initial moment; Get the current time of the first The projection relationship of stationary control points under a virtual viewpoint is calculated by minimizing the reprojection error. : ； In the formula, This refers to the six-degree-of-freedom relative motion matrix of the unmanned aerial vehicle platform relative to the initial moment. For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment Image coordinates, For the first A stationary control point in the world coordinate system homogeneous coordinates in Virtual camera coordinate system With the UAV body coordinate system The fixed extrinsic parameters between them It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix; The steps for guiding deformation displacement measurement using a global field-of-view camera specifically include: determining the pixel position of the target point in the global field-of-view camera image. and the intrinsic parameter matrix of the global field of view camera Calculate the line-of-sight vector of the target point in the global field-of-view camera coordinate system. ; In the formula, For the target point at The downward line of sight vector, Let be the homogeneous coordinates of the target point in the global field-of-view camera image. This is the intrinsic parameter matrix of the global field-of-view camera; Using fixed extrinsic parameters between the global field-of-view camera and the UAV body Transform the line-of-sight vector into the UAV body coordinate system to obtain In the formula, For the target point at The downward line of sight vector, To fix external parameters Extract the rotation matrix; according to Calculate the desired azimuth and pitch angles of the gimbal so that the deformation displacement measuring camera is pointed at the target point to be measured; The global field of view camera mentioned in step S1 Rigidly mounted on top of the UAV, it is used to photograph a large area to be tested and identify structural contours, candidate test points, artificial targets, or natural features in the images; the system determines the target point's position based on the pixel location of the target point in the global field of view image and the global field of view camera. The calibration parameters are used to calculate the target point relative to the UAV body coordinate system. The spatial line of sight, combined with the gimbal coordinate system Coordinate system with deformation displacement measurement camera Based on the installation relationship, the desired rotation angle of the gimbal is generated, so that the deformation displacement measurement camera points to the target area; Let the homogeneous coordinates of the target point in the global field-of-view camera image be... The intrinsic parameter matrix of the global field-of-view camera is The target point is in the global field-of-view camera coordinate system. The downward line of sight can be written as: ； In the formula, For the target point at The downward line of sight vector, Let be the homogeneous coordinates of the target point in the global field-of-view camera image. This is the intrinsic parameter matrix of the global field-of-view camera; By fixed external parameters Extracting the rotation matrix This line of sight can be converted to the drone's body coordinate system. : ； In the formula, For the target point at The downward line of sight vector, For the target point at The downward line of sight vector, It is a fixed external parameter Extracted rotation matrix; according to Calculate the desired azimuth and pitch angles of the gimbal; after the gimbal completes coarse pointing, the deformation displacement measurement camera acquires local high-resolution images and achieves precise locking through target feature localization or template matching. Step S3 further includes: after the UAV arrives at the measurement position, the following data are collected at the initial moment: the top platform relative motion measurement camera group takes a fixed point image; the top global field of view camera takes an overall image of the structure under test; the lower gimbal angle measurement camera takes a gimbal marker block image; the deformation displacement measurement camera on the gimbal takes a local image of the target point under test; the system records the platform pose, the gimbal's true attitude, the target point image coordinates, and the control point image coordinates at the initial moment.

4. An application of the large-range deformation displacement measurement method with virtual camera array compensation according to claim 1 in bridge deflection measurement, characterized in that, By introducing structural geometric constraints, displacement direction constraints, or structural model constraints, the target point displacement is constrained to the world coordinate system for bridge deflection measurement. Vertical direction in: ； In the formula, The true three-dimensional displacement of the target point is to be determined. World coordinate system The vertical unit vector in the middle, For the vertical displacement to be determined, the optimization model can be simplified to: ； In the formula, The pixel coordinates of the target point in the camera image are measured to indicate the deformation displacement. For perspective projection functions, Let the target point be its world coordinates at the initial moment. For deformation displacement measurement camera The intrinsic parameter matrix, The coordinate system of the camera for measuring deformation displacement at the current moment. Relative to the world coordinate system The extrinsic matrix, This is the vertical displacement. World coordinate system The vertical unit vector in the middle, These are the corresponding homogeneous coordinates.

5. A large-range deformation displacement measuring device for implementing the virtual camera array compensation method according to any one of claims 1 to 3, characterized in that, include: Unmanned aerial vehicle (UAV) platform; The platform relative motion measurement camera group is rigidly mounted on the top of the UAV body to form at least two equivalent observation angles to observe stationary control points in space; A global field-of-view camera module is rigidly mounted on top of the UAV body to acquire large-scale images of the area to be tested; The gimbal is installed below the drone body; A deformation displacement measuring camera is fixedly mounted on the pan-tilt unit to acquire high-resolution images of the target point; The marker block is fixedly mounted on the pan-tilt unit or on a component rigidly connected to the deformation displacement measuring camera. A gimbal angle measurement camera is installed on the platform below the UAV body and rigidly connected to the UAV body. It is used to observe the marker block to calculate the true pose of the gimbal. The data processing module is connected to the platform relative motion measurement camera group, the global field of view camera module, the gimbal angle measurement camera, and the deformation displacement measurement camera, respectively, and is used to execute the steps of the method.

6. The large-range deformation displacement measuring device for virtual camera array compensation according to claim 5, characterized in that, The platform relative motion measurement camera group consists of a physical camera and an optical beam splitter set at its front end. The optical beam splitter is a semi-reflective semi-transparent film, a beam splitter prism, a reflector, or a multifaceted mirror, used to simultaneously image the optical paths of stationary control points from different observation directions onto the same imaging sensor of the physical camera. The platform relative motion measurement camera group at the initial moment and the current moment Observe stationary control points or immobile feature points in space respectively; since the coordinates of these control points in the world coordinate system W do not change, the changes in their image coordinates are mainly caused by the translation and rotation of the UAV body coordinate system B; Multiple virtual perspectives formed by a group of relative motion measurement cameras Treating the UAV as multiple equivalent cameras, the initial pose of the UAV body is first calculated from the initial observations at the initial moment, and then the coordinate system of the UAV body is calculated from... arrive Six degrees of freedom relative motion; Let the first A stationary control point in the world coordinate system The homogeneous coordinates in the middle are In its first A virtual perspective At the moment The homogeneous coordinates of the image are Then the camera projection relationship can be written as: ； In the formula, As a scale factor, For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment The homogeneous coordinates of the image, For the first A stationary control point in the world coordinate system homogeneous coordinates in World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Represents the image coordinate system; Combining the transformation relationship between the virtual camera coordinate system and the UAV body coordinate system, we can obtain: ； In the formula, World coordinate system up to the current moment A virtual viewpoint coordinate system The extrinsic matrix, Coordinate systems of each virtual camera in the platform relative motion measurement camera group With the UAV body coordinate system The fixed extrinsic parameters between them Let be the six-degree-of-freedom relative motion matrix of the UAV platform at the current moment relative to the initial moment. It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix; Get the current time of the first The projection relationship of stationary control points under a virtual viewpoint is calculated by minimizing the reprojection error. : ； In the formula, This refers to the six-degree-of-freedom relative motion matrix of the unmanned aerial vehicle platform relative to the initial moment. For the first The equivalent intrinsic parameter matrix of each virtual viewpoint. For the first The stationary control point at the ... A virtual perspective At the moment Image coordinates, For the first A stationary control point in the world coordinate system homogeneous coordinates in Virtual camera coordinate system With the UAV body coordinate system The fixed extrinsic parameters between them It is the coordinate system of the UAV body from the world coordinate system W to the initial time. The transformation matrix.

7. The large-range deformation displacement measuring device for virtual camera array compensation according to claim 5, characterized in that, The global field-of-view camera module is composed of multiple wide-angle cameras. After calibration, the multiple cameras are unified to the global field-of-view coordinate system or the UAV body coordinate system to form a larger field-of-view coverage area. The system determines the target point's position based on its pixel location in the global field of view image and the global field of view camera. The calibration parameters are used to calculate the target point relative to the UAV body coordinate system. The spatial line of sight, combined with the gimbal coordinate system Coordinate system with deformation displacement measurement camera Based on the installation relationship, the desired rotation angle of the gimbal is generated, so that the deformation displacement measurement camera points to the target area; Let the homogeneous coordinates of the target point in the global field-of-view camera image be... The intrinsic parameter matrix of the global field-of-view camera is The target point is in the global field-of-view camera coordinate system. The downward line of sight can be written as: ； In the formula, For the target point at The downward line of sight vector, Let be the homogeneous coordinates of the target point in the global field-of-view camera image. This is the intrinsic parameter matrix of the global field-of-view camera; By fixed external parameters Extracting the rotation matrix This line of sight can be converted to the drone's body coordinate system. : ； In the formula, For the target point at The downward line of sight vector, For the target point at The downward line of sight vector, It is a fixed external parameter Extracted rotation matrix; according to It can calculate the desired azimuth and pitch angles of the gimbal; after the gimbal completes coarse pointing, the deformation displacement measurement camera acquires local high-resolution images and achieves precise locking through target feature localization or template matching.

8. The large-range deformation displacement measuring device for virtual camera array compensation according to claim 5, characterized in that, The marker block is equipped with cooperative marker points, coded points, dots, checkerboard patterns, AprilTag, ArUco markers, or actively emitting markers; the actively emitting markers are infrared markers or coded LED arrays, used to improve recognition stability under low light or complex background conditions. The steps for the gimbal to obtain the true attitude of the gimbal coordinate system Y relative to the UAV body coordinate system B through visual pose calculation include: set up UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters For the gimbal coordinate system To the coordinate system of the gimbal marker block The fixed extrinsic parameters, the gimbal's current pose relative to the drone body: ； In the formula, UAV body coordinate system gimbal coordinate system at the current moment The extrinsic matrix, For the gimbal coordinate system To the coordinate system of the gimbal marker block Fixed extrinsic parameters UAV body coordinate system To measure the camera coordinate system at the gimbal angle Fixed extrinsic parameters This is the extrinsic parameter of the gimbal marker block relative to the gimbal rotation angle measuring camera at the current moment, which can be solved using the existing PnP method: Deformation displacement measurement camera coordinate system With the gimbal coordinate system The fixed extrinsic parameters between them are The current pose of the deformation displacement measurement camera relative to the drone body is: ； In the formula, UAV body coordinate system Deformation displacement measurement camera coordinate system at the current moment The extrinsic matrix, Coordinate system for deformation displacement measurement camera With the gimbal coordinate system The fixed extrinsic parameters between them UAV body coordinate system gimbal coordinate system at the current moment The extrinsic parameter matrix; Visual measurement results It can directly replace the angle of the gimbal encoder, or be used to correct the actual pointing error caused by the gimbal encoder, control hysteresis, installation error and vibration coupling of the gimbal structure below; The device also includes a synchronization triggering module, which is used to synchronously trigger the acquisition of the platform relative motion measurement camera group, global field of view camera module, gimbal rotation measurement camera and deformation displacement measurement camera to eliminate time synchronization errors between multiple cameras.

Images