Calibration and measurement method of airborne binocular stereoscopic perception system

Abstract

The invention discloses a calibration and measurement method of an airborne binocular stereoscopic perception system, and relates to the technical field of computer vision. Initializing multi-sensor space-time synchronization, constructing a synchronization model, and measuring a minimum value of a value difference; constructing a motion compensation model by using the I MU data, collecting dynamic characteristics and performing motion compensation; an evaluation function is constructed from the two dimensions of the feature point speed variance and the local texture gradient, and the feature stability is quantitatively evaluated; establishing a weighted projection error model, wherein the weight comprehensively considers the feature stability and the multi-sensor consistency; a sliding window optimization strategy is adopted, and parameters are updated according to the adaptive learning rate, the Jacobian matrix and the weight matrix; establishing a depth measurement compensation model, and performing three-dimensional measurement error compensation through an offline experiment; and constructing a system confidence index, calculating the confidence, and triggering an active calibration mode when the confidence is less than a specific value, the invention provides a two-dimensional evaluation function, and effectively inhibits the mismatching of motion blur.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a calibration and measurement method for an airborne binocular stereo perception system. Background Art

[0002] With the growing demand for three-dimensional perception on mobile platforms such as drones and robots, airborne binocular stereo vision systems have been widely used due to their advantages of passive measurement and high resolution. However, their actual deployment faces many difficulties. Traditional static calibration methods rely on fixed calibration objects and are difficult to cope with the dynamic environment interference of airborne platforms. Parameter drift caused by vibration and temperature changes significantly affects measurement accuracy. Although existing online calibration methods introduce feature matching technology, feature points in dynamic environments are susceptible to motion blur, mismatching of weak-texture scenes, and interference from moving targets, resulting in a serious decrease in stability. In multi-sensor fusion schemes, the coordination efficiency of IMU and visual data is insufficient. Time delay problems and unmodeled coupling errors aggravate parameter distortion, increasing the failure rate in highly dynamic scenes. The ability to screen features in weak-texture environments is insufficient, causing the system to rely on frequent manual intervention, seriously restricting its long-term autonomous operation capabilities. Therefore, we propose a calibration and measurement method for airborne binocular stereo perception systems. Summary of the Invention

[0003] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0004] The present invention is a calibration and measurement method for an airborne binocular stereo perception system, comprising the following steps:

[0005] Step S1: Initialize the spatiotemporal synchronization of multiple sensors, build a spatiotemporal synchronization model between the IMU and the binocular camera, and determine the delay compensation amount by finding the minimum value of the angular velocity measurement difference;

[0006] Step S2: Use IMU data to build a motion compensation model, collect dynamic features and perform motion compensation;

[0007] Step S3: construct an evaluation function based on the two dimensions of feature point velocity variance and local texture gradient to quantitatively evaluate feature stability;

[0008] Step S4: establishing a weighted projection error model, where the weights comprehensively consider feature stability and multi-sensor consistency to optimize the adaptive weighted calibration;

[0009] Step S5: Using a sliding window optimization strategy, the parameters are updated through adaptive learning rate, Jacobian matrix and weight matrix to complete the dynamic update and verification of parameters;

[0010] Step S6: Establish a depth measurement compensation model and perform three-dimensional measurement error compensation through offline experiments;

[0011] Step S7: Construct a system confidence index, calculate the confidence based on the relationship between the amount of stable data and the total amount of data and the error correlation, and trigger the active calibration mode when the confidence is less than a specific value;

[0012] In step S3, the dual-dimensional evaluation function S constructed by evaluating the stability of the feature points is constructed. i The formula is as follows:

[0013]

[0014] Where, is the velocity variance of the feature point, is the local texture gradient, α and β are weight coefficients and α+β=1, x and y are the coordinate positions of the feature points.

[0015] Furthermore, in step S1, high-frequency motion signals are excited by mechanical vibration, and angular velocity is collected to construct a spatiotemporal synchronization model T sync The formula is as follows:

[0016] T sync =argmin∑||ω imu (t+Δt)-ω optical (t)|| 2 ;

[0017] Where, ω imu is the angular velocity measured by IMU, t is time, ω optical (t) is the angular velocity measurement value of the binocular camera at time t, Δt is the time synchronization compensation, and (t+Δt) is the time point after adding the time delay compensation Δt to time t.

[0018] Furthermore, in step S2, the time delay compensation Δt output by the spatiotemporal synchronization model of step S1 is used to combine the spatiotemporal synchronization model with the IMU data. In a dynamic environment, the feature points collected by the sensor are affected by the movement of the device. If the area where the feature points are located is If the value is lower than the set threshold, it is determined to be a weak texture area, and the feature point is directly removed. The motion compensation model is constructed using IMU data. The formula is as follows:

[0019]

[0020] Where H imu is the IMU pose transformation matrix, p t is the uncompensated feature point position, g is the gravitational acceleration, and Δt is the time interval of IMU sampling.

[0021] Furthermore, in step S4, the weighted projection error E model formula is as follows:

[0022]

[0023] In the formula, the difference between the transformed projection of the three-dimensional point and the actual observation point is measured, n is the number of feature points involved in the calculation, i is the index variable, V is the projection function, T is the transformation matrix, X i is the feature point in the i-th three-dimensional space, x i is the actual observed position of the i-th feature point X on the image plane;

[0024] In step S4, the weight w i The formula is as follows:

[0025]

[0026] Where w i is the weight of the i-th feature point, Si is the stability score of the i-th feature point, S j is the average stability score of all feature points, γ is the motion consistency adjustment factor, which adjusts the sensitivity of the weight to the consistency of the distance measured by IMU and visual sensor, d imu is the distance value measured by IMU, d υis The distance value measured by the vision sensor.

[0027] Furthermore, in step S5, the weighted projection error E optimized in step S4 is used as the objective function of the sliding window optimization, and the sliding window optimization strategy is used to dynamically update the parameters. The updated calibration parameter θ new The formula is as follows:

[0028] θ new =θ old +λ×(J T WJ) -1 ×J T Wr;

[0029] Where θ old is the original parameter, λ is the adaptive learning rate, J is the Jacobian matrix of the residual to parameter, W is the weight matrix, and r is the residual vector.

[0030] Furthermore, in step S6, environmental factors such as temperature and vibration may affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the compensated depth value The formula is as follows:

[0031]

[0032] Where z is the original measured depth value, k1 is the temperature coupling coefficient, k2 is the vibration coupling coefficient, ΔT is the temperature change, which is measured in real time by the temperature sensor, and Δω is the vibration intensity, which is calculated from the angular velocity variance collected by the IMU.

[0033] Furthermore, in step S7, by running in a dynamic environment, it is necessary to construct a system confidence index. The system confidence C formula is as follows:

[0034]

[0035] Where N valid is the number of valid data, N total is the total number, E is the weighted projection error, σ0 is the preset error threshold. When C is less than the set threshold, the active calibration mode is triggered to recalibrate the system.

[0036] Furthermore, when the active calibration mode is triggered in step S7, steps S1 to S5 are re-executed, and the depth value in step S6 is corrected using the updated calibration parameters. The system confidence C formula is used to dynamically evaluate the compensated system confidence until C recovers to above the set threshold.

[0037] The present invention has the following beneficial effects:

[0038] 1. To address the limitations of traditional methods that rely on a single indicator, this paper proposes a two-dimensional evaluation function based on feature point velocity variance and local texture gradient to quantify the stability of dynamic feature points. On this basis, a weighted projection error model is used to adaptively fuse the data consistency of the IMU and visual sensor, effectively suppressing mismatches caused by motion blur or illumination changes.

[0039] 2. This invention constructs a spatiotemporal synchronization model between the IMU and the binocular camera, adopts a high-frequency motion signal excitation and angular velocity difference minimization strategy to accurately compensate for the time delay error between sensors. Combined with the IMU data-driven motion compensation model, it dynamically corrects the feature point offset caused by device vibration or rapid movement, significantly improving the calibration robustness in dynamic scenarios.

[0040] 3. The present invention constructs a system confidence index, monitors the proportion of valid data and weighted projection error in real time, and dynamically evaluates system reliability. When the confidence level is lower than a preset threshold, the active calibration mode is triggered, and the sliding window optimization strategy is used to update the parameters to avoid error accumulation. Combined with the depth measurement compensation model, it can adaptively correct depth deviations caused by temperature and vibration.

[0041] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 The figure is a flow chart of a calibration and measurement method of an airborne binocular stereo perception system according to the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] See also Figure 1 As shown, the present invention is a calibration and measurement method for an airborne binocular stereo perception system, comprising the following steps:

[0046] Step S1: Initialize the spatiotemporal synchronization of multiple sensors, build a spatiotemporal synchronization model between the IMU and the binocular camera, and determine the delay compensation amount by finding the minimum value of the angular velocity measurement difference;

[0047] Step S2: Use IMU data to build a motion compensation model, collect dynamic features and perform motion compensation;

[0048] Step S3: construct an evaluation function based on the two dimensions of feature point velocity variance and local texture gradient to quantitatively evaluate feature stability;

[0049] Step S4: establishing a weighted projection error model, where the weights comprehensively consider feature stability and multi-sensor consistency to optimize the adaptive weighted calibration;

[0050] Step S5: Using a sliding window optimization strategy, the parameters are updated through adaptive learning rate, Jacobian matrix and weight matrix to complete the dynamic update and verification of parameters;

[0051] Step S6: Establish a depth measurement compensation model and perform three-dimensional measurement error compensation through offline experiments;

[0052] Step S7: Construct a system confidence index, calculate the confidence based on the relationship between the amount of stable data and the total amount of data and the error correlation, and trigger the active calibration mode when the confidence is less than a specific value;

[0053] Step S3: Evaluate the stability of feature points using the dual-dimensional evaluation function S i The formula is as follows:

[0054]

[0055] Where, is the velocity variance of the feature point, is the local texture gradient, α and β are weight coefficients and α+β=1, x and y are the coordinate positions of the feature points.

[0056] In step S1, high-frequency motion signals are excited by mechanical vibration, and angular velocity is collected to construct a spatiotemporal synchronization model T sync The formula is as follows:

[0057] T sync =argminΣ‖ω imu (t+Δt)-ω optical (t)|| 2 ;

[0058] Where, ω imu is the angular velocity measured by IMU, t is time, ω optical (t) is the angular velocity measurement value of the binocular camera at time t, Δt is the time synchronization compensation, and (t+Δt) is the time point after adding the time delay compensation Δt to time t.

[0059] In step S2, the time delay compensation Δt output by the spatiotemporal synchronization model of step S1 is combined with the IMU data through the spatiotemporal synchronization model. In a dynamic environment, the feature points collected by the sensor are affected by the movement of the device. If the area where the feature points are located is If the value is lower than the set threshold, it is determined to be a weak texture area, and the feature point is directly removed. The motion compensation model is constructed using IMU data. The formula is as follows:

[0060]

[0061] Where H imu is the IMU pose transformation matrix, p t is the uncompensated feature point position, g is the gravitational acceleration, and Δt is the time interval of IMU sampling.

[0062] In step S4, the weighted projection error E model formula is as follows:

[0063]

[0064] In the formula, the difference between the transformed projection of the three-dimensional point and the actual observation point is measured, n is the number of feature points involved in the calculation, i is the index variable, V is the projection function, T is the transformation matrix, X iis the feature point in the i-th three-dimensional space, x i is the actual observed position of the i-th feature point X on the image plane;

[0065] In step S4, the weight w i The formula is as follows:

[0066]

[0067] Where w i is the weight of the i-th feature point, Si is the stability score of the i-th feature point, S j is the average stability score of all feature points, γ is the motion consistency adjustment factor, which adjusts the sensitivity of the weight to the consistency of the distance measured by IMU and visual sensor, d imu is the distance value measured by IMU, d υis The distance value measured by the vision sensor.

[0068] In step S5, the weighted projection error E optimized in step S4 is used as the objective function of the sliding window optimization. The sliding window optimization strategy is used to dynamically update the parameters. The updated calibration parameter θ new The formula is as follows:

[0069] θ new =θ old +λ×(J T WJ) -1 ×J T Wr;

[0070] Where θ old is the original parameter, λ is the adaptive learning rate, J is the Jacobian matrix of the residual to parameter, W is the weight matrix, and r is the residual vector.

[0071] In step S6, environmental factors such as temperature and vibration will affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the compensated depth value The formula is as follows:

[0072]

[0073] Where z is the original measured depth value, k1 is the temperature coupling coefficient, k2 is the vibration coupling coefficient, ΔT is the temperature change, which is measured in real time by the temperature sensor, and Δω is the vibration intensity, which is calculated from the angular velocity variance collected by the IMU.

[0074] In step S7, by running in a dynamic environment, it is necessary to build a system confidence index. The system confidence C formula is as follows:

[0075]

[0076] Where N valid is the number of valid data, N total is the total number, E is the weighted projection error, σ0 is the preset error threshold. When C is less than the set threshold, the active calibration mode is triggered to recalibrate the system.

[0077] When the active calibration mode is triggered in step S7, steps S1 to S5 are re-executed, and the depth value in step S6 is corrected using the updated calibration parameters. The system confidence C formula is used to dynamically evaluate the compensated system confidence until C recovers to above the set threshold.

[0078] A specific application of this embodiment is:

[0079] 1. Implementation conditions

[0080] Hardware Platform

[0081] Binocular camera: FLIR Blackfly S BFS-U3-123S6C-C, baseline distance 30 cm, frame rate 30 Hz;

[0082] IMU: BMI088 six-axis inertial unit, sampling rate 200Hz;

[0083] Processor: NVIDIA Jetson AGX Xavier;

[0084] Vehicle: DJI Matrice 600Pro drone;

[0085] Environmental parameters

[0086] Vibration conditions: 0.2g-1.2g (vibration simulated by an eccentric motor);

[0087] Temperature change: 15°C → 45°C (simulated by a temperature control chamber);

[0088] Dynamic scenes: moving vehicles account for 40% and pedestrians account for 20%;

[0089] 2. Specific implementation steps

[0090] Step 1: Sensor spatiotemporal synchronization

[0091] Data collection:

[0092] Synchronously trigger the binocular camera and IMU to collect 10 seconds of angular velocity data (drone hovering state);

[0093] Delay calibration:

[0094] The cross-correlation algorithm is used to solve Δt = 8.2ms. After compensation, the angular velocity residual is reduced to 0.15° / s (the original residual is 1.8° / s);

[0095] Step 2: Dynamic Feature Processing

[0096] Feature extraction:

[0097] Use the improved ORB algorithm to extract 1000 feature points (FAST corner point + grayscale centroid method);

[0098] Motion Compensation:

[0099] Image displacement is compensated using IMU data. After compensation, the standard deviation of feature point trajectories is reduced from 12.3 pixels to 3.1 pixels.

[0100] Step 3: Feature stability assessment

[0101] Rating calculation:

[0102] Parameter settings: α = 0.6, β = 0.4 (empirical values);

[0103] Screening threshold: S i >0.75 (retain the first 30% feature points);

[0104] Results: 92% elimination rate of dynamic target (vehicle) feature points, 85% retention rate of static background points;

[0105] Step 4: Adaptive Calibration Optimization

[0106] Weight calculation:

[0107] Set γ = 0.8 (corresponding to vibration intensity 0.5g);

[0108] Typical weight distribution: w i ∈[0.2,1.8] (weight decay of low stability features is 80%);

[0109] Nonlinear Optimization:

[0110] The LM algorithm was used for five iterations, and the reprojection error was reduced from the initial 3.8 pixels to 0.9 pixels;

[0111] Step 5: Dynamic parameter update

[0112] Sliding window strategy:

[0113] Window size: 20 frames;

[0114] Calibration parameter update frequency: 1Hz (confidence level C>0.8) → 5Hz (confidence level C<0.7);

[0115] Step 6: 3D measurement compensation

[0116] Error correction:

[0117] Calibration coefficient: k1 = 3.2 × 10-4 / ° C, k2 = 1.8 × 10-3 / (rad / s);

[0118] Compensation effect: The depth error is reduced from 4.7% to 0.9% at a temperature of 45°C; the error is reduced from 2.1% to 0.6% under 1.0g vibration;

[0119] Experimental verification

[0120] Test Scenario 1: Static Calibration Plate Verification

[0121] condition Reprojection error (pixels) Depth error (at 1m) Traditional offline calibration 0.5 0.3％ Online calibration of the present invention 0.7 0.5％

[0122] Test scenario 2: Dynamic environment (60% moving targets)

[0123]

[0124]

[0125] Test scenario 3: Continuous vibration (0.5g, 4 hours)

[0126] time Baseline length drift Focal length parameter changes 0 hours 300.00mm 1200.0 pixels 4 hours 299.82mm(-0.06) 1198.7 pixels (-0.11%)

[0127] 4. Key implementation details

[0128] Motion Compensated Enhancement:

[0129] For high frequency vibration, Add a second-order term to the formula:

[0130]

[0131] Where j is the jerk measured by the IMU;

[0132] Outlier removal mechanism:

[0133] To satisfy |d imu -d υis Feature points with |>3σ are directly eliminated (σ is the standard deviation of displacement difference within the window);

[0134] 5. Conclusion of comparative test

[0135] 1. Environmental adaptability: In a scenario where dynamic targets account for 60%, the proposed method can still maintain an effective calibration rate of 87% (traditional methods only have 17%);

[0136] 2. Real-time performance: Single-frame processing takes 18.3ms (meets 30Hz real-time requirements);

[0137] 3. Robustness: When the vibration intensity reaches 1.2g, the depth error is still <1.2% (traditional methods >8.7%).

[0138] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0139] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A calibration and measurement method for an airborne binocular stereo perception system, characterized by: The following steps are included: Step S1: Initialize the spatiotemporal synchronization of multiple sensors, build a spatiotemporal synchronization model between the IMU and the binocular camera, and determine the delay compensation amount by finding the minimum value of the angular velocity measurement difference; Step S2: Use IMU data to build a motion compensation model, collect dynamic features and perform motion compensation; Step S3: construct an evaluation function based on the two dimensions of feature point velocity variance and local texture gradient to quantitatively evaluate feature stability; Step S4: establishing a weighted projection error model, where the weights comprehensively consider feature stability and multi-sensor consistency to optimize the adaptive weighted calibration; Step S5: Using a sliding window optimization strategy, the parameters are updated through adaptive learning rate, Jacobian matrix and weight matrix to complete the dynamic update and verification of parameters; Step S6: Establish a depth measurement compensation model and perform three-dimensional measurement error compensation through offline experiments; Step S7: Construct a system confidence index, calculate the confidence based on the relationship between the amount of stable data and the total amount of data and the error correlation, and trigger the active calibration mode when the confidence is less than a specific value; In step S3, the dual-dimensional evaluation function S constructed by evaluating the stability of the feature points is constructed. i The formula is as follows: Where, is the velocity variance of the feature point, is the local texture gradient, α and β are weight coefficients and α+β=1, x and y are the coordinate positions of the feature points.

2. The calibration and measurement method of an airborne binocular stereo perception system according to claim 1, characterized in that: In step S1, high-frequency motion signals are excited by mechanical vibration, and angular velocity is collected to construct a spatiotemporal synchronization model Ts. ync The formula is as follows: Ts ync =argmin∑||ω imu (t+Δt)-ω optical (t)|| 2 ; Where, ω imu is the angular velocity measured by IMU, t is time, ω optical (t) is the angular velocity measurement value of the binocular camera at time t, Δt is the time synchronization compensation, and (t+Δt) is the time point after adding the time delay compensation Δt to time t.

3. The calibration and measurement method of an airborne binocular stereo perception system according to claim 1, characterized in that: In step S2, the time delay compensation Δt output by the spatiotemporal synchronization model of step S1 is used to combine the spatiotemporal synchronization model with the IMU data. In a dynamic environment, the feature points collected by the sensor are affected by the movement of the device. If the area where the feature points are located is If the value is lower than the set threshold, it is determined to be a weak texture area, and the feature point is directly removed. The motion compensation model is constructed using IMU data. The formula is as follows: Where H imu is the IMU pose transformation matrix, p t is the uncompensated feature point position, g is the gravitational acceleration, and Δt is the time interval of IMU sampling.

4. The calibration and measurement method of an airborne binocular stereo perception system according to claim 1, characterized in that: In step S4, the weighted projection error E model formula is as follows: In the formula, the difference between the transformed projection of the three-dimensional point and the actual observation point is measured, n is the number of feature points involved in the calculation, i is the index variable, V is the projection function, T is the transformation matrix, X i is the feature point in the i-th three-dimensional space, x i is the actual observed position of the i-th feature point X on the image plane; In step S4, the weight w i The formula is as follows: In the formula, wi is the weight of the i-th feature point, Si is the stability score of the i-th feature point, S j is the average stability score of all feature points, γ is the motion consistency adjustment factor, which adjusts the sensitivity of the weight to the consistency of the distance measured by IMU and visual sensor, d imu is the distance value measured by IMU, d υis The distance value measured by the vision sensor.

5. The calibration and measurement method of an airborne binocular stereo perception system according to claim 1, characterized in that: In step S5, the weighted projection error E optimized in step S4 is used as the objective function of the sliding window optimization, and the parameters are dynamically updated using the sliding window optimization strategy. The updated calibration parameter θ new The formula is as follows: θ new =θ old +λ×(J T WJ) -1 ×J T Wr; Where θ old is the original parameter, λ is the adaptive learning rate, J is the Jacobian matrix of the residual to parameter, W is the weight matrix, and r is the residual vector.

6. The calibration and measurement method of an airborne binocular stereo perception system according to claim 1, characterized in that: In step S6, environmental factors such as temperature and vibration may affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the depth value after compensation is The formula is as follows: Where z is the original measured depth value, k1 is the temperature coupling coefficient, k2 is the vibration coupling coefficient, ΔT is the temperature change, which is measured in real time by the temperature sensor, and Δω is the vibration intensity, which is calculated from the angular velocity variance collected by the IMU.

7. The calibration and measurement method of an airborne binocular stereo perception system according to claim 1, characterized in that: In step S7, by running in a dynamic environment, it is necessary to construct a system confidence index. The system confidence C formula is as follows: Where N valid is the number of valid data, N total is the total number, E is the weighted projection error, σ0 is the preset error threshold. When C is less than the set threshold, the active calibration mode is triggered to recalibrate the system.

8. The calibration and measurement method of an airborne binocular stereo perception system according to claim S7, characterized in that: When the active calibration mode is triggered in step S7, steps S1 to S5 are re-executed, and the depth value in step S6 is corrected using the updated calibration parameters. The system confidence C formula is used to dynamically evaluate the compensated system confidence until C recovers to above the set threshold.

Images