Calibration and measurement method of airborne binocular stereo perception system

Abstract

The application discloses a kind of calibration and measurement method of airborne binocular stereo perception system, it is related to computer vision technical field, the present application includes the following steps;Initialize multi-sensor space-time synchronization, construct synchronization model, measure the minimum value of difference;Utilize IMU data to construct motion compensation model, collect dynamic characteristics and do motion compensation;From feature point speed variance and local texture gradient two dimensions, construct evaluation function, quantitatively assess feature stability;Establish weighted projection error model, wherein weight comprehensively considers feature stability and multi-sensor consistency;Adopt sliding window optimization strategy, update parameters according to adaptive learning rate, jacobian matrix and weight matrix;Establish depth measurement compensation model, through offline experiment, carry out three-dimensional measurement error compensation;Construct system confidence index, calculate confidence, when confidence is less than specific value, trigger active calibration mode, the present application proposes two-dimensional evaluation function, effectively suppresses the mismatch of motion blur.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a calibration and measurement method for an airborne binocular stereo perception system. Background Technology

[0002] With the increasing demand for 3D perception from mobile platforms such as drones and robots, airborne binocular stereo vision systems are widely used due to their advantages of passive measurement and high resolution. However, practical deployment faces many challenges. Traditional static calibration methods rely on fixed calibration objects, which are difficult to cope with the dynamic environment interference of airborne platforms. Parameter drift caused by vibration and temperature changes significantly affects measurement accuracy. Although existing online calibration methods introduce feature matching technology, feature points in dynamic environments are easily affected by motion blur, mismatch in weak texture scenes, and interference from moving targets, resulting in a serious decrease in stability. In multi-sensor fusion schemes, the collaboration efficiency between IMU and visual data is insufficient. Time delay issues and unmodeled coupling errors exacerbate parameter distortion, leading to a higher failure rate in high-dynamic scenes. The insufficient feature selection capability in weak texture environments causes the system to rely on frequent manual intervention, severely restricting its long-term autonomous operation capability. Therefore, we propose a calibration and measurement method for airborne binocular stereo perception systems. Summary of the Invention

[0003] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0004] This invention relates to a calibration and measurement method for an airborne binocular stereo perception system, comprising the following steps;

[0005] Step S1: Initialize multi-sensor spatiotemporal synchronization, construct a spatiotemporal synchronization model between the IMU and the binocular camera, and determine the time delay compensation amount by finding the minimum difference between the angular velocity measurements;

[0006] Step S2: Construct a motion compensation model using IMU data, collect dynamic features, and perform motion compensation;

[0007] Step S3: Construct an evaluation function from two dimensions: feature point velocity variance and local texture gradient, to quantitatively evaluate feature stability;

[0008] Step S4: Establish a weighted projection error model, where the weights comprehensively consider feature stability and multi-sensor consistency to optimize adaptive weighted calibration;

[0009] Step S5: Employ a sliding window optimization strategy to update parameters through adaptive learning rate, Jacobian matrix, and weight matrix, thereby completing dynamic parameter update and verification;

[0010] Step S6: Establish a depth measurement compensation model and perform 3D measurement error compensation through offline experiments;

[0011] Step S7: Construct a system confidence index. Calculate the confidence level based on the relationship between the amount of stable data and the total amount of data, as well as the error correlation. Trigger the active calibration mode when the confidence level is less than a specific value.

[0012] Step S3 involves evaluating the stability of the feature points using a two-dimensional evaluation function S. i The formula is as follows:

[0013]

[0014] In the formula, The variance of the velocity at the feature points. The gradient is the local texture gradient, α and β are weighting coefficients and α+β=1, and x,y are the coordinates of the feature points.

[0015] Furthermore, in step S1, high-frequency motion signals are excited by mechanical vibration, and angular velocities are collected to construct a spatiotemporal synchronization model T. sync The formula is as follows:

[0016] T sync =argmin∑||ω imu (t+Δt)-ω optical (t)|| 2 ;

[0017] In the formula, ω imu ω is the angular velocity measured by the IMU, t is time, and ω is the angular velocity. optical (t) represents the angular velocity measurement of the binocular camera at time t, Δt represents the time synchronization compensation amount, and (t+Δt) represents the time point after adding the time delay compensation amount Δt to the time t.

[0018] Furthermore, in step S2, the delay compensation amount Δt output by the spatiotemporal synchronization model in step S1 is used. This is combined with IMU data through the spatiotemporal synchronization model. In a dynamic environment, the feature points collected by the sensor are affected by device movement. If the area where the feature points are located... If the texture is below a set threshold, it is identified as a weak texture region, and the feature point is directly removed. A motion compensation model is then constructed using IMU data. The formula is as follows:

[0019]

[0020] In the formula, H imu p is the IMU pose transformation matrix. t Δt represents the uncompensated feature point location, g represents the gravitational acceleration, and Δt represents the IMU sampling time interval.

[0021] Furthermore, in step S4, the formula for the weighted projection error E model is as follows:

[0022]

[0023] In the formula, measures the difference between the projection of a 3D point after transformation and the actual observed point, n is the number of feature points involved in the calculation, i is the index variable, V is the projection function, T is the transformation matrix, and X... i Let x be the feature point in the i-th three-dimensional space. i Let X be the actual observed position of the i-th feature point X on the image plane;

[0024] In step S4, the weight w i The formula is as follows:

[0025]

[0026] In the formula, w i Let S be the weight of the i-th feature point, Si be the stability score of the i-th feature point, and S be the weight of the i-th feature point. j The average stability score of all feature points is given by γ, where γ is the motion consistency adjustment factor, and the adjustment weights reflect the sensitivity of the distance measurement consistency between the IMU and the vision sensor. imu The distance value measured by the IMU, d υis The distance value measured by the vision sensor.

[0027] Furthermore, in step S5, the weighted projection error E optimized in step S4 is used as the objective function for sliding window optimization. The sliding window optimization strategy is used to dynamically update the parameters, and the updated calibration parameters θ are then used. new The formula is as follows:

[0028] θ new =θ old +λ×(J T WJ) -1 ×J T Wr;

[0029] In the formula, θ old Let λ be the original parameters, λ be the adaptive learning rate, J be the Jacobian matrix of the residuals with respect to the parameters, W be the weight matrix, and r be the residual vector.

[0030] Furthermore, in step S6, environmental factors such as temperature and vibration can affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the compensated depth value is obtained. The formula is as follows:

[0031]

[0032] In the formula, z is the original measured depth value, k1 is the temperature coupling coefficient, k2 is the vibration coupling coefficient, ΔT is the temperature change, which is measured in real time by a temperature sensor, and Δω is the vibration intensity, which is calculated from the angular velocity variance collected by the IMU.

[0033] Furthermore, in step S7, by operating in a dynamic environment, it is necessary to construct a system confidence index. The formula for the system confidence index C is as follows:

[0034]

[0035] In the formula, N valid N represents the number of valid data points. total C is the total number, E is the weighted projection error, and σ0 is the preset error threshold. When C is less than the set threshold, the active calibration mode is triggered to recalibrate the system.

[0036] Furthermore, when step S7 triggers the active calibration mode, steps S1 to S5 are re-executed, and the depth value in step S6 is corrected using the updated calibration parameters. The system confidence level after compensation is dynamically evaluated using the system confidence level C formula until C recovers to above the set threshold.

[0037] The present invention has the following beneficial effects:

[0038] 1. In view of the limitations of traditional methods that rely on a single index, this invention proposes a two-dimensional evaluation function based on feature point velocity variance and local texture gradient to quantify the stability of dynamic feature points. On this basis, a weighted projection error model is used to adaptively fuse the data consistency between the IMU and the visual sensor, effectively suppressing mismatches caused by motion blur or changes in illumination.

[0039] 2. This invention constructs a spatiotemporal synchronization model between an IMU and a binocular camera, employs a strategy of high-frequency motion signal excitation and minimizing angular velocity differences to accurately compensate for time delay errors between sensors, and combines an IMU data-driven motion compensation model to dynamically correct feature point offsets caused by device vibration or rapid movement, significantly improving calibration robustness in dynamic scenarios.

[0040] 3. This invention constructs a system confidence index, monitors the proportion of effective data and weighted projection error in real time, and dynamically evaluates the system reliability. When the confidence level is lower than a preset threshold, it triggers an active calibration mode, uses a sliding window optimization strategy to update parameters, avoids error accumulation, and, combined with a depth measurement compensation model, can adaptively correct depth deviations caused by temperature and vibration.

[0041] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the calibration and measurement method of an airborne binocular stereo perception system according to the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Please see Figure 1 As shown, the present invention is a calibration and measurement method for an airborne binocular stereo perception system, comprising the following steps;

[0046] Step S1: Initialize multi-sensor spatiotemporal synchronization, construct a spatiotemporal synchronization model between the IMU and the binocular camera, and determine the time delay compensation amount by finding the minimum difference between the angular velocity measurements;

[0047] Step S2: Construct a motion compensation model using IMU data, collect dynamic features, and perform motion compensation;

[0048] Step S3: Construct an evaluation function from two dimensions: feature point velocity variance and local texture gradient, to quantitatively evaluate feature stability;

[0049] Step S4: Establish a weighted projection error model, where the weights comprehensively consider feature stability and multi-sensor consistency to optimize adaptive weighted calibration;

[0050] Step S5: Employ a sliding window optimization strategy to update parameters through adaptive learning rate, Jacobian matrix, and weight matrix, thereby completing dynamic parameter update and verification;

[0051] Step S6: Establish a depth measurement compensation model and perform 3D measurement error compensation through offline experiments;

[0052] Step S7: Construct a system confidence index. Calculate the confidence level based on the relationship between the amount of stable data and the total amount of data, as well as the error correlation. Trigger the active calibration mode when the confidence level is less than a specific value.

[0053] Step S3: Evaluate the stability of the constructed two-dimensional evaluation function S. i The formula is as follows:

[0054]

[0055] In the formula, The variance of the velocity at the feature points. The gradient is the local texture gradient, α and β are weighting coefficients and α+β=1, and x,y are the coordinates of the feature points.

[0056] In step S1, high-frequency motion signals are excited by mechanical vibration, and angular velocities are collected to construct a spatiotemporal synchronization model T. sync The formula is as follows:

[0057] T sync =argminΣ‖ω imu (t+Δt)-ω optical (t)|| 2 ;

[0058] In the formula, ω imu ω is the angular velocity measured by the IMU, t is time, and ω is the angular velocity. optical (t) represents the angular velocity measurement of the binocular camera at time t, Δt represents the time synchronization compensation amount, and (t+Δt) represents the time point after adding the time delay compensation amount Δt to the time t.

[0059] In step S2, the time delay compensation amount Δt output by the spatiotemporal synchronization model in step S1 is used. This is combined with IMU data through the spatiotemporal synchronization model. In a dynamic environment, the feature points collected by the sensor are affected by device movement. If the area where the feature points are located... If the texture is below a set threshold, it is identified as a weak texture region, and the feature point is directly removed. A motion compensation model is then constructed using IMU data. The formula is as follows:

[0060]

[0061] In the formula, H imu p is the IMU pose transformation matrix. t Δt represents the uncompensated feature point location, g represents the gravitational acceleration, and Δt represents the IMU sampling time interval.

[0062] In step S4, the formula for the weighted projection error E model is as follows:

[0063]

[0064] In the formula, measures the difference between the projection of a 3D point after transformation and the actual observed point, n is the number of feature points involved in the calculation, i is the index variable, V is the projection function, T is the transformation matrix, and X... iLet x be the i-th feature point in the three-dimensional space. i Let X be the actual observed position of the i-th feature point X on the image plane;

[0065] In step S4, the weight w i The formula is as follows:

[0066]

[0067] In the formula, w i Let S be the weight of the i-th feature point, Si be the stability score of the i-th feature point, and S be the weight of the i-th feature point. j The average stability score of all feature points is given by γ, where γ is the motion consistency adjustment factor, and the adjustment weights reflect the sensitivity of the distance measurement consistency between the IMU and the vision sensor. imu The distance value measured by the IMU, d υis The distance value measured by the vision sensor.

[0068] In step S5, the weighted projection error E optimized in step S4 is used as the objective function for sliding window optimization. The sliding window optimization strategy is used to dynamically update the parameters, and the updated calibration parameters θ are then used. new The formula is as follows:

[0069] θ new =θ old +λ×(J T WJ) -1 ×J T Wr;

[0070] In the formula, θ old Let λ be the original parameters, λ be the adaptive learning rate, J be the Jacobian matrix of the residuals with respect to the parameters, W be the weight matrix, and r be the residual vector.

[0071] In step S6, environmental factors such as temperature and vibration can affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the compensated depth value is obtained. The formula is as follows:

[0072]

[0073] In the formula, z is the original measured depth value, k1 is the temperature coupling coefficient, k2 is the vibration coupling coefficient, ΔT is the temperature change, which is measured in real time by a temperature sensor, and Δω is the vibration intensity, which is calculated from the angular velocity variance collected by the IMU.

[0074] In step S7, by running in a dynamic environment, it is necessary to construct a system confidence index. The formula for the system confidence index C is as follows:

[0075]

[0076] In the formula, N valid N represents the number of valid data points. total C is the total number, E is the weighted projection error, and σ0 is the preset error threshold. When C is less than the set threshold, the active calibration mode is triggered to recalibrate the system.

[0077] When step S7 triggers the active calibration mode, steps S1 to S5 are re-executed, and the depth value in step S6 is corrected using the updated calibration parameters. The system confidence level after compensation is dynamically evaluated using the system confidence level C formula until C recovers to above the set threshold.

[0078] One specific application of this embodiment is:

[0079] 1. Implementation conditions

[0080] hardware platform

[0081] Binocular camera: FLIR Blackfly S BFS-U3-123S6C-C, baseline distance 30cm, frame rate 30Hz;

[0082] IMU: BMI088 six-axis inertial unit, sampling rate 200Hz;

[0083] Processor: NVIDIA Jetson AGX Xavier;

[0084] Vehicle: DJI Matrice 600 Pro drone;

[0085] Environmental parameters

[0086] Vibration conditions: 0.2g-1.2g (vibration simulated by an eccentric motor);

[0087] Temperature change: 15℃→45℃ (simulated using a temperature-controlled chamber);

[0088] Dynamic scenario: moving vehicles account for 40%, pedestrians account for 20%;

[0089] 2. Specific Implementation Steps

[0090] Step 1: Sensor Spatiotemporal Synchronization

[0091] Data collection:

[0092] Simultaneously trigger the binocular camera and IMU to collect 10 seconds of angular velocity data (drone hovering state);

[0093] Delay calibration:

[0094] The cross-correlation algorithm was used to solve for Δt = 8.2 ms, and the angular velocity residual was reduced to 0.15° / s after compensation (original residual 1.8° / s).

[0095] Step 2: Dynamic Feature Processing

[0096] Feature extraction:

[0097] 1000 feature points were extracted using an improved ORB algorithm (FAST corner point + grayscale centroid method);

[0098] Exercise compensation:

[0099] Image displacement was compensated using IMU data, and the standard deviation of the feature point trajectory decreased from 12.3 pixels to 3.1 pixels after compensation.

[0100] Step 3: Characteristic Stability Assessment

[0101] Scoring Calculation:

[0102] Parameter settings: α = 0.6, β = 0.4 (empirical values);

[0103] Filtering threshold: S i >0.75 (retaining the first 30% of feature points);

[0104] Results: 92% elimination rate for dynamic target (vehicle) feature points, and 85% retention rate for static background points;

[0105] Step 4: Adaptive Calibration Optimization

[0106] Weight calculation:

[0107] Set γ = 0.8 (corresponding to a vibration intensity of 0.5g);

[0108] Typical weight distribution: w i ∈[0.2,1.8] (low stability feature weights decay by 80%);

[0109] Nonlinear optimization:

[0110] After five iterations of the LM algorithm, the reprojection error was reduced from the initial 3.8 pixels to 0.9 pixels.

[0111] Step 5: Dynamic parameter update

[0112] Sliding window strategy:

[0113] Window size: 20 frames;

[0114] Calibration parameter update frequency: 1Hz (when confidence level C > 0.8) → 5Hz (when C < 0.7);

[0115] Step 6: Three-dimensional measurement compensation

[0116] Error correction:

[0117] Calibration coefficients: k1 = 3.2 × 10⁻⁴ / ℃, k2 = 1.8 × 10⁻³ / (rad / s);

[0118] Compensation effect: at a temperature of 45℃, the depth error decreased from 4.7% to 0.9%; under 1.0g vibration, the error decreased from 2.1% to 0.6%.

[0119] Experimental verification

[0120] Test Scenario 1: Static Calibration Board Verification

[0121] Traditional offline calibration 0.5 0.3％ Online calibration of this invention 0.7 0.5％

[0122] Test Scenario 2: Dynamic Environment (60% Moving Targets)

[0123]

[0124]

[0125] Test Scenario 3: Continuous Vibration (0.5g, 4 hours)

[0126] 0 hours 300.00mm 1200.0 pixels 4 hours 299.82mm (-0.06) 1198.7 pixels (-0.11%)

[0127] 4. Key Implementation Details

[0128] Enhanced motor compensation:

[0129] For high-frequency vibration, in Add a second-order term to the formula:

[0130]

[0131] Where j is the jerk measured by the IMU (Jerk);

[0132] Outlier removal mechanism:

[0133] For satisfying |d imu -d υis Feature points with a value greater than 3σ are directly discarded (σ is the standard deviation of the displacement difference within the window);

[0134] 5. Conclusions of the comparative experiment

[0135] 1. Environmental adaptability: In scenarios where dynamic targets account for 60%, the present invention can still maintain an effective calibration rate of 87% (compared to only 17% for traditional methods);

[0136] 2. Real-time performance: Single frame processing time is 18.3ms (meets the 30Hz real-time requirement);

[0137] 3. Robustness: When the vibration intensity reaches 1.2g, the depth error is still <1.2% (compared to >8.7% for traditional methods).

[0138] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0139] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A calibration and measurement method for an airborne binocular stereo perception system, characterized in that: Includes the following steps; Step S1: Initialize multi-sensor spatiotemporal synchronization, construct a spatiotemporal synchronization model between the IMU and the binocular camera, and determine the time delay compensation amount by finding the minimum difference between the angular velocity measurements; Step S2: Construct a motion compensation model using IMU data, collect dynamic features, and perform motion compensation; Step S3: Construct an evaluation function from two dimensions: feature point velocity variance and local texture gradient, to quantitatively evaluate feature stability; Step S4: Establish a weighted projection error model, where the weights comprehensively consider feature stability and multi-sensor consistency to optimize adaptive weighted calibration; Step S5: Using a sliding window optimization strategy, the calibration parameters are updated through adaptive learning rate, Jacobian matrix and weight matrix to complete the dynamic update and verification of calibration parameters; Step S6: Establish a depth measurement compensation model and perform 3D measurement error compensation through offline experiments; Step S7: Construct a system confidence index. Calculate the confidence level based on the relationship between the amount of stable data and the total amount of data, as well as the error correlation. Trigger the active calibration mode when the confidence level is less than a specific value. Step S3 involves evaluating the two-dimensional evaluation function constructed to assess the stability of feature points. The formula is as follows: ； In the formula, The variance of the velocity at the feature points. For local texture gradient, , The weighting coefficients and , The coordinates of the feature point; In step S1, high-frequency motion signals are excited by mechanical vibration, and angular velocities are collected to construct a spatiotemporal synchronization model. The formula is as follows: ； In the formula, Angular velocity measured by the IMU For time, For binocular cameras at any time The angular velocity measurement value, This is the amount of time delay compensation. For at any time Add delay compensation amount to the basis The subsequent point in time; In step S4, the weighted projection error is... The model formula is as follows: ； In the formula, The number of feature points involved in the calculation. For index variables, For projection function, Let be the transformation matrix. For the first Feature points in three-dimensional space For the first Feature points The actual observation location on the image plane; In step S4, the weight The formula is as follows: ； In the formula, For the first The weights of each feature point For the first Stability score of each feature point The average of the stability scores for all feature points. This is a motion consistency adjustment factor, where the adjustment weights reflect the sensitivity of the distance measurement consistency between the IMU and the vision sensor. The distance value measured by the IMU. The distance value measured by the vision sensor.

2. The calibration and measurement method for an airborne binocular stereo perception system according to claim 1, characterized in that, In step S2, the delay compensation amount output by the spatiotemporal synchronization model in step S1 is used. By combining a spatiotemporal synchronization model with IMU data, in dynamic environments, the feature points collected by the sensors are affected by the movement of the equipment. If the area where the feature points are located... If the texture is below a set threshold, it is identified as a weak texture region, and the feature point is directly removed. A motion compensation model is then constructed using IMU data. The formula is as follows: ； In the formula, This is the IMU pose transformation matrix. For the location of uncompensated feature points, It is the acceleration due to gravity. The time interval for IMU sampling.

3. The calibration and measurement method for an airborne binocular stereo perception system according to claim 1, characterized in that, In step S5, the weighted projection error optimized in step S4 is... Using the sliding window optimization as the objective function, the parameters are dynamically updated using the sliding window optimization strategy. The updated calibration parameters are... The formula is as follows: ； In the formula, For the original parameters, For adaptive learning rate, Let Jacobian matrix be the residual with respect to parameters. This is the weight matrix. This is the residual vector.

4. The calibration and measurement method for an airborne binocular stereo perception system according to claim 1, characterized in that, In step S6, environmental factors such as temperature and vibration can affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the compensated depth value is obtained. The formula is as follows: ； In the formula, This is the original measured depth value. The temperature coupling coefficient is... The vibration coupling coefficient is... The change in temperature is measured in real time by a temperature sensor. The vibration intensity is calculated from the variance of the angular velocity acquired by the IMU.

5. The calibration and measurement method for an airborne binocular stereo perception system according to claim 1, characterized in that, In step S7, a system confidence index is constructed by running in a dynamic environment. The formula is as follows: ； In the formula, For the number of valid data, For the total quantity, To add weighted projection error, For the preset error threshold, when When the value is less than the set threshold, the active calibration mode is triggered to recalibrate the system.

6. The calibration and measurement method for an airborne binocular stereo perception system according to claim 5, characterized in that, In step S6, environmental factors such as temperature and vibration can affect the measurement results. By compensating for these errors, a depth measurement compensation model is established, and the compensated depth value is obtained. The formula is as follows: ； In the formula, The original measured depth value. The temperature coupling coefficient is... The vibration coupling coefficient is... The change in temperature is measured in real time by a temperature sensor. The vibration intensity is calculated from the variance of the angular velocity acquired by the IMU. Accordingly, the method further includes: When step S7 triggers the active calibration mode, steps S1 to S5 are re-executed, and the depth value in step S6 is corrected using the updated calibration parameters. Through system confidence The formula dynamically evaluates the confidence level of the compensated system until... Restore to above the set threshold.

Images