Visual displacement measuring method

Abstract

The invention discloses a visual displacement measuring method, which comprises the following steps of: utilizing a measuring camera which is only responsible for coordinate positioning of a target point; a correction camera and an IMU are utilized, and pose recognition of the correction camera is achieved through a vision-IMU fusion method; according to the pre-calibrated internal parameters of the two cameras and the external parameters between the two cameras, the pose change of the measurement camera is calculated and used for correcting the space coordinates of the target point, and target positioning deviation caused by vibration of the measurement camera is eliminated; and finally, calculating the displacement of the target according to the coordinate of the target point in the time domain by taking the initial moment as a reference. According to the method, the displacement resolution of the measured target is not sacrificed while the pose recognition of the measurement camera is ensured; and meanwhile, the robustness of visual pose recognition in complex environments such as rapid movement, complex illumination and shielding can be improved through assistance of inertial sensing.

Description

Technical Field

[0001] The present invention relates to optical image measurement, and in particular to a method for measuring visual displacement. Background Art

[0002] Vision-based displacement measurement methods can measure structural displacement, a key parameter reflecting the mechanical behavior and health of a structure. Currently, mainstream visual displacement measurement methods include overlapping field-of-view multi-view displacement measurement and non-overlapping field-of-view multi-view displacement measurement.

[0003] Multi-view displacement measurement with overlapping fields of view balances layout flexibility and pose recognition accuracy. Specifically, it uses at least two calibration cameras to improve the accuracy of the measurement camera's pose recognition, while also allowing for flexible adjustment of the angle between the cameras as needed. However, this presents a challenge: extrinsic calibration is required between the cameras to unify their coordinate systems. However, because traditional extrinsic calibration methods rely on feature matching in overlapping fields of view, they often require the addition of redundant cameras. This leads to the accumulation of calibration errors, impacting displacement measurement accuracy.

[0004] The key difference between non-overlapping and overlapping field-of-view multi-view displacement measurement is that the calibration cameras do not need to have overlapping fields of view, thus avoiding measurement errors caused by camera redundancy. However, non-overlapping field-of-view multi-view displacement measurement is a purely visual method that relies solely on visual information. Therefore, pose recognition is prone to failure in scenes with rapid motion or texture loss. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a visual displacement measurement method that can ensure camera pose recognition in scenes such as fast motion and texture loss.

[0006] Technical solution: A visual displacement measurement method of the present invention adopts a binocular vision-IMU system, which includes a measuring camera C M , calibrate camera C C and IMU, where the camera C is calibrated C Fixed connection with IMU;

[0007] The visual displacement measurement method comprises:

[0008] (1) Calibrate the measurement camera C separately M and calibration camera C C Internal reference;

[0009] (2) Calibrate and correct camera C C External reference between IMU;

[0010] (3) Set up the binocular vision-IMU system at the measurement site and adjust the measurement camera C M To enable it to clearly capture the target point, adjust and calibrate the camera C C Focus it on the stationary point; calibrate the measurement camera C M With calibration camera C C External reference between

[0011] (4) Measurement camera C M , calibrate camera C C Synchronously collect data with IMU; Based on template matching, the target point is measured at the camera C M Collect pixel coordinates (u, v) in the image for continuous positioning; Based on the camera affine transformation model, by measuring the camera C M The internal parameter matrix K projects the pixel coordinates (u, v) to the normalized plane:

[0012]

[0013] Among them, x c is a three-dimensional point in the normalized camera coordinate system;

[0014] Estimate the depth z through multi-view geometry or prior structure information to obtain the 3D camera coordinates X of the target point c =z·x c ;

[0015] (5) By calibrating camera C C Collect images and IMU data to calibrate camera C C Pose recognition;

[0016] (6) By calibrating camera C C The pose of the camera C is measured M and calibration camera C C The internal and external calibration results of the measurement camera C are converted M The pose of , including the rotation matrix R and translation vector t;

[0017] (7) Using measurement camera C M The pose of the target point is the three-dimensional camera coordinate X c Convert to world coordinate system:

[0018] X W =R -1 ·(X c -t)

[0019] Through continuous target positioning, the real-time world coordinates of the target point are subtracted from the world coordinates at the initial moment to obtain the six-degree-of-freedom spatial displacement.

[0020] Furthermore, the camera’s intrinsic calibration is performed using the Zhang Zhengyou method.

[0021] Furthermore, calibrate the camera C C The external parameter calibration between IMU and Kalibr toolbox is used.

[0022] Furthermore, the Kalibr toolbox is used to calibrate the extrinsic parameters between the two cameras.

[0023] Furthermore, the two cameras may have overlapping or non-overlapping fields of view.

[0024] Furthermore, when implementing the coordinate system 1, the measurement camera C M The camera coordinate system is the world coordinate system.

[0025] Furthermore, the measurement camera C M , calibrate camera C C The data synchronization acquisition with IMU is achieved through FPGA hardware trigger signal.

[0026] Furthermore, the camera image is fused with the IMU data to achieve the correction of the camera C C The pose recognition is implemented based on the VINS-Fusion toolbox.

[0027] Furthermore, the VINS-Fusion toolbox performs FAST corner extraction on camera images and feature matching through optical flow tracking; pre-integrates the IMU data and uses it to predict the camera pose and compensate for the inter-frame motion of visual tracking; finally, through tight coupling optimization of the visual reprojection error and the IMU pre-integration constraints, the six-degree-of-freedom pose of the corrected camera is output.

[0028] Furthermore, the cameras are fixed by screws at the bottom, and the two cameras can adjust their respective observation directions according to the actual measurement task.

[0029] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention uses a measuring camera to be responsible for the coordinate positioning of the target point, uses a correction camera and IMU, and realizes the recognition of the correction camera posture through vision-IMU fusion, and then calculates the posture change of the measuring camera based on the pre-calibrated external parameters of the two cameras, which is used to correct the spatial coordinates of the target point and eliminate the target positioning deviation caused by the vibration of the measuring camera. Finally, with the initial moment as a reference, the displacement of the target is calculated according to the coordinates of the target point in the time domain. With the assistance of the correction camera, the present invention can ensure the recognition of the measurement camera posture without sacrificing the displacement resolution of the measured target, and with the assistance of inertial sensing, it can improve the robustness of visual posture recognition in complex environments such as fast motion, complex lighting, and occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of a method for measuring visual displacement provided by an embodiment of the present invention;

[0031] Figure 2 This is the binocular vision-IMU system structure in the embodiment of the present invention;

[0032] Figure 3 Schematic diagram of equipment layout for indoor testing in an embodiment of the present invention;

[0033] Figure 4 is a comparison of posture parameter recognition results in a motion scene in an embodiment of the present invention;

[0034] Figure 5 is a comparison of pose parameter recognition results in a brightness interference scenario according to an embodiment of the present invention;

[0035] Figure 6 This is a comparison of displacement measurement results in a brightness interference scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings.

[0037] The embodiment of the present invention provides a method for measuring visual displacement, using a binocular vision-IMU system, such as Figure 2 As shown, the binocular vision-IMU system includes a measurement camera C M , calibrate camera C C and IMU (Inertial Measurement Unit), where IMU and calibration camera C C Fixed connection, the relative position of the two remains unchanged; the camera is fixed by screws at the bottom, and the two cameras can adjust their respective observation directions according to the actual measurement task.

[0038] like Figure 1 As shown, the visual displacement measurement method includes the following steps:

[0039] (1) Calibrate the measurement camera C using Zhang Zhengyou's method M and calibration camera C C Internal reference.

[0040] Typically, a 12x9 checkerboard calibration plate is used, sized so that it occupies the entire field of view. The camera captures at least 10 to 20 images of the checkerboard from various angles and distances, including tilt and rotation. Finally, the intrinsic parameters are calculated using the camera calibration toolbox provided by Matlab.

[0041] (2) Calibrate and correct camera C using the Kalibr toolbox C External reference between the IMU and the sensor.

[0042] Using a checkerboard calibration plate, slowly move the handheld device in a figure-eight pattern to ensure adequate IMU excitation (including rotation and translation). The camera frame rate is 20-30 Hz, and the IMU and camera are synchronized via hardware triggering. Data duration is 1-2 minutes, and the calibration plate must remain continuously visible in the camera's field of view. Ambient lighting must be uniform to avoid overexposure of the calibration plate or feature point detection failures.

[0043] The rosbag tool is used to record the data stream of the specified ROS topic (Topic) and generate a binary file that can be stored, replayed and analyzed. Finally, combined with the calibration camera C calibrated in step (1), C , execute the external reference calibration file.

[0044] (3) Calibrate the measurement camera C using the Kalibr toolbox M With calibration camera C C The external reference between .

[0045] Select an AprilTag calibration target (e.g., a 6×6 or 7×7 layout) as it supports feature detection in non-contiguous fields of view. Try to choose a large target (e.g., 1.5m×1m) to cover a wider range of motion and improve calibration robustness. Move the target in stages, moving it into the field of view of each camera. Each camera should capture images of the target in at least 10 different poses. Record a bag file containing topics from both cameras and execute the extrinsic calibration command.

[0046] Before calibration, the binocular vision-IMU system needs to be set up at the measurement site and the measurement camera C needs to be adjusted. M The observation direction and lens focal length of the measuring camera C M Able to clearly capture the target point and adjust the correction camera C C The observation direction and lens focal length of the correction camera C C Focus on a stationary point. The fields of view of the two cameras may or may not overlap.

[0047] When coordinates are processed uniformly, the measurement camera C M The camera coordinate system is the world coordinate system.

[0048] (4) Measurement camera C M , calibrate camera C C The data is collected synchronously with the IMU, and the synchronous data collection is realized by the FPGA hardware trigger signal; based on template matching, the target point is measured at the camera C M Collect pixel coordinates (u, v) in the image for continuous positioning; Based on the camera affine transformation model, by measuring the camera C M The internal parameter matrix K projects the pixel coordinates (u, v) to the normalized plane:

[0049]

[0050] Among them, x c is a three-dimensional point in the normalized camera coordinate system;

[0051] Estimate the depth z through multi-view geometry or prior structure information to obtain the 3D camera coordinates X of the target point c =z·x c .

[0052] (5) Based on the VINS-Fusion toolbox, by calibrating the camera C C Collect images and IMU data to calibrate camera C C pose recognition.

[0053] The VINS-Fusion toolbox performs FAST corner extraction on camera images and performs feature matching through optical flow tracking. It also pre-integrates IMU data and uses it to predict the camera pose, compensating for inter-frame motion during visual tracking. Finally, through tightly coupled optimization of the visual reprojection error and IMU pre-integration constraints, it outputs the corrected camera's six-degree-of-freedom pose.

[0054] (6) By calibrating camera C C The pose of the camera C is measured M and calibration camera C C The internal and external calibration results of the measurement camera C are converted M The pose of , including the rotation matrix R and the translation vector t.

[0055] (7) Using measurement camera C M The pose of the target point is the three-dimensional camera coordinate X c Convert to world coordinate system:

[0056] X W =R -1 ·(X c -t)

[0057] This step eliminates the coordinate system offset caused by the camera's own motion. Through continuous target positioning, the difference between the target point's real-time world coordinates and the initial world coordinates is calculated to obtain a six-degree-of-freedom spatial displacement that is not affected by camera motion.

[0058] The present invention is further described below with reference to indoor experiments.

[0059] Build as Figure 3 The test system shown, adjust the measurement camera C M Focus the target point clearly and calibrate the camera C CFocus on a static background. Calibrate system parameters and maintain the relative position of the two cameras unchanged in preparation for subsequent experiments.

[0060] A static baseline scenario and three disturbance scenarios are formulated, as shown in Table 1.

[0061] Table 1 Scenario design

[0062]

[0063] In the motion scene, the camera is subjected to random vibrations imposed by the measurement platform, and its posture changes, but there is no absolute true value of the posture for reference. The vision-IMU fusion method of the present invention is used to calculate the six-degree-of-freedom motion parameters ( Figure 4 ), further statistically analyze the standard deviation of each parameter in the time domain (Table), which is used to characterize the robustness of posture recognition. Compared with the vision-IMU fusion method adopted by the present invention, the method that only relies on camera C C Pose recognition, such as Figure 4 From the results, it is obvious that the standard deviation of the pose obtained by the vision-IMU fusion method is smaller, which means that the method proposed in this invention is more robust.

[0064] In other scenarios, since the camera remains stationary and only the ambient brightness changes or the texture is missing, the true value of the six-degree-of-freedom external parameter component is zero. The calibration camera continuously collects 120 times and calculates the pose parameters. Taking the brightness interference scene as an example, the pose recognition parameter recognition results are as follows: Figure 5 As shown in the figure, it can be seen that the pose recognition accuracy of the vision-IMU fusion method proposed in the present invention is higher.

[0065] In all the above four scenarios, the target point is controlled by the translation stage to undergo horizontal and vertical displacement. When extracting the pixel coordinates of the target point, the center positioning algorithm is used. The vision-IMU fusion method and the camera-only method are used to identify the posture changes, and the posture parameters are used to correct the target positioning results. The displacement measurement results obtained by the two methods are the measured values, and the values recorded by the translation stage are the true displacement values. Here, taking the brightness interference scene as an example, the displacement measurement results in the X and Y directions are analyzed, as shown in the figure. Figure 6 It can be seen that the displacement measurement results of the vision-IMU fusion method proposed in this invention are closer to the true value and have higher accuracy.

Claims

1. A method for measuring visual displacement, characterized in that: A binocular vision-IMU system is used, which includes a measurement camera C M , calibrate camera C C and IMU, where the camera C is calibrated C Fixed connection with IMU; The visual displacement measurement method comprises: (1) Calibrate the measurement camera C separately M and calibration camera C C Internal reference; (2) Calibrate and correct camera C C External reference between IMU; (3) Set up the binocular vision-IMU system at the measurement site and adjust the measurement camera C M To enable it to clearly capture the target point, adjust and calibrate the camera C C Focus it on the stationary point; calibrate the measurement camera C M With calibration camera C C External reference between (4) Measurement camera C M , calibrate camera C C Synchronously collect data with IMU; Based on template matching, the target point is measured at the camera C M Collect pixel coordinates (u, v) in the image for continuous positioning; Based on the camera affine transformation model, by measuring the camera C M The internal parameter matrix K of , projects the pixel coordinates (u, v) to the normalized plane: Among them, x c is a three-dimensional point in the normalized camera coordinate system; Estimate the depth z through multi-view geometry or prior structure information to obtain the 3D camera coordinates X of the target point c =z·x c ; (5) By calibrating camera C C Collect images and IMU data to calibrate camera C C Pose recognition; (6) By calibrating camera C C The pose of the camera C is measured M and calibration camera C C The internal and external calibration results of the measurement camera C are converted M The pose of , including the rotation matrix R and translation vector t; (7) Using measurement camera C M The pose of the target point is the three-dimensional camera coordinate X c Convert to world coordinate system: X W =R -1 ·(X c -t) Through continuous target positioning, the real-time world coordinates of the target point are subtracted from the world coordinates at the initial moment to obtain the six-degree-of-freedom spatial displacement.

2. The method for measuring visual displacement according to claim 1, wherein: The camera’s intrinsic calibration is performed using Zhang Zhengyou’s method.

3. The method for measuring visual displacement according to claim 1, wherein: Calibrate camera C C The external parameter calibration between IMU and Kalibr toolbox is used.

4. The method for measuring visual displacement according to claim 1, wherein: The Kalibr toolbox is used to calibrate the external parameters between the two cameras.

5. The method for measuring visual displacement according to claim 1, wherein: The two cameras may have overlapping or non-overlapping fields of view.

6. The method for measuring visual displacement according to claim 1, wherein: When implementing coordinate system 1, the measurement camera C M The camera coordinate system is the world coordinate system.

7. The method for measuring visual displacement according to claim 1, wherein: Measuring camera C M , calibrate camera C C The data synchronization acquisition with IMU is achieved through FPGA hardware trigger signal.

8. The method for measuring visual displacement according to claim 1, wherein: Camera image and IMU data fusion to achieve calibration of camera C C The pose recognition is implemented based on the VINS-Fusion toolbox.

9. The method for measuring visual displacement according to claim 8, wherein: The VINS-Fusion toolbox performs FAST corner extraction on camera images and performs feature matching through optical flow tracking. It also pre-integrates IMU data and uses it to predict the camera pose and compensate for inter-frame motion in visual tracking. Finally, it outputs the corrected camera's six-degree-of-freedom pose through tightly coupled optimization of the visual reprojection error and IMU pre-integration constraints.

10. The method for measuring visual displacement according to claim 1, wherein: The cameras are fixed by screws at the bottom, and the two cameras can adjust their respective observation directions according to the actual measurement task.

Images