A method for spatiotemporal calibration of ground penetrating radar and camera

Abstract

The application discloses a kind of ground penetrating radar and camera space-time calibration method, steps are, utilize ground penetrating radar, camera and wheel encoder acquisition multi-source data;Outer parameter space calibration is carried out to ground penetrating radar and camera;The original data collected are preprocessed;Different sensor data are associated;Finally, realize outer parameter time synchronization by distance constraint.The application solves the difficulty of outer parameter calibration caused by the large modal difference of camera and ground penetrating radar, lack of common view area, and the time misregistration problem caused by the inconsistent sampling frequency;Through indirect observation between multi-modal target and plane mirror, spatial alignment is realized;Based on anchor distance constraint and cost function optimization, accurate timestamp synchronization is completed.The application significantly improves the accuracy and reliability of space-time fusion of two types of sensors, reduces error accumulation, provides a reliable multi-modal sensor fusion calibration scheme for mobile robots, roadway detection and other fields, and has high engineering practicability and popularization value.

Description

Technical Field

[0001] This invention relates to a spatiotemporal calibration method, specifically a spatiotemporal calibration method for ground-penetrating radar and a camera. Background Technology

[0002] As intelligent coal mining continues to advance, identifying and forecasting hidden geological factors such as geological structures, water, gas, and underground fire zones is a crucial foundation for ensuring safe coal mine production. The underground environment is characterized by high gas levels, high dust levels, strong electromagnetic interference, and limited space, making it difficult for a single sensor to simultaneously meet the requirements of high precision, high efficiency, and explosion-proof capability. By integrating sensor data from different modalities, the system can achieve a more comprehensive and reliable environmental perception capability. For example, while cameras can effectively acquire high-resolution texture and geometric information of the roadway surface, they cannot penetrate coal and rock masses, and therefore cannot acquire information on high-threat hidden hazards such as roof delamination, aquifers, and goaf areas. Ground-penetrating radar (GPR), on the other hand, can penetrate coal and rock masses and detect hidden hazards such as underground anomalies that cameras cannot detect, but its data is presented as temporal and spatial signals, which are less intuitive and easily affected by the electrical properties of the medium. Therefore, video + 3D scene reconstruction technology can more intuitively and comprehensively display the underground working conditions, and has not only become a useful supplement to traditional monitoring methods, but has also gradually developed into the main means of remote intervention in underground mining operations.

[0003] The first step in camera-GPR data fusion is extrinsic parameter calibration, which determines the rigid transformation relationship between the camera coordinate system and the GPR coordinate system. Spatiotemporal synchronization addresses the data misalignment caused by inconsistencies in sampling frequency, delay, and motion during data acquisition. These differences make direct alignment of the two types of data difficult; precise calibration is necessary to establish their geometric and temporal correspondence—extrinsic parameter calibration and spatiotemporal synchronization. Currently, there are significant achievements in multi-sensor joint calibration for tunnel exploration, such as relatively mature calibration methods for LiDAR-camera integration and LiDAR-inertial measurement unit (IMU)-camera integration systems. However, cross-modal calibration between the camera and GPR remains insufficiently studied. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a spatiotemporal calibration method for ground penetrating radar and cameras. By combining multi-sensor collaborative acquisition and spatiotemporal calibration, the overall accuracy and adaptability of the data fusion of the two types of heterogeneous sensors are significantly improved.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a spatiotemporal calibration method for ground-penetrating radar and camera, comprising the following steps: S10: A mobile device equipped with a ground-penetrating radar, a camera, and a wheel encoder moves linearly on an overhead platform. Multiple target metal strips are spaced apart on the platform. A mother chessboard grid with a target metal ball placed on it is located on the road surface below the platform. A plane mirror is erected on the road surface in front of the platform, and a child chessboard grid is located above the plane mirror. During the movement, the mobile device simultaneously collects image data of the target metal ball and target metal strips scanned by the ground-penetrating radar, wheel speed pulse data collected by the wheel encoder, image data of the target metal ball and mother chessboard grid captured by the camera through reflection from the plane mirror, and image data of the child chessboard grid and target metal strips captured by the camera. S20: The target metal ball placed on the mother chessboard is indirectly observed by the camera through reflection from a plane mirror. The precise transformation matrix between the ground-penetrating radar coordinate system and the camera coordinate system is jointly optimized by combining the constraints of the sub-chessboard and the mother chessboard. S30: Gaussian filtering is used to reduce noise in the images of the target metal ball and the target metal strip captured by the camera. Bandpass filtering is used on the image data of the target metal ball and the target metal strip scanned by the ground penetrating radar and the pulse data of the wheel encoder to eliminate interference and extract effective features. S40: Establish data correlation between camera image sequences and ground-penetrating radar spatial locations by utilizing the anchor points mapped by the target metal strip; S50: Based on the spatial distance constraint of the target metal strip anchor point, the camera and ground penetrating radar timestamps are aligned through linear interpolation and cost function optimization to achieve time synchronization of the camera and ground penetrating radar extrinsic parameters.

[0006] Furthermore, the camera is located directly above the ground penetrating radar, the wheel encoder is installed at the wheel of the ground penetrating radar, and the surfaces of the target metal strip, the sub-chessboard grid, and the mother chessboard grid are all black and white alternating block structures.

[0007] Furthermore, in step S10, conduct The data acquisition was divided into a calibration set and a validation set in a 6:4 ratio. During each data acquisition, the target metal sphere was placed at a different vertex on the mother chessboard grid, and the position was different each time. The moving device was then used to capture and scan the target metal sphere. The ground-penetrating radar generated one scan image, and the camera captured multiple images of the target metal sphere from different angles. Zhang images, ultimately obtained Zhang Image and Zhang's ground-penetrating radar scan image.

[0008] Furthermore, the specific method for solving the precise transformation matrix between the ground-penetrating radar coordinate system and the camera coordinate system in step S20 is as follows: S201: Determine the transformation matrices of the common target metal sphere in the world coordinate system and in the camera coordinate system. and the projection plane of the plane mirror ; In the calibration of camera and ground-penetrating radar extrinsic parameters, four types of coordinate systems are involved: world coordinate system, ground-penetrating radar ... Mirror coordinate system GPR coordinate system With camera coordinate system ,in: The origin is located at the center of the GPR antenna, the Y-axis is parallel to the GPR movement direction, the Z-axis is perpendicular to the surface plane and points upward, and the X-axis is perpendicular to the GPR movement direction. The origin is located at the optical center of the camera, the Z-axis coincides with the optical axis and points in the direction of the camera's movement, and the X-axis and Y-axis are parallel to the horizontal and vertical directions of the charge-coupled device's sensor plane, respectively. Assuming in Middle Mother Chessboard Grid The center of the common target metal ball at each corner point Two-dimensional pixels projected onto the camera's image plane The equation is: (I); In the formula, This is the camera intrinsic parameter matrix, which is assumed to have been calibrated before leaving the factory; express With virtual camera coordinate system The space transformation matrix between them Represents virtual coordinates in mirror space; Indicates the first The pixel values ​​of a common target metal sphere on the camera image plane; Therefore, based on formula (I) and the correspondence between 3D and 2D projections, it is possible to determine the relationship using known parameters. and Find the coordinate transformation matrix ; and according to and Solve ; Assumption For plane mirrors The corresponding parameters, among which The mirror normal vector. Let be the orthogonal distance from the center point of the camera to the surface of the plane mirror; and ,in For plane mirrors The normal vector, For plane mirror to Orthogonal distance from the origin; , It can be calculated using the following formula: (II); In the formula, for The coordinates of any point in the array; set up express and The transformation matrix between them lie in middle On the plane, then Equal to rotation matrix The third column, and Then, based on the mirror image relationship of a plane mirror, the plane mirror in... The mirror reflection matrix can be represented as: (III); Similarly, a plane mirror in The specular reflection matrix in the image can be represented as: (IV); According to formulas (I) and (III), arrive The transformation matrix can be solved as follows: (V); The spatial transformation matrix between the camera and the plane mirror can be determined using formulas (I)-(IV). ; S202: Determine the center of the common target metal sphere. The position in the middle ,in , indicating the first A common target metal sphere, then calculate and The space transformation matrix between them; Assumption The center coordinates of the target metal sphere are as follows: , ,in , denoted as the first hyperbola on the hyperbola Point coordinates, It is the ground-penetrating radar at the The distance moved during each scan. It is the first The distance between the medium response of the common target metal sphere and the ground response during the next scan; Given the coordinates of the hyperbola's vertex; the maximum likelihood estimation method is used to... The measurement error is set to a Gaussian distribution with zero mean, and the covariance matrix is... Based on all measurements Its vertex coordinates The overall error function can be expressed as: (VI); In the formula, The characteristic parameters of a hyperbola, The index of the point on the hyperbola; According to formula (VI), the coordinates of the hyperbola's vertex position... Maximum likelihood estimation can be achieved by minimizing Obtain, among which To measure the sum of the main diagonals of the noise covariance matrix; Vertical distance from the surface detected by ground penetrating radar to the center of the common target metal sphere ,in, Indicates the first The vertical distance between the surface position of the common target metal sphere and the detection surface If the radius of the common target metal sphere is , then the... A common target metal ball in The position in the middle It can be calculated as follows: (VII); In obtaining and After defining the position coordinates, define , According to Horn's method and Spatial transformation matrix between The solution can be: (VIII); In the formula, The left singular vector matrix has column vectors that form Orthonormal basis; It is a singular value diagonal matrix; The right singular vector matrix has column vectors that form Orthonormal basis; S203: Since the two calibration processes S201 and S202 are independent of each other, the error will continue to accumulate and increase. Therefore, all calibration parameters are globally and jointly optimized. Assume the set of global optimization variables is: ,but The global optimization equation can be expressed as: (IX); In the formula, For the set of The measurement value; The error categories are numbered (k=1,2,3), specifically: This represents the error between the image points observed by the camera and the estimated points obtained through mirror reflection. The error between the center position of the target metal sphere detected by ground-penetrating radar and its actual position; For measurement The error between the points in the graph (such as the corner points of a chessboard) and the actual values; For the first Class of residuals The covariance inverse matrix of the residuals; For Huber kernel function, To iteratively adjust the weights, they can be represented as: (X); (XI); In the formula, express The residual value; It is a threshold parameter; The derivative of the Huber kernel function; After global optimization Then, using the formula The final extrinsic parameter matrix is ​​obtained. .

[0009] Furthermore, step S30 specifically involves the following steps: The anchor point of the target metal strip is defined as the intersection of the ground-penetrating radar or camera's projected trajectory on the ground and the edge line of the target metal strip; Gaussian filtering method is used for Image data of the target metal ball acquired by the time-lapse camera Transformation matrix and the obtained anchor point location information of the metal strip target Perform data preprocessing; This represents the total time series of camera measurements; The number of all anchor points on the target metal strip; the ground-penetrating radar scan data is processed using bandpass filtering. and wheel encoder output data Perform preprocessing and combine the labels as .

[0010] Furthermore, step S40 specifically includes the following steps: The time alignment of the camera, ground-penetrating radar, and wheel encoder is defined as the camera center position. Ground penetrating radar coordinate origin and anchor points The time state of three points collinear and perpendicular to the measured surface; if the ground-penetrating radar and camera can maintain this state throughout the measurement process, then the ground-penetrating radar and camera data can be considered to have achieved time alignment. Data correlation mainly includes the following two parts: Correlation between ground-penetrating radar and wheel encoder data: Ground Penetrating Radar Coordinate Origin When the vehicle travels directly above the target metal strip, a strong rectangular response is generated. At this time, the time alignment state will appear at the edge of the target metal strip. Therefore, we can use this response characteristic to realize the data association between the ground penetrating radar and the wheel encoder. Data correlation between camera, ground penetrating radar, and wheel encoder: The anchor point of the target's metal strip can be determined based on the high-contrast black and white pattern captured by the camera. Middle position ,in and They represent The two-dimensional and three-dimensional coordinates of the anchor point; according to The equation of the edge line of the target metal strip can be determined, as well as based on the camera in the sequence. and The trajectory of the movement between the two lines can determine the trajectory equation of its projection on the target metal strip, and the intersection of the two lines is the coordinate of the camera center position; therefore, the above steps can realize the data association between the camera, ground penetrating radar and wheel encoder.

[0011] Furthermore, step S50 specifically includes the following steps: Since ground-penetrating radar is triggered by a wheel encoder, it is generally assumed that the timestamps of the ground-penetrating radar and the wheel encoder are equal. The formula can be expressed as: (XII); In the formula, The reading of the wheel encoder corresponding to the starting edge of the first target metal strip; This is the reading of the wheel encoder corresponding to the edge of the c-th target metal strip; Indicates the radius of the wheels of the mobile device; This refers to the number of pulses per cycle for a wheel encoder. Indicates the speed of movement of the mobile device; Indicates the serial number of the edge line of the target metal strip. Indicates the number of metal strips on the target; Since camera frames and ground-penetrating radar scan indices correspond during ground-penetrating radar and camera detection, the definition... The camera frame index for the initial edge of the first target metal strip; The camera's virtual pose coordinates are the initial edge of the first target metal strip; for The actual camera pose coordinates at the index; then the distance traveled from the initial edge of the first target metal strip to the current pose can be expressed as: (XIII); To achieve time synchronization, it is necessary to establish a relationship between the camera frame index and the ground-penetrating radar timestamp; assuming the camera operates at a constant frame rate. If an image is captured, then the camera's first... The timestamp of the frame is: (XIV); Based on the constraint relationship between the distance between the camera and the ground-penetrating radar reaching the target metal strip, a linear interpolation method is used to establish the relationship equation between the camera and the ground-penetrating radar timestamps in formula (IV): (XV); To ensure time synchronization between the camera and the ground-penetrating radar, it is necessary to consider the time deviation. The time series of the moving camera is aligned with the ground-penetrating radar. To compensate for the time deviation, we introduce a range constraint cost function F, which includes camera reprojection range constraints, wheel encoder range constraints, and distance constraints between the two ends of the target metal strip: (XVI); In the formula, , and Let F represent the distance constraint cost functions for the camera, wheel encoder, and target metal strip, respectively, and their expressions are as follows: (XVII); In the formula, For the first A frame of a photograph is the set of all points visible to the camera. Measure the total time series for the camera; In the first The first one observed in the photograph The pixel coordinates of each feature point; The camera projection function projects the coordinates of the target point onto the image plane; for The covariance matrix; According to formula (XVII), after each optimization, the subsequent timestamps are shifted to compensate for the time offset. Iterative optimization and guarantee in subsequent data streams Ultimately, this achieves accurate time synchronization between the camera and the ground-penetrating radar.

[0012] Compared with existing technologies, this invention has the following advantages: It solves the difficulties in extrinsic parameter calibration caused by large modal differences and lack of shared field of view between cameras and ground-penetrating radars, as well as the time misalignment problem caused by inconsistent sampling frequencies. Spatial alignment is achieved through indirect observation using multimodal targets and plane mirrors; precise timestamp synchronization is achieved based on anchor point distance constraints and cost function optimization. This invention significantly improves the accuracy and reliability of spatiotemporal fusion of the two types of sensors, reduces error accumulation, and provides a reliable multimodal sensor fusion calibration scheme for fields such as mobile robots and tunnel detection, possessing high engineering practicality and promotional value. Attached Figure Description

[0013] Figure 1 This is a flowchart of the spatiotemporal calibration process of the present invention; Figure 2 This is a flowchart of the spatial calibration process for the external parameters of the ground-penetrating radar and camera of this invention. Figure 3 This is a schematic diagram of the spatiotemporal calibration scene layout structure of the present invention; Figure 4 This is a schematic diagram showing the coordinates of the center position of the target metal sphere in this invention; Figure 5 This is a diagram showing the correspondence between the camera, ground-penetrating radar, and wheel encoder during time alignment in this invention. Figure 6 This is a comparison chart of method errors in the validation set of 40 experimental groups in this embodiment of the invention; Figure 7 This is a scene diagram of the spatiotemporal synchronization experiment of the present invention; Figure 8 This is a multi-sensor 3D reconstruction data fusion diagram of the present invention.

[0014] In the diagram: 1. Mobile device, 2. Ground penetrating radar, 3. Camera, 4. Wheel encoder, 5. Overhead board, 6. Target metal ball, 7. Target metal strip, 8. Mother chessboard, 9. Child chessboard, 10. Plane mirror. Detailed Implementation

[0015] The invention will now be further described with reference to the accompanying drawings.

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 As shown, the present invention provides a spatiotemporal calibration method for ground penetrating radar and camera, the steps of which are as follows.

[0018] S10: As Figure 3 As shown, a mobile device 1 equipped with a ground-penetrating radar 2, a camera 3, and a wheel encoder 4 moves linearly on an overhead platform 5. Multiple target metal strips 7 are spaced apart on the overhead platform 5. A mother chessboard grid 8 with a target metal ball 6 placed on it is located on the road surface below the overhead platform 5. A plane mirror 10 is erected on the road surface directly in front of the overhead platform 5, and a child chessboard grid 9 is located above the plane mirror 10. During the movement, the mobile device 1 simultaneously collects image data of the target metal ball 6 and target metal strips 7 scanned by the ground-penetrating radar 2, wheel speed pulse data collected by the wheel encoder 4, image data of the target metal ball 6 and mother chessboard grid 8 captured by the camera 3 through reflection from the plane mirror 10, and image data of the child chessboard grid 9 and target metal strips 7 captured by the camera 3.

[0019] Camera 3 is positioned directly above ground-penetrating radar 2 for convenient spatial calibration. A wheel encoder 4 is mounted on the wheel of ground-penetrating radar 2; this bundled arrangement allows for distance-based data recording. The target metal strips 7, the sub-grids 9, and the mother grid 8 all feature a black-and-white interspersed block structure. Target metal strips 7 are equidistantly spaced on the overhead platform 5, while the mother grid 8 is placed on the road surface directly below the platform 5. The target metal sphere 6 and target metal strips 7, serving as a common target, can be detected simultaneously by ground-penetrating radar 2 and camera 3: the hyperbolic features of target metal sphere 6 and the strong rectangular response generated by ground-penetrating radar 2 as it passes over target metal strip 7 are captured by ground-penetrating radar 2; camera 3 can capture images of target metal sphere 6 via reflection from plane mirror 10, and the black-and-white interspersed target metal strips 7 can also be easily captured by camera 3. The images of the sub-chessboard 9 and the mother chessboard 8 were used for the calibration of the camera 3. In addition, the mother chessboard 8 was also used for the definition of the world coordinate system and the positioning of the target metal ball 6 in the world coordinate system.

[0020] One data acquisition cycle is defined as the movement of the mobile device 1 from the starting point to the ending point of the overhead panel 5, i.e., moving from the side furthest from the plane mirror 10 towards the side of the plane mirror 10; The data acquisition was divided into a calibration set and a validation set in a 6:4 ratio. During each data acquisition, the target metal ball 6 was placed at a different vertex in the mother chessboard grid 8, and the position was different each time. The moving device 1 was then used to take pictures and scan. The ground-penetrating radar 2 generated one scan image, and the camera 3 took pictures of the target metal ball 6 from multiple angles. Zhang images, ultimately obtained Zhang Image and Zhang's ground-penetrating radar 2 scan image.

[0021] S20: As Figure 2As shown, the target metal ball 6 placed on the mother chessboard 8 is indirectly observed by the camera 3 through the reflection of the plane mirror 10. Combining the constraints of the sub-chessboard 9 and the mother chessboard 8, the precise transformation matrix between the ground penetrating radar coordinate system and the camera coordinate system is jointly optimized and solved. S201: Determine the transformation matrix of the target metal sphere 6 in the world coordinate system and in the camera 3 coordinate system respectively. and the projection plane of plane mirror 10 ; The external parameter calibration of camera 3 and ground penetrating radar 2 involves four types of coordinate systems, namely: world coordinate system. Mirror coordinate system GPR coordinate system With camera coordinate system ,in: The origin is located at the center of the GPR antenna, the Y-axis is parallel to the GPR movement direction, the Z-axis is perpendicular to the surface plane and points upward, and the X-axis is perpendicular to the GPR movement direction. The origin is located at the optical center of camera 3. The Z-axis coincides with the optical axis and points in the forward direction of camera 3. The X-axis and Y-axis are parallel to the horizontal and vertical directions of the sensor plane of the charge-coupled device, respectively. Assuming in Middle Mother Chessboard 8th The target metal sphere at each corner point, center position 6 Two-dimensional pixels projected onto the camera's image plane The equation is: (I); In the formula, Here is the camera's 3 intrinsic parameter matrix, which is assumed to have been calibrated before leaving the factory; express With virtual camera coordinate system The space transformation matrix between them Represents virtual coordinates in mirror space; Indicates the first The pixel values ​​of the target metal sphere 6 on the camera image plane; Therefore, based on formula (I) and the correspondence between 3D and 2D projections, it is possible to determine the relationship using known parameters. and Find the coordinate transformation matrix ; and according to and Solve ; Assumption For plane mirror 10 in The corresponding parameters, among which The mirror normal vector. The orthogonal distance from the center point of camera 3 to the surface of plane mirror 10; and ,in For plane mirror 10 in The normal vector, For plane mirrors 10 to Orthogonal distance from the origin; , It can be calculated using the following formula: (II); In the formula, for The coordinates of any point in the array; set up express and The transformation matrix between them lie in middle On the plane, then Equal to rotation matrix The third column, and Then, based on the mirror image relationship of plane mirror 10, plane mirror 10 in... The mirror reflection matrix can be represented as: (III); Similarly, plane mirror 10 in The specular reflection matrix in the image can be represented as: (IV); According to formulas (I) and (III), arrive The transformation matrix can be solved as follows: (V); The spatial transformation matrix between camera 3 and plane mirror 10 can be determined according to formulas (I)-(IV). ; S202: Determine the center of the target metal sphere 6. The position in the middle ,in , indicating the first Six target metal spheres were then calculated. and The space transformation matrix between them; Assumption The center coordinates of the target metal sphere 6 are as follows: , ,in , denoted as the first hyperbola on the hyperbola Point coordinates, It is the ground-penetrating radar 2 in the The distance moved during each scan. It is the first The distance between the medium response and the ground response of the target metal sphere 6 during the next scan; Given the coordinates of the hyperbola's vertex; the maximum likelihood estimation method is used to... The measurement error is set to a Gaussian distribution with zero mean, and the covariance matrix is... Based on all measurements Its vertex coordinates The overall error function can be expressed as: (VI); In the formula, The characteristic parameters of a hyperbola, The index of the point on the hyperbola; According to formula (VI), the coordinates of the hyperbola's vertex position... Maximum likelihood estimation can be achieved by minimizing Obtain, among which To measure the sum of the main diagonals of the noise covariance matrix; like Figure 4 As shown, the vertical distance from the surface of the ground-penetrating radar 2 to the center of the target metal sphere 6 is... ,in, Indicates the first The vertical distance between the surface position of the target metal sphere 6 and the detection surface If the radius of the target metal sphere 6 is 6, then the 6th 6 target metal balls The position in the middle It can be calculated as follows: (VII); In obtaining and After defining the position coordinates, define , According to Horn's method and Spatial transformation matrix between The solution can be: (VIII); In the formula, The left singular vector matrix has column vectors that form Orthonormal basis; It is a singular value diagonal matrix; The right singular vector matrix has column vectors that form Orthonormal basis; S203: Since the two calibration processes S201 and S202 are independent of each other, the error will continue to accumulate and increase. Therefore, all calibration parameters are globally and jointly optimized. Assume the set of global optimization variables is: ,but The global optimization equation can be expressed as: (IX); In the formula, For the set of The measurement value; The error categories are numbered (k=1,2,3), specifically: This represents the error between the image points observed by camera 3 and the estimated points obtained by mirroring them through plane mirror 10. The error between the center position of the target metal sphere 6 detected by the ground penetrating radar 2 and its actual position; For measurement The error between the points in the graph (such as the corner points of a chessboard) and the actual values; For the first Class of residuals The covariance inverse matrix of the residuals; For Huber kernel function, To iteratively adjust the weights, they can be represented as: (X); (XI); In the formula, express The residual value; It is a threshold parameter; The derivative of the Huber kernel function; After global optimization Then, using the formula The final extrinsic parameter matrix is ​​obtained. .

[0022] S30: Gaussian filtering is used to reduce noise in the images of the target metal ball 6 and the target metal strip 7 captured by the camera 3. Bandpass filtering is performed on the image data of the target metal ball 6 and the target metal strip 7 scanned by the ground penetrating radar 2 and the pulse data of the wheel encoder 4 to eliminate interference and extract effective features. The anchor point of the target metal strip 7 is defined as the intersection of the ground-penetrating radar 2 or camera 3's projected trajectory on the ground and the edge line of the target metal strip 7; Gaussian filtering is used for... Image data of the target metal sphere 6 acquired by camera 3 Transformation matrix and the obtained anchor point location information of the metal strip target Perform data preprocessing; This indicates that camera 3 measures the total time series. , The number of all anchor points on the target metal strip 7; the ground-penetrating radar 2 scan data were processed using bandpass filtering. Output data of wheel encoder 4 Perform preprocessing and combine the labels as .

[0023] S40: Using the anchor points mapped by the target metal strip 7, establish a data association between the image sequence of camera 3 and the spatial position of ground penetrating radar 2; like Figure 5 As shown, the time alignment of the camera 3, ground penetrating radar 2, and wheel encoder 4 is defined as the center position of camera 3. Ground penetrating radar 2 coordinate origin and anchor points The time state where three points are collinear and perpendicular to the measured surface; if the ground penetrating radar 2 and camera 3 can maintain this state throughout the measurement process, then the data from the ground penetrating radar 2 and camera 3 can be considered to have achieved time alignment. Data correlation mainly includes the following two parts: Data association between ground-penetrating radar 2 and wheel encoder 4: Ground Penetrating Radar 2 Coordinate Origin When the vehicle travels directly above the target metal strip 7, a strong rectangular response is generated. At this time, the time alignment state will appear at the edge of the target metal strip 7. Therefore, we can use this response characteristic to realize the data association between the ground penetrating radar 2 and the wheel encoder 4. Data association between camera 3, ground penetrating radar 2, and wheel encoder 4: Based on the high-contrast black and white pattern captured by camera 3, the anchor point of the target metal strip 7 can be determined. Middle position ,in and They represent The two-dimensional and three-dimensional coordinates of the anchor point; according to The equation of the edge line of the target metal strip 7 can be determined, and the sequence of the camera 3 can be used to determine the edge line equation. and The trajectory of the two lines can determine the trajectory equation of its projection on the target metal strip 7, and the intersection of the two lines is the center position coordinate of the camera 3; therefore, the above steps can realize the data association between the camera 3, the ground penetrating radar 2 and the wheel encoder 4.

[0024] S50: Based on the spatial distance constraint of the anchor point of the target metal strip 7, the timestamps of camera 3 and ground penetrating radar 2 are aligned through linear interpolation and cost function optimization to achieve time synchronization of external parameters between camera 3 and ground penetrating radar 2; Since the ground-penetrating radar 2 is triggered by the wheel encoder 4, it is generally assumed that the timestamps of the ground-penetrating radar 2 and the wheel encoder 4 are equal. The formula can be expressed as: (XII); In the formula, The reading of the wheel encoder 4 corresponding to the starting edge of the first target metal strip 7; The reading of the wheel encoder 4 corresponding to the edge of the c-th target metal strip 7; Indicates the radius of the wheels of mobile device 1; This refers to the number of pulses in 4 cycles for a wheel encoder. Indicates the moving speed of mobile device 1; This indicates the serial number of the edge line of the target metal strip 7. Indicates the number of target metal strips 7; Since the camera 3 frames correspond to the ground penetrating radar 2 scan index during the detection process of ground penetrating radar 2, the definition is... The camera's 3-frame index is set for the initial edge of the first target metal strip 7; The virtual pose coordinates of camera 3 are the initial edge of the first target metal strip 7; for The true pose coordinates of camera 3 at the index; then the distance traveled from the initial edge of the first target metal strip 7 to the current pose can be expressed as: (XIII); To achieve time synchronization, it is necessary to establish a relationship between the camera 3 frame index and the ground penetrating radar 2 timestamp; assuming camera 3 operates at a constant frame rate. If images are acquired, then camera 3... The timestamp of the frame is: (XIV); Based on the constraint relationship between the distances from camera 3 and ground-penetrating radar 2 to the target metal strip 7, the relationship equation between the timestamps of camera 3 and ground-penetrating radar 2 is established using linear interpolation method on formula (IV): (XV); To ensure time synchronization between camera 3 and ground-penetrating radar 2, it is necessary to adjust the time deviation accordingly. The time series of the moving camera 3 is aligned with that of the ground-penetrating radar 2. To compensate for the time deviation, we introduce a range constraint cost function F, which includes the reprojection range constraint of camera 3, the range constraint of wheel encoder 4, and the range constraints at both ends of the target metal strip 7. (XVI); In the formula, , and Let the distance constraint cost functions for camera 3, wheel encoder 4, and target metal strip 7 be represented respectively, and their expressions are as follows: (XVII); In the formula, For the first The set of all points visible to the frame photo camera 3; Measure the total time series for camera 3; In the first The first one observed in the photograph The pixel coordinates of each feature point; The camera 3 projection function projects the coordinates of the target to be measured onto the image plane; for The covariance matrix; According to formula (XVII), after each optimization, the subsequent timestamps are shifted to compensate for the time offset. Iterative optimization and guarantee in subsequent data streams Ultimately, this achieves accurate time synchronization between camera 3 and ground-penetrating radar 2.

[0025] Example 1: To verify the effectiveness of the proposed method, 100 calibration experiments were conducted, divided into a calibration set and a validation set in a 6:4 ratio. In each experiment, the target metal ball 6 was placed at the vertices of different squares in the mother chessboard 8, and the data acquisition position was different each time. The moving device 1 was pushed to take pictures and scan. The ground penetrating radar 2 generated one scan image, and the camera 3 took pictures of the target metal ball 6 from multiple angles to generate 20 camera images. Finally, 2000 camera images and 100 ground penetrating radar 2 scan images were obtained.

[0026] The target metal sphere 6, which is the common observation object of the ground penetrating radar 2 and camera 3, is selected as the calibration target. The Euclidean distance between the estimated center position of the target metal sphere 6 in the world coordinate system and the measured position of the target metal sphere 6 is used as the error metric. (XVIII); In the formula, Determined directly from the validation set; and Estimated from the calibration set; It is calculated by formula (VII), where the parameters are derived from the validation set.

[0027] The ground-penetrating radar 2 used in this embodiment is model LTD-2600, and its resolution calculation formula is as follows. (XIX); In the formula, The speed at which electromagnetic waves propagate in a medium; Given the center frequency of the antenna, the theoretical resolution of the ground-penetrating radar 2 is calculated to be 16.6 cm.

[0028] This embodiment selects the stepwise calibration method (i.e., the traditional method of first independently calibrating camera 3 and then calibrating ground-penetrating radar 2) as a comparative experiment to illustrate the spatiotemporal calibration method of external parameters proposed in this invention (i.e., Figure 6 The superiority of the joint calibration method (in this context, the error of the two methods on 40 validation sets) and the results. The comparison results are as follows Figure 6 As shown.

[0029] Depend on Figure 6 Experimental data showed that the average error of the stepwise calibration method was 11.812 mm, the standard deviation was 18.403 mm, and the root mean square error (RMSE) was 12.567 mm. This indicates that the stepwise calibration method exhibits some error fluctuations during calibration, especially with a relatively large standard deviation, suggesting that the error distribution is quite dispersed across different experimental settings and calibration processes, affecting the stability and consistency of the results. In contrast, the average error of the extrinsic spatiotemporal calibration method proposed in this embodiment is reduced to 7.735 mm, the standard deviation is 17.091 mm, and the RMS error is 8.770 mm. Although the standard deviation still fluctuates somewhat, the error of the extrinsic spatiotemporal calibration method is significantly reduced compared to the stepwise calibration method, particularly the significant reduction in the RMS error, indicating an improvement in calibration accuracy in this embodiment. Furthermore, the error of the extrinsic spatiotemporal calibration method is only 65.5% of that of the stepwise calibration method, demonstrating a clear advantage in reducing error and providing more accurate and consistent calibration results. In summary, the extrinsic spatiotemporal calibration method effectively reduces error propagation and accumulation caused by factors such as inconsistent sensor positions and inaccurate time synchronization, thereby improving the overall accuracy of data fusion.

[0030] Example 2: To verify the effectiveness of the proposed method, this embodiment selects ground-penetrating radar-wheel encoder fusion alone and asynchronous wheel encoder-camera fusion as a comparative experiment to illustrate the contribution of this invention. The experimental scenario is as follows: Figure 7 As shown. The experimental procedure is as follows: 1. Consider the space between two target metal strips 7 as a segment, and select the distance from the end edge to the start edge of each target metal strip 7 segment. As an evaluation metric, to assess the overall geometric accuracy of the mapped environment; 2. Three independent synchronous experiments were conducted. In each experiment, three target metal strips 7 were laid out. The mobile device 1 was pushed to scan along the four pre-laid measurement lines and the average value was taken. The experiment was repeated three times to obtain three-dimensional reconstruction data and compare the end-to-end distance of each pair of mobile devices 1.

[0031] The error metric in this embodiment is defined as follows: The error metric for ground-penetrating radar-wheel encoder is defined as follows: The unsynchronized wheel encoder-camera fusion error metric is defined as follows: The expression is: (XXI); In the formula, For each segment of the target metal strip, the virtual pose of the ground-penetrating radar in the last frame of frame 7 is determined; For each segment of the target metal strip, the first frame of the ground-penetrating radar 2 virtual pose is used; For the first Measured length of metal strip 7 of the target segment; The virtual pose of the ground-penetrating radar in the last frame of the 7th frame of the target metal strip was not synchronized; The virtual pose of the ground-penetrating radar in the first frame of the 7th frame of the target metal strip is not synchronized.

[0032]

[0033] The experimental error results are shown in Table 1. The error value corresponding to the serial number 7 of the target metal strip in each group is the average of the errors of the four measurement lines. The experimental results show that the present invention has a significant advantage in improving the accuracy of data fusion between two types of heterogeneous sensors. In particular, compared with the traditional asynchronous wheel encoder-camera joint method, the present invention can significantly reduce the error from 60.28 mm to 9.24 mm, demonstrating the effectiveness of the present invention in solving the sensor time synchronization problem and improving data consistency and accuracy. The significant reduction in error not only verifies the high-precision calibration capability of the present invention in processing heterogeneous sensor data, but also improves the adaptability and robustness of the overall system in dynamic environments.

[0034] Cloud Compare software was selected to visualize the point cloud model, and the detailed 3D reconstruction results are shown below. Figure 8 As shown, Figure 8 The upper half shows the 3D reconstruction from the camera, and the lower half shows the synchronized ground-penetrating radar image. The red point cloud represents the camera's movement trajectory, the yellow point cloud represents the ground-penetrating radar's movement trajectory, and the white line segments in the lower half represent distance information recorded by the wheel encoder. Visualization demonstrates the effectiveness of the proposed spatiotemporal calibration method in real-world physical scenarios, enabling precise alignment of camera and ground-penetrating radar data and achieving 3D fusion of multi-sensor data.

[0035] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0036] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any minor modifications, equivalent substitutions, and improvements made to the above embodiments based on the technical essence of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatiotemporal calibration method for ground-penetrating radar and camera, characterized in that, Includes the following steps: S10: A mobile device (1) equipped with a ground-penetrating radar (2), a camera (3) and a wheel encoder (4) moves in a straight line on an overhead plate (5). Multiple target metal strips (7) are spaced apart on the overhead plate (5). A mother chessboard (8) with a target metal ball (6) is placed on the road surface below the overhead plate (5). A plane mirror (10) is erected on the road surface in front of the overhead plate (5). A child chessboard (9) is placed above the plane mirror (10). During the movement, the mobile device (1) simultaneously collects image data of the target metal ball (6) and target metal strips (7) scanned by the ground-penetrating radar (2), wheel speed pulse data collected by the wheel encoder (4), image data of the target metal ball (6) and mother chessboard (8) captured by the camera (3) through the plane mirror (10), and image data of the child chessboard (9) and target metal strips (7) captured by the camera (3). S20: The camera (3) indirectly observes the target metal ball (6) placed on the mother chessboard (8), and combines the constraints of the sub chessboard (9) and the mother chessboard (8) to jointly optimize and solve the transformation matrix between the ground-penetrating radar coordinate system and the camera coordinate system. S30: Gaussian filtering is used to reduce noise in the images of the target metal ball (6) and the target metal strip (7) captured by the camera (3), and bandpass filtering is performed on the image data of the target metal ball (6) and the target metal strip (7) scanned by the ground penetrating radar (2) and the pulse data of the wheel encoder (4). S40: Using the anchor points mapped by the target metal strip (7), establish the data association between the image sequence of the camera (3) and the spatial position of the ground penetrating radar (2); S50: Based on the spatial distance constraint of the target metal strip (7) anchor point, the camera (3) and the ground penetrating radar (2) timestamps are aligned through linear interpolation and cost function optimization to achieve time synchronization of the external parameters of the camera (3) and the ground penetrating radar (2).

2. The spatiotemporal calibration method for ground-penetrating radar and camera according to claim 1, characterized in that, The camera (3) is located directly above the ground penetrating radar (2), and the wheel encoder (4) is set at the wheel of the ground penetrating radar (2). The surfaces of the target metal strip (7), the sub-chessboard (9) and the mother chessboard (8) are all black and white interspersed blocks.

3. The spatiotemporal calibration method for ground-penetrating radar and camera according to claim 1, characterized in that, In step S10 conduct The data acquisition is divided into a calibration set and a validation set in a 6:4 ratio. In each data acquisition, the target metal ball (6) is placed at the vertices of different squares in the mother chessboard (8), and the position is different in each data acquisition. The moving device (1) is pushed to take pictures and scan. The ground penetrating radar (2) generates a scan image, and the camera (3) takes pictures of the target metal ball (6) from multiple angles to generate Zhang images, ultimately obtained Zhang Image and Zhang's ground-penetrating radar (2) scan image.

4. The spatiotemporal calibration method for ground-penetrating radar and camera according to claim 1, characterized in that, The specific method for solving the transformation matrix between the ground-penetrating radar coordinate system and the camera coordinate system in step S20 is as follows: S201: Determine the transformation matrix of the target metal sphere (6) in the world coordinate system in the camera (3) coordinate system. The projection plane of the plane mirror (10) ; In the calibration of the external parameters of the camera (3) and the ground penetrating radar (2), four types of coordinate systems are involved, namely: world coordinate system Mirror coordinate system GPR coordinate system With camera coordinate system ,in: The origin is located at the center of the GPR antenna, the Y-axis is parallel to the GPR movement direction, the Z-axis is perpendicular to the surface plane and points upward, and the X-axis is perpendicular to the GPR movement direction. The origin is located at the optical center of the camera (3), the Z-axis coincides with the optical axis and points in the forward direction of the camera (3), and the X-axis and Y-axis are parallel to the horizontal and vertical directions of the sensor plane of the charge-coupled device, respectively; Assuming in Middle mother chessboard (8) The center position of the target metal ball (6) at each corner point Two-dimensional pixels projected onto the camera's image plane The equation is: （I）； In the formula, For the camera (3) intrinsic parameter matrix; express With virtual camera coordinate system The space transformation matrix between them Represents virtual coordinates in mirror space; Indicates the first The pixel values ​​of the target metal ball (6) on the camera image plane; Therefore, based on formula (I) and the correspondence between 3D and 2D projections, it is possible to determine the relationship using known parameters. and Find the coordinate transformation matrix ; and according to and Solve ; Assumption For the plane mirror (10) in The corresponding parameters, among which The mirror normal vector. The orthogonal distance from the center point of camera (3) to the surface of plane mirror (10); and ,in For the plane mirror (10) in The normal vector, For plane mirror (10) to Orthogonal distance from the origin; , It can be calculated using the following formula: (II); In the formula, for The coordinates of any point in the array; set up express and The transformation matrix between them lie in middle On the plane, then Equal to rotation matrix The third column, and Then, based on the mirror image relationship of the plane mirror (10), the plane mirror (10) in... The mirror reflection matrix can be represented as: (III); Similarly, the plane mirror (10) in The specular reflection matrix in the image can be represented as: (IV); According to formulas (I) and (III), arrive The transformation matrix can be solved as follows: （V）； The spatial transformation matrix between the camera (3) and the plane mirror (10) can be determined according to formulas (I)-(IV). ; S202: Determine the center of the target metal sphere (6) at... The position in the middle ,in , indicating the first Target metal spheres (6), then calculate and The space transformation matrix between them; Assumption The center coordinates of the lower target metal sphere (6) are as follows: , ,in , denoted as the first hyperbola on the hyperbola Point coordinates, Ground penetrating radar (2) in the first The distance moved during each scan. It is the first The distance between the medium response and the ground response of the target metal sphere (6) during the second scan; Given the coordinates of the hyperbola's vertex; the maximum likelihood estimation method is used to... The measurement error is set to a Gaussian distribution with zero mean, and the covariance matrix is... Based on all measurements Its vertex coordinates The overall error function can be expressed as: (WE); In the formula, The characteristic parameters of a hyperbola, The index of the point on the hyperbola; According to formula (VI), the coordinates of the hyperbola's vertex position... Maximum likelihood estimation can be achieved by minimizing Obtain, among which To measure the sum of the main diagonals of the noise covariance matrix; Ground penetrating radar (2) detects the vertical distance from the surface to the center of the target metal sphere (6). ,in, Indicates the first The vertical distance between the surface position of the target metal sphere (6) and the detection surface. If the radius of the target metal sphere (6) is , then the first... Target metal spheres (6) in The position in the middle It can be calculated as follows: (VII); In obtaining and After defining the position coordinates, define , According to Horn's method and Spatial transformation matrix between The solution can be: (VIII); In the formula, The left singular vector matrix has column vectors that form Orthonormal basis; It is a singular value diagonal matrix; The right singular vector matrix has column vectors that form Orthonormal basis; S203: Perform global joint optimization of all calibration parameters; Assume the set of global optimization variables is: ,but The global optimization equation can be expressed as: (IX); In the formula, For the set of The measurement value; The error categories are numbered (k=1,2,3), specifically: This represents the error between the image points observed by the camera (3) and the estimated points obtained by mirroring through the plane mirror (10). The error between the center position of the target metal ball (6) detected by the ground penetrating radar (2) and its actual position; For measurement The error between the points in the graph and the true value; For the first Class of residuals The covariance inverse matrix of the residuals; For Huber kernel function, To iteratively adjust the weights, they can be represented as: （X）； (XI); In the formula, express The residual value; It is a threshold parameter; The derivative of the Huber kernel function; After global optimization Then, using the formula The final extrinsic parameter matrix is ​​obtained. .

5. The spatiotemporal calibration method for ground-penetrating radar and camera according to claim 1, characterized in that, The specific steps of step S30 are as follows: The anchor point of the target metal strip (7) is defined as the intersection of the ground-penetrating radar (2) or camera (3) projection trajectory on the ground and the edge line of the target metal strip (7); Gaussian filtering method is used for Image data of the target metal ball (6) acquired by the time-lapse camera (3) Transformation matrix and the obtained anchor point location information of the metal strip target Perform data preprocessing; This indicates that camera (3) measures the total time series; The number of all anchor points on the target metal strip (7); the ground-penetrating radar (2) scan data was analyzed using bandpass filtering. Output data of wheel encoder (4) Perform preprocessing and combine the labels as .

6. The spatiotemporal calibration method for ground-penetrating radar and camera according to claim 1, characterized in that, The specific steps of step S40 are as follows: The time alignment of the camera (3), ground penetrating radar (2), and wheel encoder (4) is defined as the center position of the camera (3). Ground penetrating radar (2) coordinate origin and anchor point The moment state of three collinear points perpendicular to the measured surface; data association includes the following two parts: Data association between ground-penetrating radar (2) and wheel encoder (4): Ground Penetrating Radar (2) Coordinate Origin When the vehicle travels directly above the target metal strip (7), a rectangular response will be generated. At this time, the time alignment state will appear at the edge of the target metal strip (7). This response characteristic is used to realize the data association between the ground penetrating radar (2) and the wheel encoder (4). Data association between camera (3), ground penetrating radar (2), and wheel encoder (4): Based on the high-contrast black and white pattern captured by camera (3), the anchor point of the target metal strip (7) can be determined. Middle position ,in and They represent The two-dimensional and three-dimensional coordinates of the anchor point; according to The edge line equation of the target metal strip (7) can be determined, and the sequence of the camera (3) can be used to determine the edge line equation. and The trajectory of the two lines can determine the trajectory equation of the projection onto the target metal strip (7), and the intersection of the two lines is the center position coordinate of the camera (3); therefore, the data association of the camera (3), the ground penetrating radar (2) and the wheel encoder (4) can be realized.

7. The spatiotemporal calibration method for ground-penetrating radar and camera according to claim 4, characterized in that, The specific steps of step S50 are as follows: The timestamps of the ground-penetrating radar (2) and the wheel encoder (4) are equal, as expressed by the formula: (XII); In the formula, The reading of the wheel encoder (4) corresponding to the starting edge of the first target metal strip (7); The reading of the wheel encoder (4) corresponding to the edge of the c-th target metal strip (7); Indicates the radius of the wheel of the moving device (1); The number of pulses per cycle for the wheel encoder (4); Indicates the moving speed of the mobile device (1); Indicates the serial number of the edge line of the target metal strip (7), Indicates the number of target metal strips (7); Since the camera (3) frame corresponds to the ground penetrating radar (2) scan index during the detection process of the ground penetrating radar (2), the definition is... For the initial edge of the first target metal strip (7), the camera (3) frame index; The camera (3) virtual pose coordinates for the initial edge of the first target metal strip (7); for The true pose coordinates of the camera (3) at the index; then the distance traveled from the initial edge of the first target metal strip (7) to the current pose can be expressed as: (XIII); Assume camera (3) operates at a constant frame rate When the image is acquired, the camera (3) The timestamp of the frame is: (XIV); Based on the constraint relationship between the distances from the camera (3) and the ground-penetrating radar (2) to the target metal strip (7), the relationship equation between the timestamps of the camera (3) and the ground-penetrating radar (2) is established by using linear interpolation on formula (IV): (XV); A distance constraint cost function F is introduced to compensate for time deviation, including the reprojection distance constraint of the camera (3), the distance constraint of the wheel encoder (4), and the distance constraint at both ends of the target metal strip (7): (XVI); In the formula, , and The distance constraint cost functions for the camera (3), the wheel encoder (4), and the target metal strip (7) are respectively expressed as follows: (XVII); In the formula, For the first The set of all points visible to the frame photo camera (3); Measure the total time series for camera (3); In the first The first one observed in the photograph The pixel coordinates of each feature point; (3) Projection function of the camera that projects the coordinates of the target to the image plane; for The covariance matrix; According to formula (XVII), after each optimization, the subsequent timestamps are shifted to compensate for the time offset. Iterative optimization and guarantee in subsequent data streams Ultimately, accurate time synchronization between the camera (3) and the ground-penetrating radar (2) is achieved.

Images