Multi-source space-time parameter online self-calibration method and system of mobile measurement system

Abstract

The invention belongs to the technical field of surveying and mapping, and particularly relates to a multi-source space-time parameter online self-calibration method and system of a mobile measurement system. The method comprises the following steps: acquiring three-dimensional point cloud data, a two-dimensional image and state data and spatio-temporal data of a mobile measurement system; respectively extracting a first geometric feature of the corrected three-dimensional point cloud data and a second geometric feature of the two-dimensional image, and projecting the first geometric feature to a two-dimensional image plane to establish a target feature pair; constructing a joint optimization model of a unified space-time reference, and taking the target feature pair as a feature constraint to optimize the to-be-optimized state variable; and executing online optimization solution and parameter updating of the joint optimization model based on the sliding window. The method overcomes the defects that in the prior art, the offline calibration process is tedious, parameter dynamic drift cannot be adapted, the precision is not high due to space-time parameter separation solution in online calibration, and the robustness is poor in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of surveying and mapping technology, specifically relating to an online self-calibration method and system for multi-source spatiotemporal parameters of a mobile measurement system, which is particularly suitable for the online self-calibration of spatial extrinsic parameters and time offsets among multiple heterogeneous sensors (e.g., lidar, camera, inertial measurement unit, global navigation satellite system) carried by a mobile measurement system. Background Technology

[0002] A Mobile Measurement System (MMS) is a platform that integrates multiple sensors, such as LiDAR, cameras, GNSS receivers, and IMUs, into a single unit. Examples include surveying vehicles, drones, and robots. It can rapidly acquire large-scale, high-precision 3D geospatial information while in motion. To fuse data collected by different sensors, such as LiDAR point clouds, camera images, and IMU motion data, under a unified spatiotemporal reference, it is essential to accurately understand their spatiotemporal relationships, namely: 1) Spatial extrinsic parameters: the rotation and translation relationships of each sensor's coordinate system relative to the platform's coordinate system, where the platform's coordinate system is typically based on the IMU coordinate system; 2) Time offset: the deviation of each asynchronous sensor's timestamp from the reference time, where the reference time is typically IMU or GNSS time.

[0003] In existing related technologies, Chinese patent document CN117053835A discloses an online calibration method for vehicle-mounted sensors. First, it acquires high-precision map information within a preset area. When a vehicle passes through the preset area, it acquires road information and the vehicle's relative pose within the preset area using the vehicle-mounted sensor. Based on the relative pose, it stitches the road information together to obtain the vehicle's observation information within the preset area. It then matches the observation information with the first high-precision map information and calculates the deviation between the two. If the deviation is greater than a preset value, the vehicle-mounted sensor is corrected. This method solves the problems of high algorithm difficulty and low calibration accuracy in existing online calibration methods for vehicle sensors.

[0004] Furthermore, Chinese patent document CN113340298A discloses a method for calibrating the extrinsic parameters of inertial navigation and dual-antenna GNSS. This method includes the following steps: initializing the inertial navigation system with the output of the dual-antenna GNSS; predicting the attitude, velocity, and position of the carrier using gyroscopes and accelerometers; and then using a Kalman filter to estimate the online extrinsic parameters of the inertial navigation system and the dual-antenna GNSS based on the position, velocity, and heading output of the GNSS. Compared to traditional methods that utilize specialized optical equipment to achieve higher measurement accuracy, this method requires no additional specialized equipment, saving calibration costs and improving calibration efficiency.

[0005] However, the current calibration of spatiotemporal parameters mainly suffers from the following problems: 1. Offline calibration methods are cumbersome and cannot adapt dynamically: Traditional methods rely on dedicated calibration fields, such as areas with checkerboard patterns or special geometries, to collect and process data manually or semi-automatically. This process is time-consuming and labor-intensive, and cannot solve the problem of minute drifts in spatiotemporal parameters caused by factors such as vibration, temperature changes, and long-term use in actual operation. These drifts accumulate errors and seriously affect the accuracy of the final data product.

[0006] 2. Limitations of online calibration methods: Although current research on online calibration is a hot topic, it is still imperfect, mainly in the following aspects: 1) For separate calibration: Most methods calibrate the extrinsic parameters separately from the temporal parameters, for example, first estimating the time offset, and then fixing the time parameters to solve for the extrinsic parameters. This approach ignores the strong coupling between spatiotemporal parameters, which can easily lead to the solution getting trapped in local optima and failing to achieve the global optimal accuracy; 2) Regarding calibration integrated into SLAM: While some advanced SLAM (Simultaneous Localization and Mapping) algorithms include online calibration, their primary goal is to serve the localization and mapping of the vehicle, with insufficient attention paid to the observability, convergence, and accuracy of the calibration parameters themselves. When the vehicle's motion is insufficient, such as prolonged uniform linear motion, the calibration may diverge or produce erroneous results. 3) Single features and constraints: Existing methods mostly rely on single types of features, such as point-to-point or line-to-line matching. In scenarios with sparse features or simple textures, such as tunnels, highways, and large bodies of water, feature matching has poor robustness and is prone to failure. 4) Non-continuous correction: Many online calibrations are performed once during system initialization or triggered after a large error is detected. They are not a "precision preservation" mechanism that continuously and smoothly corrects minor parameter drifts throughout the entire operation.

[0007] Therefore, there is an urgent need for a technical solution that can automatically, continuously, and with high precision jointly optimize and correct all spatiotemporal parameters during the normal operation of a mobile measurement system without relying on specific calibration objects. Summary of the Invention

[0008] This invention aims to overcome the technical defects of existing spatiotemporal parameter calibration technologies for mobile measurement systems, such as cumbersome offline calibration processes, inability to adapt to dynamic parameter drift, low accuracy due to the separation of spatiotemporal parameters in online calibration, and poor robustness in complex scenarios. It provides a method and system for online self-calibration of multi-source spatiotemporal parameters for mobile measurement systems.

[0009] The detailed technical solution of this invention is as follows: A method for online self-calibration of multi-source spatiotemporal parameters of a mobile measurement system, the method comprising: S1. Acquire multi-source measurement data from the mobile measurement system, wherein the multi-source measurement data includes collected 3D point cloud data, 2D images, and the status data and spatiotemporal data of the mobile measurement system; S2. The three-dimensional point cloud data is corrected, the first geometric feature of the corrected three-dimensional point cloud data is extracted, and the second geometric feature of the two-dimensional image is extracted. The first geometric feature of the three-dimensional point cloud data is projected onto the two-dimensional image plane, and a mapping relationship between the first geometric feature of the three-dimensional point cloud data and the second geometric feature of the two-dimensional image is established based on preset verification conditions to form a target feature pair. S3. Construct a joint optimization model with a unified spatiotemporal reference, and use the target feature pairs as feature constraints to optimize the state variables to be optimized. The state variables to be optimized include the carrier state at key frame time and spatiotemporal parameters. S4. Perform online optimization and parameter update of the joint optimization model based on a sliding window.

[0010] According to a preferred embodiment of the present invention, in step S2, extracting the first geometric feature of the corrected three-dimensional point cloud data specifically includes: Calculate the local curvature value of each three-dimensional point in the corrected three-dimensional point cloud data; Compare the local curvature value with the preset curvature value. If: If the local curvature value is greater than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as an edge point; If the local curvature value is less than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as a planar point; The edge points and the planar points are clustered respectively to form LiDAR edge features as the first geometric feature. and LiDAR planar features ; And, extracting the second geometric features of the two-dimensional image, specifically including: A preset line segment detection algorithm is used to extract line segment features from the two-dimensional image and represent them as image edge features. .

[0011] According to a preferred embodiment of the present invention, in step S2, projecting the first geometric feature of the three-dimensional point cloud data onto a two-dimensional image plane specifically includes: LiDAR edge features for 3D point cloud data and LiDAR planar features Based on the estimated values ​​of LiDAR time offset and camera time offset in the current spatiotemporal parameters, the LiDAR edge features are calculated. and LiDAR planar features The actual acquisition time is used to determine the corresponding two-dimensional image from all two-dimensional images as the target image. The LiDAR edge features are calculated by interpolating the state data from the mobile measurement system. and LiDAR planar features The carrier pose at each point at the time of data acquisition; Based on the carrier pose of each point at the time of acquisition, it is transformed from the LiDAR coordinate system to the world coordinate system and the camera coordinate system in sequence, and finally projected into image pixel coordinates, which respectively form a projection dashed line and a two-dimensional projection area.

[0012] According to a preferred embodiment of the present invention, in step S2, interpolation is performed using the state data of the mobile measurement system to calculate the LiDAR edge features. The carrier pose of each edge point at the time of acquisition, specifically including: All high-frequency IMU measurements of the LiDAR scan frame between the start and end times are acquired and pre-integrated by the IMU to obtain the pre-integrated value of the LiDAR scan frame; wherein, the high-frequency IMU measurement value is defined as the measurement value corresponding to the inertial measurement unit IMU sampling frequency being higher than the LiDAR scan frequency; The LiDAR edge features from the start time to the start time are calculated using the IMU pre-integral interpolation. The relative pose transformation of the carrier at each edge point acquisition time; Multiplying the relative pose transformation of the carrier with the carrier pose at the start of the LiDAR scan frame yields the carrier pose of each edge point at the time of acquisition.

[0013] According to a preferred embodiment of the present invention, in step S2, based on the carrier pose of the edge point at the time of acquisition, it is sequentially transformed from the LiDAR coordinate system to the world coordinate system and the camera coordinate system, and finally projected into image pixel coordinates, specifically including: Let the coordinates of the edge point in the LiDAR coordinate system be... ; The edge points are transformed from the LiDAR coordinate system to the carrier coordinate system using LiDAR extrinsic parameters to obtain the carrier coordinate system coordinates. : , Indicates LiDAR external parameters; Using the carrier pose of the edge points at the time of acquisition, the carrier coordinate system coordinates are... Transform to the world coordinate system to obtain world coordinate system coordinates. : , This represents the carrier pose of the edge points at the time of data acquisition. Using the camera's pose in the world coordinate system, the world coordinate system coordinates are... Transform to the camera coordinate system to obtain the camera coordinate system coordinates. : , This represents the camera's pose in the world coordinate system, and , Indicates camera external parameters; Using camera intrinsic matrix The camera coordinate system coordinates are determined using a pinhole camera model. Projection to image pixel coordinates , This indicates transpose.

[0014] According to a preferred embodiment of the present invention, in step S2, the preset verification conditions include geometric structure consistency verification and photometric information consistency verification; wherein, the geometric structure consistency verification includes: Directional consistency: Calculate the angle between the direction vector of the projected dashed line and the direction vector of the candidate image edge features. If the angle is less than a first preset angle threshold, directional consistency is satisfied. The candidate image edge features are defined as: image edge features falling within a strip-shaped region centered on the projected dashed line and with a preset pixel width. These are considered as candidate image edge features; Position overlap: Calculate the distance between the center point of the projected dashed line and the edge features of the candidate image. If the distance is less than the first preset pixel threshold, the position overlap is satisfied. The consistency verification of the photometric information includes: Obtain the LiDAR edge features Edge points with reflectance greater than a preset intensity threshold are identified, and their average grayscale gradient within the corresponding projection area of ​​the target image is calculated. If the average gray-level gradient corresponding to the edge point is greater than the preset gray-level gradient threshold, then the consistency of photometric information is satisfied.

[0015] According to a preferred embodiment of the present invention, in step S2, establishing a mapping relationship between the first geometric features of the three-dimensional point cloud data and the second geometric features of the two-dimensional image based on preset verification conditions to form a target feature pair specifically includes: Candidate image edge features that pass the geometric structure consistency verification are used as target image edge features. ; The LiDAR edge features formed by the edge points that have passed the consistency verification of the photometric information. As the edge feature of the target LiDAR ; The target LiDAR edge features and target image edge features Constructing target feature pairs .

[0016] According to a preferred embodiment of the present invention, in step S3, the feature constraint includes: Kinematic constraints: Introducing IMU factors, where an IMU factor connects two adjacent state nodes, indicating that the IMU measurement constrains the relative motion at consecutive time points; Cross-modal geometric constraints: Introducing LiDAR-camera geometric factors, which apply the target features to... For each target LiDAR edge feature and target image edge features The target feature pairs are used to construct a point-to-line distance residual as a geometric constraint for the LiDAR camera. Absolute positioning constraints: Introducing GNSS factors, where a GNSS factor is derived from the state of a certain carrier. The node is drawn out to indicate that the absolute position of the carrier at that moment is constrained by GNSS observations; The construction of the LiDAR-camera geometric constraints specifically includes: From the target LiDAR edge features Select several edge points. For any one of these edge points, its coordinates in the LiDAR coordinate system are: The current state variable to be optimized is projected onto the image plane to obtain the projection point. ; Edge features of the target image Represented as a straight line in the image ; Calculate edge points geometric residuals That is, the projection point to the straight line orthogonal distance; The residual terms of all edge points contained in the target feature pair are used to construct a LiDAR-camera geometric constraint. And / or, the spatiotemporal parameters of the state variable to be optimized include LiDAR extrinsic parameters. LiDAR time offset Camera external parameters and camera time offset .

[0017] According to a preferred embodiment of the present invention, in step S3, the objective function of the joint optimization model is expressed as:

[0018] In the formula: It represents the set of all state variables to be optimized; , , These are the IMU pre-integration residuals, GNSS position residuals, and LiDAR-camera point-line geometric residuals, respectively. Represents the square of the Mahalanobis distance. This is the covariance matrix corresponding to the measurement noise.

[0019] In another aspect of the present invention, a system is provided for implementing the online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system as described above, the system comprising at least: The data acquisition module is used to acquire multi-source measurement data from the mobile measurement system. The multi-source measurement data includes collected three-dimensional point cloud data, two-dimensional images, and the status data and spatiotemporal data of the mobile measurement system. The feature extraction and association module is used to perform correction processing on the three-dimensional point cloud data, extract the first geometric features of the corrected three-dimensional point cloud data, and extract the second geometric features of the two-dimensional image, project the first geometric features of the three-dimensional point cloud data onto the two-dimensional image plane, and establish a mapping relationship between the first geometric features of the three-dimensional point cloud data and the second geometric features of the two-dimensional image based on preset verification conditions to form a target feature pair; The joint optimization module is used to construct a joint optimization model with a unified spatiotemporal reference. The target feature pair is used as a feature constraint to optimize the state variables to be optimized. The state variables to be optimized include the carrier state at key frame time and spatiotemporal parameters. The parameter management and update module is used to perform online optimization and parameter updates of the joint optimization model based on a sliding window.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention achieves full automation and high efficiency: no manual intervention or special calibration field is required, and calibration is automatically completed during normal system operation, which greatly improves the efficiency and convenience of operation.

[0021] (2) The present invention achieves high precision and strong coupling processing: by constructing a unified joint optimization model, all spatiotemporal parameters (i.e., the external parameters of the sensor and their time offset) and the carrier trajectory are solved together, which fully considers the strong coupling relationship between the parameters, avoids the suboptimal problem caused by step-by-step solution, and achieves higher calibration accuracy.

[0022] (3) The present invention achieves high robustness: by extracting and associating multiple types of structured natural features (points, lines, and surfaces), and by adopting a dual verification mechanism of geometry and photometry, the feature matching success rate and calibration robustness in challenging scenarios such as weak texture and weak structure are significantly improved.

[0023] (4) The present invention achieves continuous accuracy maintenance: The present invention provides a continuously running background accuracy maintenance mechanism that can compensate for minor parameter drifts caused by environmental changes and equipment aging in real time, ensuring that the mobile measurement system can maintain the highest fusion data quality throughout its entire life cycle and operation. Attached Figure Description

[0024] Figure 1 This is a flowchart of the online self-calibration method for multi-source spatiotemporal parameters of the mobile measurement system described in Embodiment 1 of the present invention.

[0025] Figure 2 This is a schematic diagram of cross-modal structured natural feature extraction and association in Embodiment 1 of the present invention.

[0026] Figure 3 This is a schematic diagram of the factor graph representation of the joint optimization model with a unified spatiotemporal reference in Embodiment 1 of the present invention.

[0027] Figure 4 This is a structural block diagram of the system that implements the online self-calibration method for multi-source spatiotemporal parameters of the mobile measurement system in Embodiment 2 of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] In the case of no conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0032] The present invention aims to overcome the technical defects existing in the offline calibration process of the spatio-temporal parameter calibration technology of the existing mobile measurement system, which is cumbersome, unable to adapt to parameter dynamic drift, the spatio-temporal parameter separation and solution in online calibration results in low accuracy, and poor robustness in complex scenarios. A multi-source spatio-temporal parameter online self-calibration method for a mobile measurement system is provided. During the normal operation of the mobile measurement system, by extracting and associating the cross-modal natural features in the data of multiple sensors in real time, a unified joint optimization model is constructed and solved, so as to continuously and optimally estimate and dynamically correct the carrier trajectory of the mobile measurement system and the spatio-temporal parameters (including spatial extrinsic parameters and time offsets) between multiple sensors.

[0033] The following further illustrates the multi-source spatio-temporal parameter online self-calibration method and system of the mobile measurement system of the present invention in conjunction with specific embodiments.

[0034] Embodiment 1: Refer Figure 1 , this embodiment provides a multi-source spatio-temporal parameter online self-calibration method for a mobile measurement system, and the method includes: S1. Obtain multi-source measurement data of the mobile measurement system, and the multi-source measurement data includes the collected three-dimensional point cloud data, two-dimensional images, and the state data and spatio-temporal data of the mobile measurement system.

[0035] The mobile measurement system serves as a carrier, and various sensors are integrated thereon to collect observation data with time stamps. In this embodiment, the sensors integrated in the mobile measurement system include a LiDAR, a camera, an inertial measurement unit IMU, and a GNSS receiver.

[0036] The system comprises several components: a LiDAR (Light Detection and Ranging) system that acquires 3D point cloud data at a preset frequency (e.g., 10Hz), including the coordinates, reflection intensity, and timestamps of each 3D point; a camera that acquires 2D image data at a preset frequency (e.g., 20Hz); an IMU (Inertial Measurement Unit) that acquires state data of the mobile measurement system at a preset frequency (e.g., 200Hz), including three-axis angular velocity and three-axis acceleration data; and a GNSS receiver that acquires spatiotemporal data of the mobile measurement system at a preset frequency (e.g., 5Hz), including absolute position.

[0037] Understandably, the coordinate system of the inertial measurement unit (IMU) is defined as the body frame (B), and its clock is used as the time reference. The camera's intrinsic parameters have been pre-calibrated. Strict hardware synchronization is not performed between the LiDAR, the camera, and the IMU.

[0038] S2. The three-dimensional point cloud data is corrected, the first geometric feature of the corrected three-dimensional point cloud data is extracted, and the second geometric feature of the two-dimensional image is extracted. The first geometric feature of the three-dimensional point cloud data is projected onto the two-dimensional image plane, and a mapping relationship between the first geometric feature of the three-dimensional point cloud data and the second geometric feature of the two-dimensional image is established based on preset verification conditions to form a target feature pair.

[0039] In this embodiment, the correction processing of the 3D point cloud data mainly involves motion distortion correction. Specifically, IMU pre-integration is performed using the state data during the time interval between two consecutive frames of point cloud data from a LiDAR scan to estimate the pose transformation of each 3D point at the acquisition time relative to the start time of the scan of that frame. This corrects all 3D points within the frame to the same coordinate system at the same time, thereby obtaining a frame of distortion-free corrected point cloud.

[0040] Then, the first geometric features of the corrected 3D point cloud data are extracted, denoted as LiDAR features, such as edge point sets and planar fragments. The specific implementation is as follows: Calculate the local curvature value of each three-dimensional point in the corrected three-dimensional point cloud data; Compare the local curvature value with the preset curvature value. If: If the local curvature value is greater than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as an edge point; If the local curvature value is less than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as a planar point.

[0041] Reference Figure 2 As shown on the left, in three-dimensional space, three-dimensional points at the corners of a building are identified as edge points, and three-dimensional points on the wall surface are identified as planar points.

[0042] Furthermore, the edge points and the planar points are clustered respectively to form LiDAR edge features as the first geometric feature. and LiDAR planar features .

[0043] Simultaneously, the second geometric features of the two-dimensional image, such as edge line segments, are extracted. Specifically, a preset line segment detection algorithm, such as the LSD algorithm, is used to extract straight line segment features from the two-dimensional image and represent them as image edge features. .

[0044] Reference Figure 2 As shown on the right, a clear corner outline is detected in the camera image, which is the image edge feature.

[0045] Subsequently, the first geometric features of the 3D point cloud data are projected onto the 2D image plane, and a mapping relationship between the first geometric features of the 3D point cloud data and the second geometric features of the 2D image is established based on preset verification conditions to establish a highly reliable cross-modal feature correspondence.

[0046] Specifically, for LiDAR edge features of 3D point cloud data It can utilize a pre-stored set of estimated values ​​for current spatiotemporal parameters and project them onto a corresponding camera image. In this embodiment, the spatiotemporal parameters include LiDAR extrinsic parameters. LiDAR time offset Camera external parameters and camera time offset The projection process is as follows: First, based on the LiDAR time offset in the current spatiotemporal parameters and camera time offset The estimated value is used to calculate the LiDAR edge features. The actual acquisition time (relative to the IMU time base) is determined, and the camera image with the closest actual acquisition time from all the two-dimensional images acquired by the camera is selected as the target image.

[0047] Subsequently, the state data acquired by the inertial measurement unit (IMU) is interpolated to calculate the LiDAR edge features. The precise carrier pose at each edge point at the time of acquisition. This process can be implemented based on IMU pre-integration techniques well-known in the art. The interpolation and calculation process includes: a) IMU pre-integration: For a LiDAR scan frame, acquire all high-frequency IMU measurements (angular velocity and acceleration) between its start and end times. Here, "high-frequency" refers to the IMU's sampling frequency being significantly higher than the LiDAR's scanning frequency, typically above 100Hz, such as 200Hz in this embodiment, to ensure accurate capture of the carrier's minute movements during a single LiDAR scan frame. By integrating these IMU measurements over the time period, a relative motion increment independent of the absolute pose at the start time can be obtained, i.e., the IMU pre-integration.

[0048] b) Pose interpolation calculation: for the LiDAR edge features Any edge point on the surface has a precise acquisition timestamp. Using the IMU pre-integration results described above, the distance from the start time of the LiDAR frame to the acquisition timestamp can be accurately interpolated and calculated. The relative pose transformation of the carrier at each moment. Multiplying this relative transformation by the carrier pose at the start of the frame yields the precise carrier pose of the edge point at the moment of acquisition. This process also effectively corrects motion distortion in LiDAR point clouds.

[0049] Finally, the edge points are sequentially transformed from their own coordinate system to the world coordinate system and the camera coordinate system, and finally projected into image pixel coordinates to form a projected dashed line.

[0050] The coordinate transformation here can be achieved based on standard three-dimensional rigid body transformation, and the specific implementation process is as follows: Let the coordinates of an edge point in its own coordinate system L be... The precise carrier pose at the moment of acquisition (i.e., the pose of carrier coordinate system B in world coordinate system W) is: The LiDAR extrinsic parameters to be calibrated (i.e., the transformation from the LiDAR coordinate system L to the carrier coordinate system B) are as follows: The camera's extrinsic parameters (i.e., the transformation from the camera coordinate system C to the carrier coordinate system B) are: The camera intrinsic parameter matrix is .So: 1) Transform to world coordinate system: First, convert the point... Transform from LiDAR coordinate system to carrier (IMU) coordinate system to obtain Then, using the precise carrier pose at that point... Transform it to the world coordinate system to obtain ; 2) Transform to camera coordinate system: Next, transform the points in the world coordinate system... Transform to the camera coordinate system. This requires the camera's pose in the world coordinate system. . Precise carrier pose and camera external parameters To obtain, that is Therefore, the coordinates of the point in the camera coordinate system are: ; 3) Projecting to image pixel coordinates: Finally, using the camera intrinsic parameter matrix... Using a pinhole camera model, the edge points in the camera coordinate system are... Projection to image pixel coordinates , This indicates transpose.

[0051] By analyzing the LiDAR edge features By performing the above transformation on all edge points, a projected dashed line can be formed on the target image.

[0052] Meanwhile, for LiDAR planar features of 3D point cloud data It also performs projection association operations. Specifically, it constitutes the features of the LiDAR plane. The projection transformation of each planar point from the LiDAR coordinate system to the image pixel coordinates is mathematically identical to the projection operation of the aforementioned edge points. This is due to the characteristics of the LiDAR planar surface. Composed of multiple points distributed in a planar pattern in space, its overall projection result forms a two-dimensional projection region on the target image. Then, this LiDAR planar feature... It is associated with the corresponding image region features. For example, by constructing a "photometric error" constraint, it is required that the projection area of ​​the LiDAR planar fragment on the target image has photometric consistency. This planar-based constraint complements the edge-based line-to-line constraint, jointly enhancing the robustness of calibration in different scenarios.

[0053] It should be noted that the estimated values ​​of the spatiotemporal parameters here refer to the latest optimization results provided by the system's parameter management and update module up to the current moment. Since this method employs a continuously iterative online optimization framework, all parameters to be calibrated are constantly being estimated and corrected throughout the process, and are not fixed true values; hence, they are called "estimated values." The optimization process itself is a computational process of optimally estimating the state variables using the nonlinear least squares method.

[0054] Furthermore, the preset verification conditions in this embodiment include geometric structure consistency verification and photometric information consistency verification.

[0055] The geometric consistency verification involves searching within the image neighborhood of the projected dashed line. Here, "neighborhood" refers to a strip-shaped region centered on the projected dashed line with a preset pixel width (e.g., 20 pixels). All image edge features falling within this neighborhood are considered. All of them are considered as candidate image edge features.

[0056] Subsequently, the direction and position of the projected dashed line are compared with the edge features of each candidate image. The specific judgment criteria are as follows: a) Directional Consistency: Calculate the angle between the direction vector of the projected dashed line and the direction vector of the candidate image edge features. If this angle is less than a first preset angle threshold, for example... =10°, then the directional consistency is satisfied.

[0057] b) Positional overlap: Calculate the overlap between the projected dashed line and the edge features of the candidate image. For example, the distance between the center points of two line segments can be calculated; if this distance is less than a first preset pixel threshold, for example... =5 pixels, then the position overlap is satisfied.

[0058] A candidate image edge feature is considered to have passed the geometric consistency verification and is used as the target image edge feature only when it simultaneously meets the above criteria for directional consistency and positional overlap. And the LiDAR edge features corresponding to the projected dashed line. This forms the initial feature pair.

[0059] The consistency verification of photometric information: This verification targets the edge features that constitute LiDAR. The process is performed on the original edge points. Specifically, for an initial feature pair that has passed the above geometric verification, the features constituting the LiDAR edge are further compared. The physical properties of edge points (especially those with high reflectivity) and the pixel properties of their projected positions on the target image are used to eliminate false matches. The specific criteria for eliminating false matches are as follows: 1) Information Extraction: Obtaining LiDAR Edge Features Edge points with high reflectance values ​​(e.g., greater than 0.7 after normalization) are identified. Simultaneously, the average grayscale gradient value of these edge points within the neighborhood of their projected locations on the target image (e.g., a 3x3 pixel window) is calculated.

[0060] 2) Establish verification rules: High reflectivity is usually caused by abrupt changes in the material or color of an object, which corresponds to significant grayscale changes in the image. Therefore, a grayscale gradient threshold is set, for example, a threshold of 50. The verification rule is: if the normalized reflectivity of an edge point is greater than 0.7, then the average grayscale gradient of its corresponding target image region must be greater than 50. Conversely, if the reflectivity of an edge point is very low, it should not be projected into a strong gradient region in the target image.

[0061] 3) Eliminating false matches: Feature correspondences that do not meet the above verification rules will be considered false matches and eliminated. For example, if a high-reflectivity edge point projects onto a region of uniform color and low grayscale gradient on the target image, this is likely a false match.

[0062] Based on the above criteria, edge points that meet the requirements are selected and used to form the target LiDAR edge features. .

[0063] Understandably, a feature pair is only considered a highly reliable cross-modal feature correspondence if it passes both of the above verifications simultaneously. Therefore, target LiDAR edge features that pass both verifications are considered highly reliable. and target image edge features They will be constructed as target feature pairs, denoted as .

[0064] S3. Construct a joint optimization model with a unified spatiotemporal reference, and use the target feature pair as feature constraints to optimize the state variables to be optimized. The state variables to be optimized include the carrier state at key frame time and spatiotemporal parameters.

[0065] This embodiment employs a factor graph-based nonlinear least squares optimization framework to construct a joint optimization model, whose structural parameters... Figure 3 .

[0066] Within a sliding window containing N keyframes, the set of state variables to be optimized includes: a) The carrier state at N keyframe moments, where carrier state refers to the state of the motion measurement system. For example... Figure 3 circular nodes As shown, Used as a carrier state, it represents the carrier in the first... The state at each keyframe moment, each node represents the carrier state at one keyframe moment. Understandably, the carrier state here... Includes the pose of the carrier at that moment (i.e., the pose of the carrier coordinate system B in the world coordinate system W) ) and speed ,Right now .

[0067] b) Spatiotemporal parameters, such as Figure 3 As shown in the diamond-shaped node, this parameter contains LiDAR extrinsic parameters. LiDAR time offset Camera external parameters and camera time offset , denoted as spatiotemporal parameter .

[0068] The objective function for model optimization is constructed to minimize the sum of Mahalanobis distances for all constraint terms. These constraint terms are the target feature pairs established in step S2. The measurement data from other sensors are converted into mathematical constraints on the state variables, specifically implemented as follows: 1) Kinematic constraints (IMU factors): Introducing IMU factors, where one IMU factor connects two adjacent state nodes, such as... and This indicates that the IMU measurement constrains the relative motion at consecutive moments.

[0069] 2) Cross-modal geometric constraints (LiDAR-camera geometric factors): This is the core constraint of this invention, which directly applies the target feature pairs verified in step S2. For each target LiDAR edge feature and target image edge features The target feature pairs are used to construct a point-to-line distance residual. Specifically: a) From the target LiDAR edge features Several sampling points are selected above. For any one of these sampling points, its coordinates in the LiDAR coordinate system are: Using the current state variables to be optimized, including the carrier state and spatiotemporal parameters Following the coordinate transformation process described in step S2, the coordinates are projected onto the image plane to obtain the projection points. .

[0070] b) Edge features of the target image In the image, it can be represented by the equation of a straight line. : .

[0071] c) This sampling point geometric residuals That is, the projection point to the straight line The orthogonal distance.

[0072] A target feature pair can construct multiple such residual terms, that is, each sampling point has one residual term. These residual terms together constitute the LiDAR-camera geometric constraint, which associates a geometric observation (feature pair) with the vehicle pose, the extrinsic parameters of the two sensors, and the time offset. When the parameters are inaccurate, the projection points will deviate from the image line and the residual will increase; the optimization process is to adjust all parameters to make the residual tend to zero.

[0073] 3) Absolute positioning constraint: Introduce GNSS factors. One GNSS factor is led out from a node of a certain vehicle state indicating that the absolute position of the vehicle at this moment is constrained by the GNSS observation.

[0074] Based on the above constraints, the objective function of the joint optimization model can be expressed as:

[0075] In the formula: represents the set composed of all state variables to be optimized; , , are the IMU pre-integration residual, the GNSS position residual, and the LiDAR-camera point-line geometric residual respectively; represents the square of the Mahalanobis distance, is the covariance matrix of the corresponding measurement noise.

[0076] By solving this non-linear least squares problem, the vehicle trajectory and all spatio-temporal parameters can be optimized simultaneously.

[0077] S4. Perform online optimization solution and parameter update of the joint optimization model based on a sliding window.

[0078] Within a fixed-size key-frame sliding window, use non-linear optimization algorithms such as the Gauss-Newton method or the Levenberg-Marquardt method to solve the above joint optimization model to obtain the incremental correction of spatio-temporal parameters. Through a smoothing filtering mechanism, such as weighted average and other methods, update the corrected spatio-temporal parameters to the mobile measurement system to achieve continuous and smooth correction of the parameters. The methods used are all well-known technical means in the art and will not be elaborated here.

[0079] Embodiment 2 Refer Figure 4 , this embodiment provides a system for implementing an online self-calibration method for multi-source spatio-temporal parameters of a mobile measurement system. The system at least includes a data acquisition module and a processing unit 100. The processing unit 100 at least includes a feature extraction and association module 101, a joint optimization module 102, and a parameter management and update module 103.

[0080] Specifically, the data acquisition module is used to acquire multi-source measurement data from the mobile measurement system, which includes collected 3D point cloud data, 2D images, and the status data and spatiotemporal data of the mobile measurement system.

[0081] In this embodiment, the data acquisition module includes at least a LiDAR, a camera, an inertial measurement unit (IMU), and a GNSS receiver.

[0082] The system comprises several components: a LiDAR (Light Detection and Ranging) system that acquires 3D point cloud data at a preset frequency (e.g., 10Hz), including the coordinates, reflection intensity, and timestamps of each 3D point; a camera that acquires 2D image data at a preset frequency (e.g., 20Hz); an IMU (Inertial Measurement Unit) that acquires state data of the mobile measurement system at a preset frequency (e.g., 200Hz), including three-axis angular velocity and three-axis acceleration data; and a GNSS receiver that acquires spatiotemporal data of the mobile measurement system at a preset frequency (e.g., 5Hz), including absolute position.

[0083] The LiDAR, camera, inertial measurement unit (IMU), and GNSS receiver are all communicatively connected to the processing unit 100 to transmit the acquired data to the processing unit 100.

[0084] Understandably, the coordinate system of the inertial measurement unit (IMU) is defined as the body frame (B), and its clock is used as the time reference. The camera's intrinsic parameters have been pre-calibrated. Strict hardware synchronization is not performed between the LiDAR, the camera, and the IMU.

[0085] The feature extraction and association module 101 is used to perform correction processing on the three-dimensional point cloud data, extract the first geometric features of the corrected three-dimensional point cloud data, and extract the second geometric features of the two-dimensional image, project the first geometric features of the three-dimensional point cloud data onto the two-dimensional image plane, and establish a mapping relationship between the first geometric features of the three-dimensional point cloud data and the second geometric features of the two-dimensional image based on preset verification conditions to form a target feature pair.

[0086] In this embodiment, the correction processing of the three-dimensional point cloud data mainly involves correcting its motion distortion.

[0087] Specifically, the feature extraction and association module 101 uses the state data during the time interval between two consecutive frames of point cloud data from LiDAR scanning to perform IMU pre-integration, and estimates the pose transformation of each three-dimensional point at the acquisition time relative to the start time of the frame scan, thereby correcting all three-dimensional points in the frame to the same coordinate system at the same time, so as to obtain a frame of distortion-free corrected point cloud.

[0088] Then, the feature extraction and association module 101 extracts the first geometric features of the corrected 3D point cloud data, denoted as LiDAR features, such as edge point sets and planar fragments. Specifically: Calculate the local curvature value of each 3D point in the corrected 3D point cloud data; Compare the local curvature value with the preset curvature value. If: If the local curvature value is greater than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as an edge point; If the local curvature value is less than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as a planar point.

[0089] In three-dimensional space, the system identifies three-dimensional points at the corners of a building as edge points and three-dimensional points on the wall surface as planar points.

[0090] Furthermore, the feature extraction and association module 101 clusters the edge points and the planar points respectively to form LiDAR edge features as the first geometric feature. and LiDAR planar features .

[0091] Simultaneously, the feature extraction and association module 101 extracts the second geometric features of the two-dimensional image, such as edge line segments. Specifically, the feature extraction and association module 101 uses a preset line segment detection algorithm, such as the LSD algorithm, to extract straight line segment features in the two-dimensional image and represent them as image edge features. .

[0092] Detecting a clear corner outline in a camera image is known as an image edge feature.

[0093] Subsequently, the feature extraction and association module 101 projects the first geometric features of the three-dimensional point cloud data onto the two-dimensional image plane, and establishes a mapping relationship between the first geometric features of the three-dimensional point cloud data and the second geometric features of the two-dimensional image based on preset verification conditions, so as to establish a highly reliable cross-modal feature correspondence relationship.

[0094] Specifically, for one of the LiDAR edge features The feature extraction and association module 101 uses a set of estimated values ​​of current spatiotemporal parameters stored in the parameter management and update module 103 to project them onto a corresponding camera image. In this embodiment, the spatiotemporal parameters include LiDAR extrinsic parameters. LiDAR time offset Camera external parameters and camera time offset .

[0095] The projection process includes: First, the feature extraction and association module 101 uses the LiDAR time offset in the current spatiotemporal parameters. and camera time offset The estimated value is used to calculate the LiDAR edge features. The actual acquisition time (relative to the IMU time reference) is determined, and the camera image with the closest actual acquisition time from all the two-dimensional images acquired by the camera is selected as the target image. Subsequently, the feature extraction and association module 101 uses the state data collected by the inertial measurement unit (IMU) to perform interpolation and calculate the LiDAR edge features. The precise carrier pose at each edge point at the time of acquisition. This process can be implemented based on IMU pre-integration techniques well-known in the art. The interpolation and calculation process includes: a) IMU pre-integration: For a LiDAR scan frame, acquire all high-frequency IMU measurements (angular velocity and acceleration) between its start and end times. Here, "high-frequency" refers to the IMU's sampling frequency being significantly higher than the LiDAR's scanning frequency, typically above 100Hz, such as 200Hz in this embodiment, to ensure accurate capture of the carrier's minute movements during a single LiDAR scan frame. By integrating these IMU measurements over the time period, a relative motion increment independent of the absolute pose at the start time can be obtained, i.e., the IMU pre-integration.

[0096] b) Pose interpolation calculation: for the LiDAR edge features Any edge point on the surface has a precise acquisition timestamp. Using the IMU pre-integration results described above, the distance from the start time of the LiDAR frame to the acquisition timestamp can be accurately interpolated and calculated. The relative pose transformation of the carrier at each moment. Multiplying this relative transformation by the carrier pose at the start of the frame yields the precise carrier pose of the edge point at the moment of acquisition. This process also effectively corrects motion distortion in LiDAR point clouds.

[0097] Finally, the feature extraction and association module 101 transforms the edge points from their own coordinate system to the world coordinate system and the camera coordinate system in sequence, and finally projects them into image pixel coordinates to form a projected dashed line.

[0098] The coordinate transformation here can be achieved based on standard three-dimensional rigid body transformation, and the specific implementation process is as follows: Let the coordinates of an edge point in its own coordinate system L be... The precise carrier pose at the moment of acquisition (i.e., the pose of carrier coordinate system B in world coordinate system W) is: The LiDAR extrinsic parameters to be calibrated (i.e., the transformation from the LiDAR coordinate system L to the carrier coordinate system B) are as follows: The camera's extrinsic parameters (i.e., the transformation from the camera coordinate system C to the carrier coordinate system B) are: The camera intrinsic parameter matrix is .So: 1) Transform to world coordinate system: First, convert the point... Transform from LiDAR coordinate system to carrier (IMU) coordinate system to obtain Then, using the precise carrier pose at that point... Transform it to the world coordinate system to obtain ; 2) Transform to camera coordinate system: Next, transform the points in the world coordinate system... Transform to the camera coordinate system. This requires the camera's pose in the world coordinate system. . Precise carrier pose and camera external parameters To obtain, that is Therefore, the coordinates of the point in the camera coordinate system are: ; 3) Projecting to image pixel coordinates: Finally, using the camera intrinsic parameter matrix... Using a pinhole camera model, the edge points in the camera coordinate system are... Projection to image pixel coordinates , This indicates transpose.

[0099] By analyzing the LiDAR edge features By performing the above transformation on all edge points, a projected dashed line can be formed on the target image.

[0100] Meanwhile, for LiDAR planar features of 3D point cloud data The feature extraction and association module 101 also performs projection association operations. Specifically, it constructs the features of the LiDAR plane. The projection transformation of each planar point from the LiDAR coordinate system to the image pixel coordinates is mathematically identical to the projection operation of the aforementioned edge points. This is due to the characteristics of the LiDAR planar surface. Composed of multiple points distributed in a planar pattern in space, its overall projection result forms a two-dimensional projection region on the target image. Then, this LiDAR planar feature... It is associated with the corresponding image region features. For example, by constructing a "photometric error" constraint, it is required that the projection area of ​​the LiDAR planar fragment on the target image has photometric consistency. This planar-based constraint complements the edge-based line-to-line constraint, jointly enhancing the robustness of calibration in different scenarios.

[0101] It should be noted that the estimated values ​​of the spatiotemporal parameters here refer to the latest optimization results provided by the system's parameter management and update module up to the current moment. Since this method employs a continuously iterative online optimization framework, all parameters to be calibrated are constantly being estimated and corrected throughout the process, and are not fixed true values; hence, they are called "estimated values." The optimization process itself is a computational process of optimally estimating the state variables using the nonlinear least squares method.

[0102] Furthermore, the preset verification conditions in this embodiment include geometric structure consistency verification and photometric information consistency verification.

[0103] The geometric consistency verification involves searching within the image neighborhood of the projected dashed line. Here, "neighborhood" refers to a strip-shaped region centered on the projected dashed line with a preset pixel width (e.g., 20 pixels). All image edge features falling within this neighborhood are considered. All of them are considered as candidate image edge features.

[0104] Subsequently, the direction and position of the projected dashed line are compared with the edge features of each candidate image. The specific judgment criteria are as follows: a) Directional Consistency: Calculate the angle between the direction vector of the projected dashed line and the direction vector of the candidate image edge features. If this angle is less than a first preset angle threshold, for example... =10°, then the directional consistency is satisfied.

[0105] b) Positional overlap: Calculate the overlap between the projected dashed line and the edge features of the candidate image. For example, the distance between the center points of two line segments can be calculated; if this distance is less than a first preset pixel threshold, for example... =5 pixels, then the position overlap is satisfied.

[0106] A candidate image edge feature is considered to have passed the geometric consistency verification and is used as the target image edge feature only when it simultaneously meets the above criteria for directional consistency and positional overlap. And the LiDAR edge features corresponding to the projected dashed line. This forms the initial feature pair.

[0107] The consistency verification of photometric information: This verification targets the edge features that constitute LiDAR. The process is performed on the original edge points. Specifically, for an initial feature pair that has passed the above geometric verification, the features constituting the LiDAR edge are further compared. The physical properties of edge points (especially those with high reflectivity) and the pixel properties of their projected positions on the target image are used to eliminate false matches. The specific criteria for eliminating false matches are as follows: 1) Information Extraction: Obtaining LiDAR Edge Features Edge points with high reflectance values ​​(e.g., greater than 0.7 after normalization) are identified. Simultaneously, the average grayscale gradient value of these edge points within the neighborhood of their projected locations on the target image (e.g., a 3x3 pixel window) is calculated.

[0108] 2) Establish verification rules: High reflectivity is usually caused by abrupt changes in the material or color of an object, which corresponds to significant grayscale changes in the image. Therefore, a grayscale gradient threshold is set, for example, a threshold of 50. The verification rule is: if the normalized reflectivity of an edge point is greater than 0.7, then the average grayscale gradient of its corresponding target image region must be greater than 50. Conversely, if the reflectivity of an edge point is very low, it should not be projected into a strong gradient region in the target image.

[0109] 3) Eliminating false matches: Feature correspondences that do not meet the above verification rules will be considered false matches and eliminated. For example, if a high-reflectivity edge point projects onto a region of uniform color and low grayscale gradient on the target image, this is likely a false match.

[0110] Based on the above criteria, edge points that meet the requirements are selected and used to form the target LiDAR edge features. .

[0111] Understandably, a feature pair is only considered a highly reliable cross-modal feature correspondence if it passes both of the above verifications simultaneously. Therefore, target LiDAR edge features that pass both verifications are considered highly reliable. and target image edge features They will be constructed as target feature pairs, denoted as And it is sent to the joint optimization module 102 as a “feature constraint”.

[0112] The joint optimization module 102 is used to construct a joint optimization model with a unified spatiotemporal reference, and uses the target feature pair as feature constraints to optimize the state variables to be optimized. The state variables to be optimized include the carrier state at key frame time and spatiotemporal parameters.

[0113] This embodiment uses a factor graph-based nonlinear least squares optimization framework to construct the joint optimization model.

[0114] Within a sliding window containing N keyframes, the set of state variables to be optimized includes: a) The carrier state at N keyframe moments, where carrier state refers to the state of the motion measurement system. For example... Figure 3 circular nodes As shown, Used as a carrier state, it represents the carrier in the first... The state at each keyframe moment, each node represents the carrier state at one keyframe moment. Understandably, the carrier state here... Includes the pose of the carrier at that moment (i.e., the pose of the carrier coordinate system B in the world coordinate system W) ) and speed ,Right now .

[0115] b) Spatiotemporal parameters, such as Figure 3 As shown in the diamond-shaped node, this parameter contains LiDAR extrinsic parameters. LiDAR time offset Camera external parameters and camera time offset , denoted as spatiotemporal parameter .

[0116] The objective function for model optimization is constructed to minimize the sum of Mahalanobis distances for all constraint terms. These constraint terms establish the target feature pairs. The measurement data from other sensors are converted into mathematical constraints on the state variables, specifically implemented as follows: 1) Kinematic constraints (IMU factors): Introducing IMU factors, where one IMU factor connects two adjacent state nodes, such as... and This indicates that the IMU measurement constrains the relative motion at consecutive moments.

[0117] 2) Cross-modal geometric constraints (LiDAR-camera geometric factors): This is the core constraint of this invention, which directly applies the target feature pairs verified in step S2. For each target LiDAR edge feature and target image edge features The target feature pairs are used to construct a point-to-line distance residual. Specifically: a) From the target LiDAR edge features Several sampling points are selected above. For any one of these sampling points, its coordinates in the LiDAR coordinate system are: Using the current state variables to be optimized, including the carrier state and spatiotemporal parameters Following the coordinate transformation process described in step S2, the coordinates are projected onto the image plane to obtain the projection points. .

[0118] b) Edge features of the target image In the image, it can be represented by the equation of a straight line. : .

[0119] c) This sampling point geometric residuals That is, the projection point to the straight line The orthogonal distance.

[0120] A single target feature pair can generate multiple residual terms, meaning there is one residual term for each sampling point. These residual terms collectively constitute the LiDAR-camera geometric constraint, which correlates a geometric observation (feature pair) with the carrier pose, the extrinsic parameters of the two sensors, and the time offset. When the parameters are inaccurate, the projection point will deviate from the image line, increasing the residual; the optimization process involves adjusting all parameters to bring the residual close to zero.

[0121] 3) Absolute positioning constraints: Introduce GNSS factors, where a GNSS factor is determined from the state of a certain carrier. The node is drawn out to indicate that the absolute position of the carrier at that moment is constrained by GNSS observations.

[0122] Based on the above constraints, the objective function of the joint optimization model can be expressed as:

[0123] In the formula: It represents the set of all state variables to be optimized; , , These are the IMU pre-integration residuals, GNSS position residuals, and LiDAR-camera point-line geometric residuals, respectively. Represents the square of the Mahalanobis distance. This is the covariance matrix corresponding to the measurement noise.

[0124] By solving this nonlinear least squares problem, the vehicle trajectory and all spatiotemporal parameters can be optimized simultaneously.

[0125] The parameter management and update module 103 is used to perform online optimization and parameter update of the joint optimization model based on a sliding window.

[0126] Optimization Solution: In a background thread, the joint optimization module 102 uses nonlinear optimization algorithms such as the Gauss-Newton method or the Levenberg-Marquardt method to optimize the constructed factor graph, obtaining the increment of the state variable to be optimized. This increment is the correction amount of the spatiotemporal parameter. After optimization, the joint optimization module 102 sends the "optimized parameter increment" to the parameter management and update module 103.

[0127] Parameter smoothing update: After receiving the correction amount, the parameter management and update module 103 updates the spatiotemporal parameters using a preset smoothing filtering mechanism. The parameter management and update module 103 provides the updated spatiotemporal parameters to the feature extraction and association module 101 and the joint optimization module 102 via a feedback loop for the next round of calculation. Simultaneously, it can also output high-precision spatiotemporal parameters to external applications.

[0128] Furthermore, the processing unit 100 may also include a quality monitoring module 104, and the joint optimization module 102 sends the optimized "residual information" to the quality monitoring module 104. If the residual continues to increase, it indicates that the calibration quality has decreased, and the quality monitoring module 104 may send an "optimization strategy adjustment" instruction to the joint optimization module 102, such as increasing the number of iterations.

[0129] By cyclically executing the above operations, the method and system provided by this invention can continuously and automatically calibrate and correct the spatiotemporal parameters between multiple sensors during the operation of a mobile measurement system, thereby ensuring the high accuracy and long-term stability of multi-sensor fusion data.

[0130] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for online self-calibration of multi-source spatiotemporal parameters of a mobile measurement system, characterized in that, The method includes: S1. Acquire multi-source measurement data from the mobile measurement system, wherein the multi-source measurement data includes collected 3D point cloud data, 2D images, and the status data and spatiotemporal data of the mobile measurement system; S2. The three-dimensional point cloud data is corrected, the first geometric feature of the corrected three-dimensional point cloud data is extracted, and the second geometric feature of the two-dimensional image is extracted. The first geometric feature of the three-dimensional point cloud data is projected onto the two-dimensional image plane, and a mapping relationship between the first geometric feature of the three-dimensional point cloud data and the second geometric feature of the two-dimensional image is established based on preset verification conditions to form a target feature pair. S3. Construct a joint optimization model with a unified spatiotemporal reference, and use the target feature pairs as feature constraints to optimize the state variables to be optimized. The state variables to be optimized include the carrier state at key frame time and spatiotemporal parameters. S4. Perform online optimization and parameter update of the joint optimization model based on a sliding window.

2. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 1, characterized in that, In step S2, the extraction of the first geometric feature of the corrected 3D point cloud data specifically includes: Calculate the local curvature value of each three-dimensional point in the corrected three-dimensional point cloud data; Compare the local curvature value with the preset curvature value. If: If the local curvature value is greater than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as an edge point; If the local curvature value is less than the preset curvature value, then the three-dimensional point corresponding to the local curvature value is classified as a planar point; The edge points and the planar points are clustered respectively to form LiDAR edge features as the first geometric feature. and LiDAR planar features ; And, extracting the second geometric features of the two-dimensional image, specifically including: A preset line segment detection algorithm is used to extract line segment features from the two-dimensional image and represent them as image edge features. .

3. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 2, characterized in that, In step S2, projecting the first geometric feature of the three-dimensional point cloud data onto the two-dimensional image plane specifically includes: LiDAR edge features for 3D point cloud data and LiDAR planar features Based on the estimated values ​​of LiDAR time offset and camera time offset in the current spatiotemporal parameters, the LiDAR edge features are calculated. and LiDAR planar features The actual acquisition time is used to determine the corresponding two-dimensional image from all two-dimensional images as the target image. The LiDAR edge features are calculated by interpolating the state data from the mobile measurement system. and LiDAR planar features The carrier pose at each point at the time of data acquisition; Based on the carrier pose of each point at the time of acquisition, it is transformed from the LiDAR coordinate system to the world coordinate system and the camera coordinate system in sequence, and finally projected into image pixel coordinates, which respectively form a projection dashed line and a two-dimensional projection area.

4. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 3, characterized in that, In step S2, interpolation is performed using the state data of the mobile measurement system to calculate the LiDAR edge features. The carrier pose of each edge point at the time of acquisition, specifically including: All high-frequency IMU measurements of the LiDAR scan frame between the start and end times are acquired and pre-integrated by the IMU to obtain the pre-integrated value of the LiDAR scan frame; wherein, the high-frequency IMU measurement value is defined as the measurement value corresponding to the inertial measurement unit IMU sampling frequency being higher than the LiDAR scan frequency; The LiDAR edge features from the start time to the start time are calculated using the IMU pre-integral interpolation. The relative pose transformation of the carrier at each edge point acquisition time; Multiplying the relative pose transformation of the carrier with the carrier pose at the start of the LiDAR scan frame yields the carrier pose of each edge point at the time of acquisition.

5. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 3, characterized in that, In step S2, based on the carrier pose of the edge points at the time of acquisition, the pose is sequentially transformed from the LiDAR coordinate system to the world coordinate system and the camera coordinate system, and finally projected into image pixel coordinates. Specifically, this includes: Let the coordinates of the edge point in the LiDAR coordinate system be... ; The edge points are transformed from the LiDAR coordinate system to the carrier coordinate system using LiDAR extrinsic parameters to obtain the carrier coordinate system coordinates. : , Indicates LiDAR external parameters; Using the carrier pose of the edge points at the time of acquisition, the carrier coordinate system coordinates are... Transform to the world coordinate system to obtain world coordinate system coordinates. : , This represents the carrier pose of the edge points at the time of data acquisition. Using the camera's pose in the world coordinate system, the world coordinate system coordinates are... Transform to the camera coordinate system to obtain the camera coordinate system coordinates. : , This represents the camera's pose in the world coordinate system, and , Indicates camera external parameters; Using camera intrinsic matrix The camera coordinate system coordinates are determined using a pinhole camera model. Projection to image pixel coordinates , This indicates transpose.

6. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 3, characterized in that, In step S2, the preset verification conditions include geometric structure consistency verification and photometric information consistency verification; wherein, the geometric structure consistency verification includes: Directional consistency: Calculate the angle between the direction vector of the projected dashed line and the direction vector of the candidate image edge features. If the angle is less than a first preset angle threshold, directional consistency is satisfied. The candidate image edge features are defined as: image edge features falling within a strip-shaped region centered on the projected dashed line and with a preset pixel width. These are considered as candidate image edge features; Position overlap: Calculate the distance between the center point of the projected dashed line and the edge features of the candidate image. If the distance is less than the first preset pixel threshold, the position overlap is satisfied. The consistency verification of the photometric information includes: Obtain the LiDAR edge features Edge points with reflectance greater than a preset intensity threshold are identified, and their average grayscale gradient within the corresponding projection area of ​​the target image is calculated. If the average gray-level gradient corresponding to the edge point is greater than the preset gray-level gradient threshold, then the consistency of photometric information is satisfied.

7. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 6, characterized in that, In step S2, a mapping relationship is established between the first geometric feature of the three-dimensional point cloud data and the second geometric feature of the two-dimensional image based on preset verification conditions to form a target feature pair, specifically including: Candidate image edge features that pass the geometric structure consistency verification are used as target image edge features. ; The LiDAR edge features formed by the edge points that have passed the consistency verification of the photometric information. As the edge feature of the target LiDAR ; The target LiDAR edge features and target image edge features Constructing target feature pairs .

8. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 7, characterized in that, In S3, the feature constraints include: Kinematic constraints: Introducing IMU factors, where an IMU factor connects two adjacent state nodes, indicating that the IMU measurement constrains the relative motion at consecutive time points; Cross-modal geometric constraints: Introducing LiDAR-camera geometric factors, which apply the target features to... For each target LiDAR edge feature and target image edge features The target feature pairs are used to construct a point-to-line distance residual as a geometric constraint for the LiDAR camera. Absolute positioning constraints: Introducing GNSS factors, where a GNSS factor is derived from the state of a certain carrier. The node is drawn out to indicate that the absolute position of the carrier at that moment is constrained by GNSS observations; The construction of the LiDAR-camera geometric constraints specifically includes: From the target LiDAR edge features Select several edge points. For any one of these edge points, its coordinates in the LiDAR coordinate system are: The current state variable to be optimized is projected onto the image plane to obtain the projection point. ; Edge features of the target image Represented as a straight line in the image ; Calculate edge points geometric residuals That is, the projection point to the straight line orthogonal distance; The residual terms of all edge points contained in the target feature pair are used to construct a LiDAR-camera geometric constraint. And / or, the spatiotemporal parameters of the state variable to be optimized include LiDAR extrinsic parameters. LiDAR time offset Camera external parameters and camera time offset .

9. The online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system according to claim 8, characterized in that, In S3, the objective function of the joint optimization model is expressed as: In the formula: It represents the set of all state variables to be optimized; , , These are the IMU pre-integration residuals, GNSS position residuals, and LiDAR-camera point-line geometric residuals, respectively. Represents the square of the Mahalanobis distance. This is the covariance matrix corresponding to the measurement noise.

10. A system for implementing the online self-calibration method for multi-source spatiotemporal parameters of a mobile measurement system as described in any one of claims 1 to 9, characterized in that, The system includes at least: The data acquisition module is used to acquire multi-source measurement data from the mobile measurement system. The multi-source measurement data includes collected three-dimensional point cloud data, two-dimensional images, and the status data and spatiotemporal data of the mobile measurement system. The feature extraction and association module is used to perform correction processing on the three-dimensional point cloud data, extract the first geometric features of the corrected three-dimensional point cloud data, and extract the second geometric features of the two-dimensional image, project the first geometric features of the three-dimensional point cloud data onto the two-dimensional image plane, and establish a mapping relationship between the first geometric features of the three-dimensional point cloud data and the second geometric features of the two-dimensional image based on preset verification conditions to form a target feature pair; The joint optimization module is used to construct a joint optimization model with a unified spatiotemporal reference. The target feature pair is used as a feature constraint to optimize the state variables to be optimized. The state variables to be optimized include the carrier state at key frame time and spatiotemporal parameters. The parameter management and update module is used to perform online optimization and parameter updates of the joint optimization model based on a sliding window.

Images