A kind of on-board multi-sensor space-time online calibration method under complex environment of fully mechanized coal face

Abstract

The application discloses an on-board multi-sensor space-time online calibration method in a fully mechanized coal mining face complex environment and belongs to the technical field of image processing and environment perception. With the main IMU time axis as a unified reference, the time offset of the camera and the laser radar inertial measurement is estimated and compensated online, and a continuous time pose prediction model is established. The point cloud is subjected to motion compensation and time unification, combined with the time sequence motion and echo intensity consistency discrimination to remove the jitter ghost points, and the de-ghosting point cloud is obtained. Combined with the camera continuous time pose and the rolling shutter exposure model of each row, a spatially changing blur kernel is constructed, the non-blind deconvolution of the image is carried out to obtain a clear image. Finally, based on the de-ghosting point cloud and the clear image, a cross-modal structure consistency constraint is constructed, and the camera-laser radar space external participation time offset is solved through nonlinear optimization. The application can realize high-precision, lightweight and sustainable online space-time calibration, and improve the stability and reliability of the multi-modal data fusion of the on-board equipment in the fully mechanized coal mining face.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces. Background Technology

[0002] Fully mechanized mining face airborne multi-sensor systems typically carry multiple sensors to achieve environmental information perception. Due to their complementary performance, lidar and cameras have become the mainstream configuration for multimodal data fusion. LiDAR can acquire target spatial geometry and pose information over a wide range, but its semantic expression capability is limited; cameras can acquire images rich in semantic information, but cannot retain target spatial pose information. The fusion of these two technologies can compensate for their respective shortcomings and is a crucial technological foundation for fully mechanized mining face airborne equipment to complete environmental perception, fusion positioning, and online collaborative operations.

[0003] Reliable data fusion between lidar and cameras requires precise spatial extrinsic parameter and temporal offset calibration, i.e., clearly defining the spatial pose transformation relationship between sensors and ensuring time synchronization. Traditional calibration methods mostly rely on static environmental features or dedicated calibration targets, and are generally carried out offline when the equipment is stationary or operating at low speeds. However, the working environment of a fully mechanized mining face is complex. The airborne platform is prone to vibration and bumps during operation. In addition, problems such as dust scattering, obstruction, low light, and highly repetitive scene structures can cause image and point cloud quality degradation, leading to motion blur in rolling shutter images, distortion and ghosting of lidar point clouds, making it difficult to extract effective features stably. At the same time, interference from dynamic targets such as pedestrians, vehicles, and equipment in the scene can further aggravate the extrinsic parameter estimation error, making it difficult for traditional calibration methods to be stably applied in actual operations for a long period of time. Some solutions mitigate the effects of motion through mechanical stabilizers or external image stabilization devices. However, these devices are bulky, heavy, and cumbersome to install and maintain. Furthermore, they cannot address issues such as sensor time offset drift due to operating conditions and clock asynchrony among multiple IMUs, making it difficult to meet the lightweight and continuous calibration requirements of fully mechanized mining face airborne platforms during online operations. Therefore, there is an urgent need for a calibration method that uses the main IMU as a unified reference, can perform online time alignment of multiple IMUs and provide continuous time motion priors, and can suppress point cloud ghosting and image blurring while enabling online calibration of time offsets from outside the spatial domain.

[0004] Existing self-calibration techniques rely heavily on stable geometric or texture features in the scene. Under complex working conditions at the fully mechanized mining face, they are susceptible to decreased observation reliability due to sparse features, repetitive structures, occlusion, and dust interference. At the same time, they are not capable of jointly suppressing dynamic target interference, point cloud trailing, and spatial distortion blurring of rolling shutter. In dynamic operation scenarios, calibration errors are significant, making it difficult to obtain high-precision and continuously updated spatiotemporal calibration results. Summary of the Invention

[0005] The purpose of this invention is to propose a spatiotemporal online calibration method for airborne multi-sensors in complex environments of fully mechanized mining faces, overcoming the problems of existing calibration methods that rely on targets, operate offline, and have poor resistance to vibration and fuzziness. By unifying spatiotemporal references and motion compensation, point cloud ghosting and image deblurring are achieved, improving the calibration accuracy and robustness of camera-lidar extrinsic parameters and time offsets, and meeting the long-term stable, lightweight, and high-precision multimodal fusion sensing requirements of airborne equipment in complex dynamic environments of fully mechanized mining faces.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: an airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces, applied to a system including an airborne platform for fully mechanized mining faces, a main IMU, a camera module, a lidar module, and an information processing module. The main IMU, camera module, and lidar module are installed on the airborne platform for fully mechanized mining faces and connected to the information processing module. The camera module includes a camera and a corresponding Camera-IMU, and the lidar module includes a lidar and a corresponding LiDAR-IMU. The method includes the following steps:

[0007] Step S10: Real-time acquisition of main IMU inertial data, camera module images and corresponding inertial data, and lidar module point cloud and corresponding inertial data;

[0008] Step S20: Using the main IMU time axis as a reference, estimate and compensate for the time offset of the corresponding inertial measurement units of the camera module and the lidar module online, and establish a continuous time pose prediction model; the continuous time pose prediction model includes a lidar continuous time pose prediction model and a camera continuous time pose prediction model.

[0009] Step S30: Based on the continuous time pose prediction model of lidar, perform motion compensation and time unification on the point cloud, identify and remove jittery trailing points, and obtain a trailing point cloud.

[0010] Step S40: Combine the camera's continuous time pose prediction model with the rolling shutter line-by-line exposure model to construct a spatial variation blur kernel and perform non-blind deconvolution on the image to obtain a clear image;

[0011] Step S50: Construct cross-modal structural consistency constraints based on the de-ghosting point cloud and the clear image, and solve the camera-LiDAR spatial extra-space participation time offset through nonlinear optimization to output the calibration results.

[0012] As a further improvement of the present invention, in step S20, the online estimation and compensation of the time offset between the camera and the lidar inertial measurement, based on the main IMU time axis, to establish a continuous-time pose prediction model, specifically includes the following steps:

[0013] Step S201: Using the local time axis of the main IMU as a unified time reference, the information processing module performs online alignment of the inertial measurement timestamps output by the LiDAR-IMU, and estimates the time offset of the LiDAR-IMU relative to the main IMU using the angular velocity sequence or angular velocity magnitude sequence within the sliding time window as the alignment object. After the alignment is completed, a time mapping relationship is established.

[0014] ;

[0015] In the formula, and Main IMU timestamp, For LiDAR-IMU local timestamps, The time offset for aligning the LiDAR-IMU timestamp to the main IMU time axis;

[0016] Step S202: Using the local time axis of the main IMU as a unified time reference, the information processing module performs online alignment of the inertial measurement timestamps output by the Camera-IMU, and estimates the time offset of the Camera-IMU relative to the main IMU using the angular velocity sequence or angular velocity magnitude sequence within the sliding time window as the alignment object. After the alignment is completed, a time mapping relationship is established.

[0017] ;

[0018] In the formula, For Camera-IMU local timestamps, The time offset for aligning the Camera-IMU timestamp to the main IMU time axis;

[0019] Step S203: The information processing module establishes the continuous-time pose trajectory of the carrier on the main IMU time axis, and uses Lie groups. The B-spline control points are parameterized; the continuous-time pose trajectory control points and LiDAR-IMU time offsets are jointly optimized within a sliding time window. Camera-IMU time offset The continuous-time pose trajectory is updated online with IMU bias parameters to serve as a common motion reference during lidar scanning and camera exposure.

[0020] Step S204: Based on the inertial measurements of the main IMU and the rigid mounting relationship between the LiDAR-IMU and Camera-IMU relative to the main IMU, the information processing module establishes multi-source inertial consistency constraints under a unified time axis.

[0021] ;

[0022] In the formula, Indicates the type of auxiliary inertial measurement unit. LiDAR-IMU, Indicates Camera-IMU; and These are the main IMU debiasing angular velocity and acceleration measurements, respectively. and The first The bias debias angular velocity and acceleration of the auxiliary IMU were measured. For the first Fixed rotational extrinsic parameters from auxiliary IMU to main IMU; For the first The time offset of each auxiliary IMU to the time axis of the main IMU;

[0023] Step S205: After obtaining the continuous-time pose trajectory of the carrier, the information processing module establishes a continuous-time pose prediction model for the lidar and a continuous-time pose prediction model for the camera under a unified time axis, based on the rigid mounting relationship between the lidar and the camera relative to the main IMU and their current external parameter estimates. This serves as the motion prior for point cloud de-ghosting and image de-blurring.

[0024] ;

[0025] ;

[0026] In the formula, For a moment Pose transformation matrix from the main IMU to the world coordinate system; External parameters of a fixed rigid body for laser radar to the main IMU; External parameters of the fixed rigid body from the camera to the main IMU; For a moment The pose transformation matrix of lidar to the world coordinate system; For a moment The pose transformation matrix from the camera to the world coordinate system.

[0027] As a further improvement of the present invention, in step S30, the motion compensation and time unification of the point cloud based on the continuous time pose prediction model of lidar, the identification and removal of jittery trailing points, and the obtaining of the trailing point cloud specifically include the following steps:

[0028] Step S301: The information processing module performs temporal unfolding on the single-frame point cloud output by the lidar module and reads the relative sampling time of each point. and the start timestamp of the corresponding frame The sampling time of the point is aligned to the time axis of the main IMU using the time offset output in step S20, thus obtaining the global sampling time of the point:

[0029] ;

[0030] In the formula, It is the time offset between the lidar timestamp and the main IMU time axis;

[0031] Step S302: The information processing module queries the sampling time on the continuous-time pose prediction model of the lidar obtained in step S20. Reference time The lidar pose is obtained, the relative motion increment from the sampling time to the reference time is calculated, and the Lie algebra representation of the relative motion increment is expressed. As a measure of jitter intensity during scanning:

[0032] ;

[0033] In the formula, It is a moment Pose transformation matrix from lidar coordinate system to world coordinate system; ;

[0034] Step S303: The information processing module unifies the points from their sampling time to the lidar coordinate system at the reference time based on the relative pose increment, in order to eliminate the geometric stretching and ghosting caused by motion during scanning, and obtain the motion-compensated points:

[0035] ;

[0036] In the formula, It is the first The coordinates of a point in the lidar coordinate system; These are the coordinates after compensation to the reference time. From the sampling time to reference time The relative pose transformation matrix;

[0037] Step S304: The information processing module combines the temporal motion consistency, neighborhood geometric consistency, and laser echo intensity consistency of the point cloud itself to distinguish between real points and jittery trailing points, and constructs a trailing discrimination score.

[0038] Step S305: Establish robust weights for points based on the motion blur discrimination score and perform removal processing; the robust weight is defined as:

[0039] ;

[0040] And based on the threshold The trailing indicator variable of the construction point:

[0041] ;

[0042] In the formula, It is the first Robust weights at each point, It is a motion blur indicator variable. It is the motion blur detection threshold. This indicates jittery or ghosting points that have been removed. Indicates the actual location;

[0043] Step S306: Take the point cloud after removing the trailing points as the output of the de-trailing point cloud, and use it as the geometric constraint input for subsequent camera-LiDAR calibration.

[0044] As a further improvement of the present invention, in step S304, the information processing module further combines the temporal motion consistency, neighborhood geometric consistency, and laser echo intensity consistency of the point cloud itself to distinguish between real points and jittery trailing points, and constructs a trailing discrimination score, specifically as follows:

[0045] Construct a nearest neighbor set within the temporal and spatial neighborhoods of the compensation point. Fit a local plane on the nearest neighbor set and calculate the distance from the point to the plane. Calculate the coordinate consistency metric between the compensation point and its nearest neighbors. Calculate the laser echo intensity consistency metric The temporal motion consistency, neighborhood geometric consistency, laser echo intensity consistency, and relative motion intensity are used together for motion blur discrimination to construct a motion blur discrimination score:

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] In the formula, It is the unit normal vector of the local plane fitted by the nearest neighbor set. It is the bias term of the local plane. It is the first Laser echo intensity values ​​at each point; It is the motion blur discrimination score; , , and It is the weighting coefficient.

[0051] As a further improvement of the present invention, in step S305, the trailing detection threshold is... The settings are based on the lidar ranging noise level, the statistical distribution of relative motion intensity, or a preset signal interval, in order to balance the thoroughness of ghosting removal with the integrity of real-point retention.

[0052] As a further improvement of the present invention, in step S40, the step of combining the camera continuous-time pose prediction model and the rolling shutter line-by-line exposure model to construct a spatially varying blur kernel and non-blindly deconvolve the image to obtain a clear image specifically includes the following steps:

[0053] Step S401: The information processing module establishes a line-by-line exposure time model for the RGB rolling shutter camera, and uses the camera time offset output in step S20 to align the line-by-line exposure times to the time axis of the main IMU; for the... A frame image, let its first line exposure start timestamp be... The total number of rows in the image is The line readout time is Then the first The exposure time of a line is represented as:

[0054] ;

[0055] Then align the camera timestamps to the main IMU timeline to obtain:

[0056] ;

[0057] In the formula, It is the time offset between the camera timestamp and the main IMU timeline;

[0058] Step S402: Based on the camera continuous-time pose prediction model obtained in step S20, the information processing module queries the camera pose at any time during the exposure period and calculates the camera's pose relative to the reference time. Continuous time relative rotation The relative rotation is used as a motion prior for deblurring the rolling shutter motion;

[0059] Step S403: In the rolling shutter model, the information processing module constructs pixel trajectories using relative rotation during exposure; and processes homogeneous pixel coordinates on the reference time image. Construct a homography transformation induced by relative rotation:

[0060] ;

[0061] And obtain the motion trajectory of the pixel over time:

[0062] ;

[0063] In the formula, It is the relative rotation matrix of the camera relative to a reference time during the exposure period. It is the camera intrinsic parameter matrix. It is a perspective projection function;

[0064] Step S404: The information processing module processes the pixel trajectory within the exposure integration time. Discrete sampling is performed, and the pixel displacement sequence obtained from the discrete sampling is used to construct a spatial variation motion blur kernel or convolution kernel related to the pixel position. Based on the blur kernel, the blurred image is represented as a convolution superposition model of the clear image and the spatially varying motion blur kernel. Under the condition that the blur kernel is known, non-blind deconvolution with regularization constraints is used to achieve image deblurring. The deblurring optimization objective is denoted as:

[0065] ;

[0066] In the formula, It is a blurry image. It is a clear image to be recovered. It is a convolution operator. It is the total variation regularization term. It is the regularization weight;

[0067] Step S405: Perform robust post-processing on the deblurring result to suppress noise amplification and deconvolution ringing artifacts, and perform boundary consistency processing on the image boundary and occluded area to reduce boundary convolution error. Output a clear image after motion compensation and deblurring processing, and use it as the visual constraint input for camera-LiDAR calibration in subsequent steps.

[0068] As a further improvement of the present invention, step S401 further includes the following step: the information processing module uses the line-by-line exposure time as the time index for rolling shutter deblurring, so that the motion compensation of each row of pixels is consistent with the exposure time.

[0069] As a further improvement of the present invention, in step S403, the construction process of the pixel trajectory does not depend on the spatial extrinsic parameters between the lidar and the camera, and does not depend on the lidar point cloud to provide depth for the pixel, thereby avoiding the implicit introduction of calibration results or external depth information into the image deblurring process.

[0070] As a further improvement of the present invention, in step S50, the construction of cross-modal structural consistency constraints based on the de-ghosting point cloud and the clear image, and the joint solution of the camera-LiDAR out-of-space participation time offset through nonlinear optimization to output the calibration result, specifically includes the following steps:

[0071] Step S501: Using the camera-LiDAR calibration results as output, construct a set of variables to be estimated within a sliding window, and introduce the B-spline control points of the main IMU's continuous time trajectory and the IMU bias as intermediate variables to provide pose query results for the point cloud and image on a unified time axis. Define the state to be optimized as:

[0072] ;

[0073] In the formula, It is the set of states to be optimized within the sliding window. and These are the rigid external parameters of the camera and lidar relative to the main IMU, respectively. and These represent the time offsets for aligning the camera and LiDAR timestamps to the main IMU timeline. It is a continuous time pose trajectory The set of B-spline control points, It is the set of IMU bias parameters within the sliding window;

[0074] The information processing module uses historical calibration values ​​as... and The initial value is used, and the output of the time-aligned submodule is used as the initial value. and The initial value is determined to ensure that the online optimization has convergent initial conditions;

[0075] Step S502, by and Calculate the spatial extrinsic parameters between the lidar and the camera, and by and Calculate the time offset between the two, and define the output as:

[0076] ;

[0077] ;

[0078] In the formula, It is the external parameter of the rigid body from the lidar coordinate system to the camera coordinate system. It is a time offset that maps the lidar timestamp to the camera timeline;

[0079] Step S503: Within the sliding window, jointly minimize the IMU kinematic residuals, lidar geometric residuals, and cross-modal structural residuals to establish the overall optimization objective:

[0080]

[0081] In the formula, It is the IMU kinematic residual. It is the IMU residual covariance matrix. and These are the scale parameters of the lidar geometric residuals and the cross-modal structural residuals, respectively. It is a robust kernel function;

[0082] The lidar geometric residuals include the residuals from the midpoint of the de-ghosting point cloud to the local plane, from the point to the edge line, or the registration residuals of the point clouds in adjacent frames.

[0083] The IMU kinematic residuals include the pre-integration / kinetic consistency residuals between the continuous-time B-spline trajectory derivative and the main IMU inertial measurement, as well as the multi-source inertial consistency residuals between the auxiliary IMU and the main IMU.

[0084] The cross-modal structural residuals include negative values ​​of mutual information calculated from the joint histogram of the lidar projection view and the sharp image, or include alignment residuals constructed based on the mutual information.

[0085] Step S504: Solve the overall optimization objective using a nonlinear least squares iterative algorithm to obtain the converged result. and The calibration results are used as the calibration results for camera-LiDAR, and these calibration results are then used for multimodal fusion of point cloud and image.

[0086] As a further improvement to the present invention, the information processing module is iteratively optimized according to the following process:

[0087] Initialize the estimated state with the initial values ​​of the external parameters and the initial value of the time offset from step S501;

[0088] In each iteration, the IMU kinematic residual, the lidar geometric residual, and the cross-modal structural residual are calculated sequentially, and an optimization objective is constructed.

[0089] The estimated state is incrementally updated and then substituted back to the external parameters and time offset;

[0090] The iteration stops when both the external parameter increment and the time offset increment are less than the preset threshold or the objective function decrease is less than the preset threshold.

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] 1. This invention uses the master IMU as a unified reference to achieve online alignment of multiple IMU timestamps and establishes a continuous time pose prediction model that can be queried at any time. This enables point cloud motion compensation and rolling shutter imaging motion estimation to have consistent motion priors, and can effectively overcome the impact of underground vehicle vibration, bumps and time asynchrony on the convergence stability of online spatiotemporal calibration.

[0093] 2. This invention achieves unified reference time for point clouds based on continuous-time motion priors, and combines relative motion increment information with the temporal motion consistency, neighborhood geometric consistency, and echo intensity consistency of the point cloud to identify and remove trailing points, resulting in a deblurred point cloud without trailing. Simultaneously, a spatially varying motion blur kernel is constructed based on the line-by-line exposure characteristics of the rolling shutter, and non-blind deconvolution is performed to obtain a clear image. The image deblurring process does not rely on the spatial extrinsic parameters between the LiDAR and the camera, nor does it rely on the point cloud to provide depth for pixels, thereby reducing the interference of point cloud distortion and image blur on cross-modal structural constraints.

[0094] 3. This invention employs a joint optimization of cross-modal structural consistency constraints and IMU kinematic constraints to achieve an integrated online solution for spatially involved time migration. It is less dependent on environmental features and additional calibration targets under downhole conditions such as weak texture, illumination changes, dust scattering, occlusion, and structural repetition. Furthermore, it does not require a bulky and heavy external image stabilization device, which facilitates equipment integration and long-term online operation, and can obtain more accurate and robust spatiotemporal calibration results. Attached Figure Description

[0095] Figure 1 This is a flowchart summarizing the spatiotemporal online calibration method for airborne multi-sensor sensors in complex environments of fully mechanized mining faces, as described in this invention.

[0096] Figure 2 This is a detailed flowchart of the airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces, as described in this invention.

[0097] Figure 3 This is a schematic diagram of sensor time alignment in an embodiment of the present invention.

[0098] Figure 4 This is a comparison image of point cloud before and after deblurring in an embodiment of the present invention.

[0099] Figure 5 This is a comparison image before and after deblurring in an embodiment of the present invention.

[0100] Figure 6 This is a schematic diagram of the airborne platform structure for the fully mechanized mining face of the present invention.

[0101] Explanation of reference numerals in the attached diagram: 1-Airborne platform for fully mechanized mining face, 2-Main IMU, 3-Camera, 4-LiDAR, 5-Information processing module. Detailed Implementation

[0102] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0103] This invention discloses an airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining areas. The inventive concept is as follows: This invention proposes a motion compensation deblurring approach based on multi-IMU time alignment and continuous-time motion prediction. Using the local time axis of the main IMU as a unified time reference, the time offset of the camera and corresponding IMUs of the lidar relative to the main IMU is estimated and compensated online, establishing a continuous-time pose prediction model that can be queried at any time. Subsequently, the continuous-time pose of the lidar is used to unify the reference time of the point cloud, and jittery points are discriminated against and weighted or eliminated by combining temporal motion consistency and neighborhood geometric consistency to obtain a deblurred point cloud. A rolling shutter line-by-line exposure time model and the continuous-time pose of the camera are used to construct pixel motion trajectories, and a spatially varying motion blur kernel is established for non-blind deconvolution to obtain a clear image. Furthermore, the deblurred point cloud and the clear image are used to construct cross-modal structural consistency constraints, and the spatial extrinsic parameters and temporal offset between the lidar and the camera are jointly solved through nonlinear optimization.

[0104] like Figure 6 As shown, the present invention is based on the following multi-sensor system, including a fully mechanized mining face airborne platform 1, a main IMU 2, a camera 3, a lidar 4, and an information processing module 5. The information processing module 5 is integrated in the fully mechanized mining face airborne platform 1, and the main IMU 2, camera 3, and lidar 4 are located on the fully mechanized mining face airborne platform 1 and connected to the information processing module 5.

[0105] Please refer to Figure 1 and Figure 2 The calibration method of the present invention includes the following steps:

[0106] In step S10, during the operation of the fully mechanized mining face airborne platform 1, the information processing module 5 receives in real time the high-frequency IMU data from the main IMU2, the rolling shutter RGB image frames and their exposure time information and Camera-IMU data from the camera module 3, and the point cloud data, timestamps and LiDAR-IMU data from the lidar module 4.

[0107] In step S20, the information processing module 5 uses the local time axis of the main IMU2 as a unified reference to estimate and compensate for the time offset of the Camera-IMU and LiDAR-IMU relative to the main IMU2 online, thereby achieving multi-IMU timestamp alignment. Under the unified time axis, based on high-frequency inertial observation, a continuous time pose prediction model for the lidar and a continuous time pose prediction model for the camera are established respectively, which can be queried at any time, as motion priors for point cloud de-ghosting and rolling shutter image de-blurring.

[0108] Step S201: Using the local time axis of the main IMU2 as a unified time reference, the information processing module 5 performs online alignment of the inertial measurement timestamps output by the LiDAR-IMU, and estimates the time offset of the LiDAR-IMU relative to the main IMU2 using the angular velocity sequence or angular velocity magnitude sequence within the sliding time window as the alignment object; after alignment, a time mapping relationship is established:

[0109] ;

[0110] In the formula, and Main IMU timestamp, For LiDAR-IMU local timestamps, The time offset for aligning the LiDAR-IMU timestamp to the main IMU time axis.

[0111] Step S202: Using the local time axis of the main IMU2 as a unified time reference, the information processing module 5 performs online alignment of the inertial measurement timestamps output by the Camera-IMU, and estimates the time offset of the Camera-IMU relative to the main IMU2 using the angular velocity sequence or angular velocity magnitude sequence within the sliding time window as the alignment object; after alignment, a time mapping relationship is established:

[0112] ;

[0113] In the formula, For Camera-IMU local timestamps, Align the Camera-IMU timestamp with the time offset of the main IMU time axis.

[0114] Step S203: Information processing module 5 establishes the continuous-time pose trajectory of the carrier on the main IMU time axis, and uses Lie groups. The B-spline control points are parameterized; within a sliding time window, the continuous-time pose trajectory control points, LiDAR-IMU time offset, Camera-IMU time offset, and IMU bias parameters are jointly optimized, and the continuous-time pose trajectory is updated online as a common motion reference during LiDAR scanning and camera exposure.

[0115] Step S204: Based on the inertial measurements of the main IMU2 and the rigid mounting relationship between the LiDAR-IMU and Camera-IMU relative to the main IMU2, the information processing module 5 establishes multi-source inertial consistency constraints under a unified time axis.

[0116] ;

[0117] In the formula, Indicates the type of auxiliary inertial measurement unit. LiDAR-IMU, Indicates Camera-IMU; and These are the main IMU debiasing angular velocity and acceleration measurements, respectively. and The first The bias debias angular velocity and acceleration of the auxiliary IMU were measured. For the first Fixed rotational extrinsic parameters from auxiliary IMU to main IMU; For the first Each auxiliary IMU is aligned to the time offset of the main IMU's time axis.

[0118] Step S205: After obtaining the continuous-time pose trajectory of the carrier, the information processing module 5 establishes a continuous-time pose prediction model for the lidar and a continuous-time pose prediction model for the camera under a unified time axis, based on the fixed installation extrinsic parameters of the lidar and camera relative to the main IMU2. These models serve as motion priors for point cloud de-ghosting and image de-blurring.

[0119] ;

[0120] ;

[0121] In the formula, For a moment Pose transformation matrix from the main IMU to the world coordinate system; External parameters of a fixed rigid body for laser radar to the main IMU; External parameters of the fixed rigid body from the camera to the main IMU; For a moment The pose transformation matrix of lidar to the world coordinate system; For a moment The pose transformation matrix from the camera to the world coordinate system.

[0122] like Figure 3The diagram illustrates the multi-sensor data time synchronization and unified time axis establishment of the present invention. The main IMU time axis is used as the unified time reference, and the diagram shows the data arrival relationship between the lidar, camera, and main IMU at different sampling frequencies. The upper part is the lidar data sampling sequence, the lower part is the camera image sampling sequence, and the middle part is the main IMU high-frequency inertial sampling sequence. The dashed lines are used to mark the alignment position of each sensor data on the unified time axis, thereby demonstrating the process of mapping the lidar timestamp and camera timestamp to the main IMU time axis to complete the unified time marking.

[0123] In step S30, the information processing module 5 queries the corresponding pose of each point in the lidar point cloud based on the sampling time of each point in step S20 on the lidar continuous time pose prediction model, performs motion compensation on the point cloud and unifies it to the reference time; combines the relative motion increment information obtained from the pose prediction with the temporal motion consistency, neighborhood geometric consistency and echo intensity consistency of the point cloud to perform trailing discrimination on the points, and removes the trailing points to obtain a de-blurred point cloud without trailing.

[0124] Step S301: Information processing module 5 performs temporal unrolling on the single-frame point cloud output by lidar module 4 and reads the relative sampling time of each point. and the start timestamp of the corresponding frame Then, using the time offset output in step S20, the sampling time of the point is aligned to the time axis of the main IMU2 to obtain the global sampling time of the point:

[0125] ;

[0126] In the formula, It is the time offset between the lidar timestamp and the main IMU time axis.

[0127] Step S302: Information processing module 5 queries the sampling time on the continuous-time pose prediction model of the lidar obtained in step S20. Reference time The lidar pose is obtained, the relative motion increment from the sampling time to the reference time is calculated, and the Lie algebra representation of the relative motion increment is expressed. As a measure of jitter intensity during scanning:

[0128] ;

[0129] In the formula, It is a moment Pose transformation matrix from lidar coordinate system to world coordinate system; .

[0130] Step S303: Information processing module 5 unifies the points from their sampling time to the lidar coordinate system at the reference time based on the relative pose increment, in order to eliminate the geometric stretching and ghosting caused by motion during scanning, and obtain motion-compensated points:

[0131] ;

[0132] In the formula, It is the first The coordinates of a point in the lidar coordinate system; These are the coordinates after compensation to the reference time. From the sampling time to reference time The relative pose transformation matrix.

[0133] Step S304: Information processing module 5 further combines the temporal motion consistency, neighborhood geometric consistency, and laser echo intensity consistency of the point cloud itself to distinguish between real points and jittery trailing points. Specifically, it constructs a nearest neighbor set within the temporal and spatial neighborhoods of the compensation point. Fit a local plane on the nearest neighbor set and calculate the distance from the point to the plane. Calculate the coordinate consistency metric between the compensation point and its nearest neighbors. Calculate the laser echo intensity consistency metric The temporal motion consistency, neighborhood geometric consistency, laser echo intensity consistency, and relative motion intensity are used together for motion blur discrimination to construct a motion blur discrimination score:

[0134] ;

[0135] ;

[0136] ;

[0137] ;

[0138] In the formula, It is the unit normal vector of the local plane fitted by the nearest neighbor set. It is the bias term of the local plane. It is the first Laser echo intensity values ​​at each point; It is the motion blur discrimination score; , , and It is the weighting coefficient.

[0139] Step S305: Based on the ghosting discrimination score, establish robust weights for the points and perform removal processing. The robust weights are defined as follows:

[0140] ;

[0141] And based on the threshold The trailing indicator variable of the construction point:

[0142] ;

[0143] In the formula, It is the first Robust weights at each point, It is a motion blur indicator variable. It is the motion blur detection threshold. This indicates jittery or ghosting points that have been removed. Indicates the actual location.

[0144] Furthermore, the trailing detection threshold The settings are based on the lidar ranging noise level, the statistical distribution of relative motion intensity, or a preset signal interval, in order to balance the thoroughness of ghosting removal with the integrity of real-point retention.

[0145] Step S306: Take the point cloud after removing the trailing points as the output of the deblurred point cloud and use it as the geometric constraint input for subsequent camera-LiDAR calibration.

[0146] like Figure 4 The image shows a comparison diagram before and after point cloud removal of ghosting and blurring. Figure 4 The image shows the original point cloud without motion compensation and motion blur processing. Typical motion blur areas are marked with dashed boxes in the image. Figure 4 b shows the point cloud results after applying the motion compensation based on continuous time pose prediction and combined with motion blur discrimination and removal processing according to the present invention.

[0147] In step S40, the information processing module 5 calculates the pixel motion trajectory during the exposure period based on the camera continuous time pose prediction model and the rolling shutter line-by-line exposure time model obtained in step S20; constructs a spatially varying motion blur kernel based on the pixel motion trajectory; and performs non-blind deconvolution on the blurred image under the condition that the blur kernel is known to obtain a clear image.

[0148] Step S401: Information processing module 5 establishes a line-by-line exposure time model for the RGB rolling shutter camera, and uses the camera time offset output in step S20 to align the line-by-line exposure times to the time axis of the main IMU2; for the first... A frame image, let its first line exposure start timestamp be... The total number of rows in the image is The line readout time is Then the first The exposure time of a line is represented as:

[0149] ;

[0150] Then align the camera timestamps to the main IMU timeline to obtain:

[0151] ;

[0152] In the formula, It is the time offset between the camera timestamp and the main IMU time axis.

[0153] Furthermore, the information processing module 5 uses the line-by-line exposure time as the time index for rolling shutter deblurring, so that the motion compensation of each row of pixels is consistent with the exposure time.

[0154] In step S402, the information processing module 5, based on the camera continuous-time pose prediction model obtained in step S20, queries the camera pose at any time during the exposure period and calculates the camera's pose relative to the reference time. Continuous time relative rotation The relative rotation is used as a motion prior for deblurring the rolling shutter motion.

[0155] Step S403: In the rolling shutter model, the information processing module 5 constructs pixel trajectories using relative rotation during exposure; and processes homogeneous pixel coordinates on the reference time image. Construct a homography transformation induced by relative rotation:

[0156] ;

[0157] And obtain the motion trajectory of the pixel over time:

[0158] ;

[0159] In the formula, It is the relative rotation matrix of the camera relative to a reference time during the exposure period. It is the camera intrinsic parameter matrix. It is a perspective projection function.

[0160] Furthermore, the pixel trajectory construction process does not rely on the spatial extrinsic parameters between the LiDAR and the camera, nor does it rely on the LiDAR point cloud to provide depth for the pixels, thereby avoiding the implicit introduction of calibration results or external depth information into the image deblurring process.

[0161] Step S404: Information processing module 5 processes pixel trajectories within the exposure integration time. Discrete sampling is performed, and the pixel displacement sequence obtained from the discrete sampling is used to construct a spatial variation motion blur kernel or convolution kernel related to the pixel position. Based on the blur kernel, the blurred image is represented as a convolution superposition model of the clear image and the spatially varying motion blur kernel. Under the condition that the blur kernel is known, non-blind deconvolution with regularization constraints is used to achieve image deblurring. The deblurring optimization objective is written as:

[0162] ;

[0163] In the formula, It is a blurry image. It is a clear image to be recovered. It is a convolution operator. It is the total variation regularization term. It is the regularization weight.

[0164] Step S405: Perform robust post-processing on the deblurring result to suppress noise amplification and deconvolution ringing artifacts, and perform boundary consistency processing on the image boundary and occluded area to reduce boundary convolution error. Output a clear image after motion compensation deblurring, and use it as the visual constraint input for camera-LiDAR calibration in subsequent steps.

[0165] like Figure 5 The image shown is a comparison diagram of motion deblurring before and after rolling shutter speeds, in which... Figure 5 Image a is an unprocessed rolling shutter image, which exhibits ghosting and blurring at the edges under motion or vibration conditions; Figure 5 b shows the deblurred image after non-blind deconvolution and robust post-processing, constructed using the rolling shutter line-by-line exposure time model and camera continuous time pose prediction of this invention.

[0166] Step S50: Information processing module 5 constructs a camera-LiDAR joint calibration model based on the de-blurred and de-aliased point cloud obtained in step S30 and the clear image obtained in step S40; the spatial extrinsic parameters of LiDAR and camera, as well as their temporal offset, are used as parameters to be estimated; the de-blurred and de-aliased point cloud is projected onto the camera imaging plane to generate a LiDAR projection view, and cross-modal structural consistency constraints are established with the structural information of the clear image; the camera-LiDAR calibration results are output through nonlinear optimization joint solution.

[0167] Step S501: Using the camera-LiDAR calibration results as output, construct a set of variables to be estimated within a sliding window, and introduce the B-spline control points of the main IMU's continuous time trajectory and the IMU bias as intermediate variables to provide pose query results for the point cloud and image on a unified time axis. Define the state to be optimized as:

[0168] ;

[0169] In the formula, It is the set of states to be optimized within the sliding window. and These are the rigid external parameters of the camera and lidar relative to the main IMU, respectively. and These represent the time offsets for aligning the camera and LiDAR timestamps to the main IMU timeline. It is a continuous time pose trajectory The set of B-spline control points, It is the set of IMU bias parameters within the sliding window.

[0170] Furthermore, information processing module 5 uses historical calibration values ​​as... and The initial value is used, and the output of the time-aligned submodule is used as the initial value. and The initial value is determined to ensure that the online optimization has convergent initial conditions.

[0171] Step S502, by and Calculate the spatial extrinsic parameters between the lidar and the camera, and by and Calculate the time offset between the two, and define the output as:

[0172] ;

[0173] ;

[0174] In the formula, It is the external parameter of the rigid body from the lidar coordinate system to the camera coordinate system. It is a time offset that maps the LiDAR timestamp to the camera timeline.

[0175] Furthermore, the information processing module 5 is based on Mutual information is calculated using the joint histogram of the clear image, and the negative value of the mutual information is used as the cross-modal structural residual to improve alignment robustness under conditions of weak texture and varying illumination in the downhole environment. To enhance the stability of structural alignment, points in the point cloud with significant geometric abrupt changes or intensity variations are preferentially selected for the calculation of the cross-modal structural residual, and a robust kernel function is introduced into the residual to reduce the impact of outlier matching on optimization.

[0176] Step S503: Within the sliding window, jointly minimize the IMU kinematic residuals, lidar geometric residuals, and cross-modal structural residuals to establish the overall optimization objective:

[0177] ;

[0178] In the formula, It is the IMU kinematic residual. It is the IMU residual covariance matrix. and These are the scale parameters of the lidar geometric residuals and the cross-modal structural residuals, respectively. It is a robust kernel function.

[0179] The geometric residuals of the lidar include registration residuals from the midpoint of the de-ghosting point cloud to the local plane, from the point to the edge line, or from the point cloud registration of adjacent frames.

[0180] The IMU kinematic residuals include the pre-integration / kinetic consistency residuals between the continuous-time B-spline trajectory derivative and the main IMU inertial measurement, as well as the multi-source inertial consistency residuals between the auxiliary IMU and the main IMU.

[0181] The cross-modal structural residuals include negative values ​​of mutual information calculated from the joint histogram of the lidar projection view and the sharp image, or alignment residuals constructed based on the mutual information.

[0182] Furthermore, the information processing module 5 performs iterative optimization according to the following process: the estimated state is initialized with the initial values ​​of the extrinsic parameters and the initial values ​​of the time offset in step S501; in each iteration, the IMU kinematic residual, the lidar geometric residual and the cross-modal structural residual are calculated sequentially and the optimization objective is constructed; the estimated state is incrementally updated and substituted back to the extrinsic parameters and the time offset; the iteration stops when the increment of the extrinsic parameters and the increment of the time offset are both less than the preset threshold or the decrease in the objective function is less than the preset threshold.

[0183] Step S504: Solve the overall optimization objective using a nonlinear least squares iterative algorithm to obtain the converged result. and The calibration results are used as the calibration results for camera-LiDAR, and these calibration results are then used for multimodal fusion of point cloud and image.

[0184] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes that can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention are all within the protection scope of the claims of the present invention.

Claims

1. A method for online spatiotemporal calibration of airborne multi-sensor systems under complex environments in fully mechanized mining faces, applied to a system comprising an airborne platform for fully mechanized mining faces, a main IMU, a camera module, a lidar module, and an information processing module, wherein the main IMU, camera module, and lidar module are installed on the airborne platform for fully mechanized mining faces and connected to the information processing module, the camera module comprising a camera and a corresponding Camera-IMU, and the lidar module comprising a lidar and a corresponding LiDAR-IMU; characterized in that, Includes the following steps: Step S10: Real-time acquisition of main IMU inertial data, camera module images and corresponding inertial data, and lidar module point cloud and corresponding inertial data; Step S20: Using the main IMU time axis as a reference, estimate and compensate for the time offset of the corresponding inertial measurement units of the camera module and the lidar module online, and establish a continuous time pose prediction model; the continuous time pose prediction model includes a lidar continuous time pose prediction model and a camera continuous time pose prediction model. Step S30: Based on the continuous time pose prediction model of lidar, perform motion compensation and time unification on the point cloud, identify and remove jittery trailing points, and obtain a trailing point cloud. Step S40: Combine the camera continuous time pose prediction model and the rolling shutter line-by-line exposure model to construct a spatial variation blur kernel and perform non-blind deconvolution on the image to obtain a clear image; Step S50: Construct cross-modal structural consistency constraints based on the de-ghosting point cloud and the clear image, and solve the camera-LiDAR spatial extra-space participation time migration jointly through nonlinear optimization to output the calibration results.

2. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 1, characterized in that, In step S20, the step of estimating and compensating for the time offset between the camera and the lidar inertial measurement based on the main IMU time axis, and establishing a continuous-time pose prediction model, specifically includes the following steps: Step S201: Using the local time axis of the main IMU as a unified time reference, the information processing module performs online alignment of the inertial measurement timestamps output by the LiDAR-IMU, and estimates the time offset of the LiDAR-IMU relative to the main IMU using the angular velocity sequence or angular velocity magnitude sequence within the sliding time window as the alignment object. After the alignment is completed, a time mapping relationship is established. ； In the formula, and Main IMU timestamp, For LiDAR-IMU local timestamps, The time offset for aligning the LiDAR-IMU timestamp to the main IMU time axis; Step S202: Using the local time axis of the main IMU as a unified time reference, the information processing module performs online alignment of the inertial measurement timestamps output by the Camera-IMU, and estimates the time offset of the Camera-IMU relative to the main IMU using the angular velocity sequence or angular velocity magnitude sequence within the sliding time window as the alignment object. After the alignment is completed, a time mapping relationship is established. ； In the formula, For Camera-IMU local timestamps, The time offset for aligning the Camera-IMU timestamp to the main IMU time axis; Step S203: The information processing module establishes the carrier's continuous-time pose trajectory on the main IMU time axis, and uses Lie groups. The B-spline control points are parameterized; the continuous-time pose trajectory control points and LiDAR-IMU time offset are jointly optimized within the sliding time window. Camera-IMU time offset The continuous-time pose trajectory is updated online with IMU bias parameters to serve as a common motion reference during lidar scanning and camera exposure. Step S204: Based on the inertial measurements of the main IMU and the rigid mounting relationship between the LiDAR-IMU and Camera-IMU relative to the main IMU, the information processing module establishes multi-source inertial consistency constraints under a unified time axis. ； In the formula, Indicates the type of auxiliary inertial measurement unit. LiDAR-IMU, Indicates Camera-IMU; and These are the main IMU debiasing angular velocity and acceleration measurements, respectively. and The first The bias debias angular velocity and acceleration of the auxiliary IMU were measured. For the first Fixed rotational extrinsic parameters from auxiliary IMU to main IMU; For the first The time offset of each auxiliary IMU to the time axis of the main IMU; Step S205: After obtaining the continuous-time pose trajectory of the carrier, the information processing module establishes a continuous-time pose prediction model for the lidar and a continuous-time pose prediction model for the camera under a unified time axis, based on the rigid mounting relationship between the lidar and the camera relative to the main IMU and their current external parameter estimates. This serves as the motion prior for point cloud de-ghosting and image de-blurring. ； ； In the formula, For a moment Pose transformation matrix from the main IMU to the world coordinate system; Fixed rigid external parameters for laser radar to the main IMU; External parameters of the fixed rigid body from the camera to the main IMU; For a moment The pose transformation matrix of lidar to the world coordinate system; For a moment The pose transformation matrix from the camera to the world coordinate system.

3. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 1, characterized in that, In step S30, the motion compensation and time unification of the point cloud based on the continuous-time pose prediction model of the lidar, the identification and removal of jittery and trailing points, and the obtaining of the trailing point cloud specifically include the following steps: Step S301: The information processing module performs temporal unrolling on the single-frame point cloud output by the lidar module and reads the relative sampling time of each point. and the start timestamp of the corresponding frame The sampling time of the point is aligned to the time axis of the main IMU using the time offset output in step S20 to obtain the global sampling time of the point: ； In the formula, It is the time offset between the lidar timestamp and the main IMU time axis; Step S302: The information processing module queries the sampling time on the continuous-time pose prediction model of the lidar obtained in step S20. Reference time The lidar pose is obtained, the relative motion increment from the sampling time to the reference time is calculated, and the Lie algebra representation of the relative motion increment is expressed. As a measure of jitter intensity during scanning: ； In the formula, It is a moment Pose transformation matrix from lidar coordinate system to world coordinate system; ; Step S303: The information processing module unifies the points from their sampling time to the lidar coordinate system at the reference time based on the relative pose increment, in order to eliminate the geometric stretching and ghosting caused by motion during scanning, and obtain the motion-compensated points: ； In the formula, It is the first The coordinates of a point in the lidar coordinate system; These are the coordinates after compensation to the reference time; From the sampling time to reference time The relative pose transformation matrix; Step S304: The information processing module combines the temporal motion consistency, neighborhood geometric consistency, and laser echo intensity consistency of the point cloud itself to distinguish between real points and jittery trailing points, and constructs a trailing discrimination score. Step S305: Establish robust weights for points based on the motion blur discrimination score and perform removal processing; the robust weight is defined as: ； And based on the threshold The trailing indicator variable of the construction point: ； In the formula, It is the first Robust weights at each point, It is a motion blur indicator variable. It is the motion blur detection threshold. This indicates jittery or ghosting points that have been removed. Indicates the actual location; Step S306: Take the point cloud after removing the trailing points as the output of the de-trailing point cloud, and use it as the geometric constraint input for subsequent LiDAR-camera calibration.

4. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 3, characterized in that, In step S304, the information processing module combines the temporal motion consistency, neighborhood geometric consistency, and laser echo intensity consistency of the point cloud itself to distinguish between real points and jittery trailing points, and constructs a trailing discrimination score, specifically: Construct a nearest neighbor set within the temporal and spatial neighborhoods of the compensation point. Fit a local plane on the nearest neighbor set and calculate the distance from the point to the plane. Calculate the coordinate consistency metric between the compensation point and its nearest neighbors. Calculate the laser echo intensity consistency metric The temporal motion consistency, neighborhood geometric consistency, laser echo intensity consistency, and relative motion intensity are used together for motion blur discrimination to construct a motion blur discrimination score: ； ； ； ； In the formula, It is the unit normal vector of the local plane fitted by the nearest neighbor set. It is the bias term of the local plane. It is the first Laser echo intensity values ​​at each point; It is the motion blur discrimination score; , , and It is the weighting coefficient.

5. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 3, characterized in that, In step S305, the motion blur detection threshold The settings are based on the lidar ranging noise level, the statistical distribution of relative motion intensity, or a preset signal interval, in order to balance the thoroughness of ghosting removal with the integrity of real-point retention.

6. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 1, characterized in that, In step S40, the process of combining the camera's continuous-time pose prediction model with the rolling shutter line-by-line exposure model to construct a spatially variable blur kernel and non-blindly deconvolve the image to obtain a clear image specifically includes the following steps: Step S401: The information processing module establishes a line-by-line exposure time model for the RGB rolling shutter camera, and uses the camera time offset output in step S20 to align the line-by-line exposure times to the time axis of the main IMU; for the... A frame image, let its first line exposure start timestamp be... The total number of rows in the image is The line readout time is Then the first The exposure time of a line is represented as: ； Then align the camera timestamps to the main IMU timeline to obtain: ； In the formula, It is the time offset between the camera timestamp and the main IMU time axis; Step S402: Based on the camera continuous-time pose prediction model obtained in step S20, the information processing module queries the camera pose at any time during the exposure period and calculates the camera's pose relative to the reference time. Continuous time relative rotation The relative rotation is used as a motion prior for deblurring the rolling shutter motion; Step S403: In the rolling shutter model, the information processing module constructs pixel trajectories using relative rotation during exposure; and processes homogeneous pixel coordinates on the reference time image. Construct a homography transformation induced by relative rotation: ； And obtain the motion trajectory of the pixel over time: ； In the formula, It is the relative rotation matrix of the camera relative to a reference time during the exposure period. It is the camera intrinsic parameter matrix. It is a perspective projection function; Step S404: The information processing module processes the pixel trajectory within the exposure integration time. Discrete sampling is performed, and the pixel displacement sequence obtained from the discrete sampling is used to construct a spatial variation motion blur kernel or convolution kernel related to the pixel position. Based on the blur kernel, the blurred image is represented as a convolution superposition model of the clear image and the spatially varying motion blur kernel. Under the condition that the blur kernel is known, non-blind deconvolution with regularization constraints is used to achieve image deblurring. The deblurring optimization objective is denoted as: ； In the formula, It is a blurry image. It is a clear image to be recovered. It is a convolution operator. It is the total variation regularization term. It is the regularization weight; Step S405: Perform robust post-processing on the deblurring result to suppress noise amplification and deconvolution ringing artifacts, and perform boundary consistency processing on the image boundary and occluded area to reduce boundary convolution error. Output a clear image after motion compensation and deblurring processing, and use it as the visual constraint input for LiDAR-camera calibration in subsequent steps.

7. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 6, characterized in that, Step S401 further includes the following step: the information processing module uses the line-by-line exposure time as the time index for rolling shutter deblurring, so that the motion compensation of each row of pixels is consistent with the exposure time.

8. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 6, characterized in that, In step S403, the construction process of the pixel trajectory does not rely on the spatial extrinsic parameters between the LiDAR and the camera, nor does it rely on the LiDAR point cloud to provide depth for the pixels, thereby avoiding the implicit introduction of calibration results or external depth information into the image deblurring process.

9. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 1, characterized in that, In step S50, the process of constructing cross-modal structural consistency constraints based on the de-ghosting point cloud and the clear image, and jointly solving the camera-LiDAR out-of-space participation time migration through nonlinear optimization to output the calibration result, specifically includes the following steps: Step S501: Using the camera-LiDAR calibration results as output, construct a set of variables to be estimated within a sliding window, and introduce the B-spline control points of the main IMU's continuous time trajectory and the IMU bias as intermediate variables to provide pose query results for the point cloud and image on a unified time axis. Define the state to be optimized as: ； In the formula, It is the set of states to be optimized within the sliding window. and These are the rigid external parameters of the camera and lidar relative to the main IMU, respectively. and These represent the time offsets for aligning the camera and LiDAR timestamps to the main IMU timeline. It is a continuous time pose trajectory The set of B-spline control points, It is the set of IMU bias parameters within the sliding window; The information processing module uses historical calibration values ​​as... and The initial value is used, and the output of the time-aligned submodule is used as the initial value. and The initial value is determined to ensure that the online optimization has convergent initial conditions; Step S502, by and Calculate the spatial extrinsic parameters between the lidar and the camera, and by and Calculate the time offset between the two, and define the output as: ； ； In the formula, It is the external parameter of the rigid body from the lidar coordinate system to the camera coordinate system. It is a time offset that maps the lidar timestamp to the camera timeline; Step S503: Within the sliding window, jointly minimize the IMU kinematic residuals, lidar geometric residuals, and cross-modal structural residuals to establish the overall optimization objective: ； In the formula, It is the IMU kinematic residual. It is the IMU residual covariance matrix. and These are the scale parameters of the lidar geometric residuals and the cross-modal structural residuals, respectively. It is a robust kernel function; The lidar geometric residuals include the residuals from the midpoint of the de-ghosting point cloud to the local plane, from the point to the edge line, or the registration residuals of the point clouds in adjacent frames. The IMU kinematic residuals include the pre-integration / kinetic consistency residuals between the continuous-time B-spline trajectory derivative and the main IMU inertial measurement, as well as the multi-source inertial consistency residuals between the auxiliary IMU and the main IMU. The cross-modal structural residuals include negative values ​​of mutual information calculated from the joint histogram of the lidar projection view and the sharp image, or include alignment residuals constructed based on the mutual information. Step S504: Solve the overall optimization objective using a nonlinear least squares iterative algorithm to obtain the converged result. and The calibration results are used as the calibration results for camera-LiDAR, and these calibration results are then used for multimodal fusion of point cloud and image.

10. The airborne multi-sensor spatiotemporal online calibration method for complex environments in fully mechanized mining faces according to claim 9, characterized in that, The information processing module is iteratively optimized according to the following process: Initialize the estimated state with the initial values ​​of the external parameters and the initial value of the time offset from step S501; In each iteration, the IMU kinematic residual, the lidar geometric residual, and the cross-modal structural residual are calculated sequentially, and an optimization objective is constructed. The estimated state is incrementally updated and then substituted back to the external parameters and time offset; The iteration stops when both the external parameter increment and the time offset increment are less than the preset threshold or the objective function decrease is less than the preset threshold.

Images