Multi-modal information fusion odometer construction method and system for star catalogue positioning

Abstract

The invention discloses a multi-modal information fusion odometer construction method for star catalogue positioning, and relates to the technical field of star catalogue patroller positioning. The method comprises the following steps: carrying out space joint calibration on a monocular camera, a laser radar and an inertial measurement unit, reconstructing a laser radar point cloud by using a timestamp of a camera image, and realizing time synchronization of the camera image and the laser radar point cloud; establishing an IMU pre-integration error model; and performing motion compensation distortion removal on the laser point cloud by using an IMU pre-integration result, and extracting geometric features of the distorted laser point cloud based on a neighbor region smoothness calculation method of a fixed measurement distance. By researching a multimodal information fusion odometer method, the accumulative error of motion measurement is reduced, the positioning precision and stability are improved, technical support is provided for design and development of a star catalogue navigation system, and the problems that a single-modal star catalogue positioning method is weak in environment adaptive capacity and poor in algorithm generalization are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of star chart rover positioning, and in particular to a method and system for constructing a multimodal information fusion odometer for star chart positioning. Background Art

[0002] In recent years, the implementation of lunar and Martian surface rover exploration missions has significantly advanced the development of high-efficiency deep-space scientific exploration technologies, providing critical data support for studies of the evolutionary mechanisms of extraterrestrial bodies and planetary habitability. Autonomous positioning on the surface of the star is a foundational technology for rover exploration systems. By using measurement information from solver-borne sensors, it provides high-precision pose parameters and scene point cloud maps. This provides a spatiotemporal reference for trajectory planning algorithms, mission decision models, and motion control systems, effectively supporting the smooth implementation of scientific missions such as resource exploration, mapping, and the search for life.

[0003] However, the complex characteristics of the star-surface environment, such as dust occlusion, extreme illumination changes, high-similarity scenes, and unstructured terrain, can easily degrade vision, lidar, and other sensors. In visually degraded scenes such as drastic illumination changes and motion blur, positioning methods have difficulty effectively capturing or tracking visual image features based on grayscale changes. In lidar-degraded scenes such as sparse vertical markers and particle occlusion, the lack of laser measurement return values ​​leads to insufficient constraints on the positioning algorithm. Existing star-surface positioning methods that use a single sensor or single modal information often have low confidence levels due to problems such as sensor degradation or insufficient observation information, and face the risk of functional failure. To this end, the present invention proposes a method and system for constructing a multi-modal information fusion odometer for star-surface positioning. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for constructing a multimodal information fusion odometer for star chart positioning. By studying the odometer method of multimodal information fusion, the cumulative error of motion measurement is reduced, the positioning accuracy and stability are improved, and technical support is provided for the design and development of star chart navigation systems.

[0005] According to a first aspect of the present invention, to achieve the above-mentioned purpose, the present invention provides the following technical solution: a method for constructing a multimodal information fusion odometer for star chart positioning, applied to a star chart cruiser equipped with a monocular camera, a lidar, and an inertial measurement unit, comprising the following steps:

[0006] S1. Perform spatial joint calibration of the monocular camera, lidar, and inertial measurement unit, receive camera image data, lidar point cloud data, and inertial measurement data, reconstruct the lidar point cloud using the camera image timestamp, and achieve time synchronization between the camera image and the lidar point cloud.

[0007] S2. Frame the camera image to obtain continuous visual image frames, pre-integrate the IMU measurement values ​​between two adjacent image frames in the continuous visual image frames, obtain the IMU short-term prediction value, compare the relative change of the IMU short-term prediction value with the nearest neighbor IMU measurement value of the two image frames, and establish an IMU pre-integration error model;

[0008] S3. Using the IMU pre-integration result of step S2, the laser point cloud is subjected to motion compensation dedistortion, and the geometric features of the dedistorted laser point cloud are extracted based on the neighborhood smoothness calculation method with a fixed measurement distance;

[0009] S4. Extract visual corner features from the camera image, project the dedistorted continuous frame point cloud output from step S3 into the camera image coordinate system, associate the geometric features of the laser point cloud with the depth values ​​of the visual corner features, and perform outlier filtering on the associated depth values ​​by verifying the consistency of reprojection across time-series images;

[0010] S5. Based on the visual corner feature in step S4, use the descriptor algorithm to match adjacent frame images to obtain the corresponding tracking feature, project the tracking feature to the image coordinate system where the current visual corner feature is located, and establish a visual reprojection error model between the projected point and the visual corner point;

[0011] S6. Perform nearest neighbor matching on the geometric features of the point cloud in step S3 and construct a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface;

[0012] S7. Combine the IMU pre-integration error model in step S2, the visual reprojection error model in step S5, and the point-to-random surface point cloud matching error model in step S6 to construct the objective function of the multimodal information fusion odometry, and calculate the position and pose of the rover using the incremental graph optimization solver.

[0013] S8. Repeat steps S2 to S7 to achieve continuous pose transformation estimation and complete the construction of the multimodal information fusion odometry.

[0014] Furthermore, in step S1, spatial joint calibration is performed on the monocular camera, lidar, and inertial measurement unit, and camera image data, lidar point cloud data, and inertial measurement data are received. The lidar point cloud is reconstructed using the timestamp of the camera image to achieve time synchronization between the camera image and the lidar point cloud, as follows:

[0015] Step S1-1. Implement the intrinsic calibration of the monocular camera, the extrinsic calibration of the camera and lidar, and the extrinsic calibration of the camera and IMU respectively, and achieve accurate fusion of camera, lidar, and IMU data in a unified coordinate system, as follows:

[0016] Camera intrinsic parameter calibration: Use the OpenCV calibration tool to generate a checkerboard calibration plate pattern. Based on this, use a monocular camera to observe the calibration plate image, establish the optical projection function of the calibration plate corner points, and use Zhang Zhengyou camera calibration method to solve the camera's intrinsic parameter matrix and distortion coefficient;

[0017] Camera and LiDAR extrinsic parameter calibration: Use the Autoware open-source calibration tool to estimate the extrinsic parameters between a single camera and LiDAR. Use a calibration plate or 3D target with spatial features, simultaneously acquire image and point cloud data, extract the characteristic corner points corresponding to the calibration plate or 3D target, establish an observation constraint function for the characteristic corner points, and solve the change matrix between the camera and LiDAR to achieve camera and LiDAR extrinsic parameter calibration.

[0018] External parameter calibration of the camera and IMU: The IMU and camera are calibrated using the Kalibr tool and motion registration method. The IMU is rigidly mounted on the camera bracket, and a segment of data containing motion information is simultaneously recorded. The angular velocity or acceleration data of the IMU and the camera's position change are extracted, and the external parameters from the IMU to the camera are solved through optimization.

[0019] After the IMU and camera are calibrated, the transformation relationship from IMU to LiDAR is indirectly obtained by knowing the external parameters from the camera to LiDAR, thereby completing the spatial joint calibration of the camera, LiDAR and IMU.

[0020] Step S1-2. Reconstruct the LiDAR point cloud using the timestamp of the camera image to achieve time synchronization between the camera image and the LiDAR point cloud, as follows:

[0021] Taking advantage of the continuous scanning characteristics of the rotating lidar, each frame of the lidar scan is divided into multiple scanning segments according to the camera frequency, and the scanning segments are spliced ​​in time series to generate a high-frequency reconstructed scan aligned with the camera time. After reconstruction, the lidar point cloud acquisition frequency is improved and kept synchronized with the camera image acquisition frequency.

[0022] Furthermore, in step S2, the camera image is framed to obtain continuous visual image frames, and the IMU measurement values ​​between two adjacent image frames in the continuous visual image frames are pre-integrated to obtain the IMU short-term prediction value. The relative change of the IMU short-term prediction value and the nearest neighbor IMU measurement value of the two image frames is compared to establish the IMU pre-integration error model, which is as follows:

[0023] Step S2-1. The IMU pre-integration between two adjacent image frames i and j is as follows:

[0024]

[0025]

[0026] In the formula They are the rotation component, velocity component and translation component of the IMU pre-integrated quantity respectively; R i and R j are the rotation matrices of the IMU measurements closest to image frame i and image frame j respectively; v i and v j Indicates the speed measured by IMU, p i and p j represents the translation measured by the IMU; Δt represents the time interval between image frame i and image frame j; k represents the sequence number of the IMU measurement between image frame i and image frame j; w k and a k are the angular velocity and translational acceleration measured by the IMU at the kth frame; g represents the gravity vector; Δv ik , ΔR ik They represent the velocity change and rotation change measured by the IMU at the kth frame relative to the ith frame respectively; and are the zero bias of the accelerometer and gyroscope in the IMU at image frame i, respectively, which are modeled as random noise;

[0027] Step S2-2. The short-term prediction value of the IMU pre-integration in the lidar or camera coordinate system is:

[0028]

[0029] The superscript O in the formula represents the state in the laser radar or camera coordinate system; They are the rotation component, velocity component and translation component of the IMU pre-integration in the laser radar or camera coordinate system; is the rotation matrix of the external parameters between the lidar or camera coordinate system and the IMU coordinate system;

[0030] IMU pre-integration error model Expressed as:

[0031]

[0032] in, Respectively represent the components of the IMU pre-integration error model in rotation, velocity, translation, accelerometer bias, and gyroscope bias; subscript I ij Represents IMU measurement; the superscript g represents the state quantity in the world coordinate system; Represents the rotation matrix of the IMU measurement in the global coordinate system of the i-th frame; and Represent the IMU velocity in the global coordinate system of the i-th frame and the j-th frame respectively; and Respectively represent the IMU translation in the global coordinate system of the i-th frame and the j-th frame; X i , X j are the poses of the i-th frame and the j-th frame respectively.

[0033] Furthermore, in step S3, the laser point cloud is subjected to motion compensation dedistortion using the IMU pre-integration result of step S2, and the geometric features of the dedistorted laser point cloud are extracted based on the neighborhood smoothness calculation method with a fixed measurement distance, as follows:

[0034] Step S3-1. Using the high-frequency acceleration and angular velocity data provided by the IMU, perform spatiotemporal compensation on each laser point, unifying them to the same reference coordinate system to achieve dedistortion. The continuous motion trajectory within the lidar scan time is obtained using the IMU measurements and pre-integration results from step S2, along with the short-term predictions in the lidar coordinate system. The corresponding motion matrix is ​​obtained based on the linear interpolation method and the lidar scan point sampling time. The distorted points are then converted to the current lidar coordinate system through the inverse transformation of the motion matrix, forming an undistorted point cloud at the same time.

[0035] Step S3-2. Extracting point cloud features based on the smoothness of the neighboring area point cloud at a fixed measurement distance;

[0036] Smoothness estimation of the dedistorted point cloud is performed to distinguish the planar feature points in the point cloud;

[0037] The points adjacent to the feature points to be extracted are limited to a fixed area with a certain distance threshold. Laser points that are close have more neighboring points, while points that are far away have fewer neighboring points. This keeps the smoothness of all points in the current frame scan at the same scale. The point cloud smoothness is calculated as:

[0038]

[0039] Where C(·) is the mathematical expression of point cloud smoothness, subscript n l Indicates the serial number of the point in the current scan point cloud; the superscript L indicates the observation target of the lidar; Represents a lidar point The ranging range; express nth l The distance measurement range of the nearest neighbor points; r blind and r max They are the near blind area of ​​the return point and the maximum measurement distance; D i is the laser point in the i-th frame laser scanning The set of consecutive points of the nearest neighbors; |D i |=180H / πhr n D iThe number of consecutive points in the curve; H and h are the curve segment length and horizontal angular resolution respectively;

[0040] On this basis, if the smoothness of the target point is greater than the preset threshold τ = 5, the point is set as an edge point, otherwise it is set as a plane point;

[0041] The feature point set of the i-th lidar scan is recorded as F i , the edge feature point set is recorded as The plane feature point set is recorded as The superscripts e and p represent edge features and plane features, respectively.

[0042] Furthermore, in step S4, the visual corner features of the camera image are extracted, the dedistorted continuous frame point cloud output from step S3 is projected into the camera image coordinate system, the depth values ​​of the laser point cloud geometric features and the visual corner features are associated, and the associated depth values ​​are filtered out by reprojection consistency verification across time-series images, as follows:

[0043] Step S4-1. Use the Shi-Tomasi algorithm to quickly extract sparse corner features with large grayscale gradients in the image;

[0044] Step S4-2: Depth value association between laser point cloud and visual corner feature:

[0045] The multi-frame LiDAR scan data within a continuous time window is fused into a dense point cloud in the same coordinate system. The 3D point cloud is projected to the camera coordinate system using the extrinsic parameter transformation matrix between the LiDAR and the camera. The point cloud is then further projected onto the image plane using the camera intrinsic parameter matrix to form a dense depth point distribution.

[0046] The projected point cloud is then downsampled and indexed using a two-dimensional KD tree. All points are projected onto a unit sphere with the camera center as the origin, and each point is represented by its polar coordinate angle to accelerate the search.

[0047] The visual features extracted from the image are back-projected onto the unit sphere, and the three nearest neighbor lidar points are searched on the sphere. The geometric consistency of these three lidar points in three-dimensional space is checked to determine the validity of the association, eliminating occlusion or cross-object situations. For visual features that pass the validity check, the line of sight from the camera center to the visual feature is constructed, and its intersection with the plane formed by the three lidar points is calculated. The distance between the intersection points is the depth value of this visual feature. For point features that are not successfully associated with depth, triangulation based on disparity information is used for supplementary estimation;

[0048] Step S4-3: Depth value outlier filtering based on the reprojection consistency verification method across time-series images:

[0049] Extract the 3D feature points with the same name that can be observed in multiple consecutive image frames, record their initial depth assignments, reproject these 3D points to the image planes of other frames, and calculate the reprojection error between them and the real image feature points;

[0050] For each set of image frames that can simultaneously observe 3D points, the feature points in one frame are projected onto other frames through relative pose transformation, and the projection error is evaluated to see if it falls within a preset threshold. If the error exceeds the limit, the depth value corresponding to the current frame is judged to be abnormal.

[0051] All depth assignments form a Gaussian distribution centered on the average depth. By removing abnormal projection points that exceed the projection error threshold, feature points with good reprojection consistency are retained. If the effective depth number of a feature is lower than the minimum threshold, this depth will not be used for fusion; if the observation quantity requirement is met, the average depth after removing the outliers is used as the final depth result.

[0052] Furthermore, in step S5, based on the visual corner point features in step S4, the descriptor algorithm is used to match adjacent frame images to obtain corresponding tracking features, and the tracking features are projected to the image coordinate system where the current visual corner point features are located. A visual reprojection error model is established between the projected points and the visual corner points, as follows:

[0053] Based on the extracted visual corner features, the KLT optical flow tracking method is used to match the corner points in adjacent image frames to obtain stable tracking feature point pairs. The 3D feature points tracked in the previous frame are projected into the current frame image coordinate system based on the camera pose estimation and correspond to the visual corner features extracted in the current frame.

[0054] The visual reprojection error model between the projection point and the actual observation point is established as follows:

[0055]

[0056] where r c (·) is the mathematical expression of the point feature projection model; and Refers to observing the cth 3D point in the i-th frame and the j-th frame, yes The 2D corner feature obtained by projecting on the image plane of the i-th frame; π(·) is the optical projection function used to project the image feature points from the camera coordinate system to the image plane; b1 and b2 are Two orthogonal bases of tangent planes.

[0057] Furthermore, in step S6, the geometric features of the point cloud in step S3 are matched with the nearest neighbor points, and a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface is constructed, as follows:

[0058] Step S6-1. Model the local point cloud neighborhood as a Gaussian surface with mean and uncertainty, and express the residual error between the point and the surface in the form of Mahalanobis distance, so as to achieve fine modeling and registration of non-flat surfaces in the unstructured environment of the star catalog;

[0059] For the target point in the laser scan of the i-th frame Extract N in its neighborhood a laser points, due to the existence of laser radar measurement noise and cumulative error, The position of is uncertain, which is recorded as the uncertainty matrix The uncertainty matrix is ​​the inverse matrix of the covariance matrix W, which can be calculated as:

[0060]

[0061] The covariance matrix is ​​calculated from the difference between the neighboring points and the Gaussian mean, where The Gaussian mean of the neighborhood is usually set to The three-dimensional position of is the summation symbol, n′ represents The sequence number of the neighbor point, q n′ for The neighboring points of -1 Characterizes the uncertainty of the point to the Gaussian surface, defined as the weight matrix of the Mahalanobis distance metric;

[0062] Step S6-2. For the local map M from the i-th frame i Points to be registered The subscript ma represents the neighboring points in the map, which is The matching error is expressed as Then the residual between the point cloud feature and the target random surface in the i-th frame scan is defined as:

[0063]

[0064] where r pg (·) is the mathematical representation of the Mahalanobis distance from a point to a random surface; N is the number of points to be matched in the i-th frame scan;

[0065] The inverse covariance matrix W -1 By the eigenvector matrix N pg and the diagonal matrix Λ pg Therefore, by decomposing the matrix W -1 , convert the distance from a point to a random surface into:

[0066]

[0067] Where diag(·) is the mathematical representation of a diagonal matrix, λ1,λ2,λ3 are the matrices W -1 The descending eigenvalues ​​of , ε1, ε2, ε3 are the eigenvectors corresponding to the eigenvalues ​​λ1, λ2, λ3; the superscript T is the transpose operation of the matrix and vector;

[0068] Step S6-3. Convert the point-to-random surface distance model into a point-to-surface cost function, which is the sum of squared Euclidean distances weighted by direction:

[0069]

[0070] That is, it is expressed as a weighted projection error on multiple direction vectors, which is used to measure the degree of spatial deviation between the matching point and the random surface.

[0071] Furthermore, in step S7, the objective function of the multimodal information fusion odometry is constructed as follows:

[0072]

[0073] where N s is a set of frame numbers; c is the set of visual observation feature points; r0 is the prior error of the initial state; represents the square of the 2-norm; The independent variable X that makes the function reach its minimum value i The value of

[0074] By solving the objective function using factor graph optimization, the position of the star rover at the i-th frame can be obtained.

[0075] According to a second aspect of the present invention, the present invention provides a multimodal information fusion odometer construction system for star chart positioning, which is used to implement the multimodal information fusion odometer construction method for star chart positioning described in the first aspect, including:

[0076] The sensor spatiotemporal synchronization module is used to perform spatial joint calibration for the monocular camera, lidar, and inertial measurement unit. It receives camera image data, lidar point cloud data, and inertial measurement data, reconstructs the lidar point cloud using the camera image timestamp, and achieves time synchronization between the camera image and the lidar point cloud.

[0077] The first building module is used to perform frame processing on the camera image to obtain continuous visual image frames, pre-integrate the IMU measurement values ​​between two adjacent image frames in the continuous visual image frames, obtain the IMU short-term prediction value, compare the relative change of the IMU short-term prediction value with the nearest neighbor IMU measurement values ​​of the two image frames, and establish an IMU pre-integration error model;

[0078] The feature extraction module is used to perform motion compensation dedistortion on the laser point cloud using the IMU pre-integration results, and extract the geometric features of the dedistorted laser point cloud based on the neighborhood smoothness calculation method with a fixed measurement distance;

[0079] The depth value association module is used to extract visual corner features from camera images, project the dedistorted continuous frame point cloud into the camera image coordinate system, associate the geometric features of the laser point cloud with the depth values ​​of the visual corner features, and filter outliers from the associated depth values ​​by verifying the consistency of reprojection across time-series images.

[0080] The second construction module is used to match adjacent frame images using a descriptor algorithm based on the visual corner feature to obtain the corresponding tracking feature, project the tracking feature to the image coordinate system where the current visual corner feature is located, and establish a visual reprojection error model between the projection point and the visual corner point;

[0081] The third construction module is used to perform nearest neighbor matching on the geometric features of the point cloud and build a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface;

[0082] The multimodal fusion positioning module is used to combine the IMU pre-integration error model, the visual reprojection error model, and the point cloud matching error model from point to random surface to construct the objective function of the multimodal information fusion odometry and calculate the position and pose of the rover through the incremental graph optimization solver.

[0083] The iterative optimization module is used for repeated iterations to achieve continuous pose transformation estimation and complete the construction of a multimodal information fusion odometry.

[0084] According to a third aspect of the present invention, the present invention provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, the multimodal information fusion odometer construction method for star catalog positioning described in the first aspect is adopted.

[0085] The present invention has at least the following beneficial effects:

[0086] 1. The lidar scan reconstruction method based on camera sampling timestamps adopted in this invention can effectively align camera images and laser point clouds, further improve the correction frequency of IMU bias, reduce the cumulative error of IMU motion measurement, and improve the positioning accuracy of the odometer.

[0087] 2. The method for calculating the smoothness of the point cloud in the neighboring area with a fixed measurement distance proposed in the present invention normalizes the smoothness of the entire frame of laser point cloud scanning, solves the problem of inconsistent smoothness scales caused by different distances, and is conducive to constructing a random surface model of the point cloud.

[0088] 3. The depth-correlated point removal method proposed in this invention based on reprojection consistency verification across time-series images reduces the interference of measurement errors in motion estimation, extrinsic parameter estimation, and depth interpolation on the pose estimation of the visual odometry subsystem, thereby ensuring the accuracy of lidar-camera fusion.

[0089] 4. The point-to-random surface distance model proposed in this paper for point cloud matching constructs an environmental modeling method that simultaneously represents the Gaussian mean and surface uncertainty. This breaks through the traditional matching algorithm's reliance on the regular plane assumption, more accurately depicts the local geometric morphology in the unstructured terrain of the star map, and improves the robustness and positioning accuracy of point cloud registration in rugged and complex terrain.

[0090] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 Schematic diagram of the flow of the odometer construction method of the present invention;

[0092] Figure 2 This is a schematic diagram of the framework of the odometer construction method of the present invention;

[0093] Figure 3 This is a schematic diagram of the multi-sensor joint calibration principle of the present invention;

[0094] Figure 4 This is a schematic diagram of the LiDAR scan reconstruction principle based on camera timestamps of the present invention;

[0095] Figure 5 This is a schematic diagram of the laser point cloud-visual feature depth value correlation principle of the present invention;

[0096] Figure 6 Schematic diagram of the structural principle of the point-to-random surface distance model of the present invention. DETAILED DESCRIPTION

[0097] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0098] Example 1:

[0099] See also Figures 1-6 The present invention provides a technical solution: a method for constructing a multimodal information fusion odometer for star catalog positioning, comprising the following steps:

[0100] Step 1: Perform spatial joint calibration on the camera, lidar, and IMU, reconstruct the lidar point cloud using the timestamp of the camera image, and achieve time synchronization between the camera image and the lidar point cloud;

[0101] Step 1-1: respectively realize the intrinsic calibration of the monocular camera, the extrinsic calibration of the camera and the lidar, and the extrinsic calibration of the camera and the IMU, and realize the precise fusion of the camera, lidar and IMU data in a unified coordinate system. The specific process is as follows: Figure 3 As shown;

[0102] Camera intrinsic parameter calibration: Use the OpenCV calibration tool to generate a checkerboard calibration plate pattern. Based on this, use a monocular camera to observe the calibration plate image, establish the optical projection function of the calibration plate corner points, and use Zhang Zhengyou camera calibration method to solve the camera's intrinsic parameter matrix and distortion coefficient;

[0103] Camera and LiDAR extrinsic parameter calibration: Use the Autoware open-source calibration tool to estimate the extrinsic parameters between a single camera and LiDAR. Use a calibration plate or 3D target with spatial features, simultaneously acquire image and point cloud data, extract the characteristic corner points corresponding to the calibration plate or 3D target, establish an observation constraint function for the characteristic corner points, and solve the change matrix between the camera and LiDAR to achieve camera and LiDAR extrinsic parameter calibration.

[0104] Camera and IMU extrinsic calibration: The IMU and camera can be calibrated using the Kalibr tool and motion registration. The IMU is rigidly mounted on a camera bracket, and a segment of data containing rich motion information is simultaneously recorded. The angular velocity / acceleration data of the IMU and the camera's position change are extracted, and the extrinsic parameters from the IMU to the camera are solved through optimization.

[0105] After the IMU and camera are calibrated, the transformation relationship from IMU to LiDAR can be indirectly obtained by knowing the external parameters from the camera to LiDAR, thereby completing the spatial joint calibration of the camera, LiDAR, and IMU.

[0106] Step 1-2: Reconstruct the lidar point cloud using the timestamp of the camera image to achieve time synchronization between the camera image and the lidar point cloud; propose a method for fine-grained reconstruction of the lidar scan data based on the camera image sampling time, such as Figure 4 As shown in the figure, taking a camera frequency of 30Hz and a lidar frequency of 10Hz as an example, the continuous scanning characteristics of the rotating lidar are utilized to divide each lidar scan frame into three scan segments according to the camera frequency. The scan segments are then spliced ​​in time sequence to generate a high-frequency reconstructed scan aligned with the camera time. After reconstruction, the lidar point cloud acquisition frequency is increased to three times the original frequency and kept synchronized with the camera image acquisition frequency.

[0107] Step 2: Pre-integrate the IMU measurements between adjacent image frames to obtain the IMU short-term prediction value. Compare the relative changes of the IMU short-term prediction value with the nearest neighbor IMU measurement value of the two visual image frames to establish the IMU pre-integration error model.

[0108] IMU pre-integration value between image frame i and image frame j:

[0109]

[0110]

[0111] In the formula They are the rotation component, velocity component and translation component of the IMU pre-integrated quantity respectively; R i and R j are the rotation matrices of the IMU measurements at the closest moments of image frame i and image frame j respectively; similarly, v i and v j Indicates the speed measured by IMU, p i and p j represents the translation measured by the IMU; Δt represents the time interval between image frame i and image frame j; k represents the sequence number of the IMU measurement between image frame i and image frame j; w k and a k are the angular velocity and translational acceleration measured by the IMU at the kth frame; g represents the gravity vector; Δv ik , ΔR ik They represent the velocity change and rotation change measured by the IMU at the kth frame relative to the ith frame respectively; and They refer to the zero bias of the accelerometer and gyroscope in the IMU at image frame i, respectively, and are usually modeled as random noise;

[0112] The short-term prediction value of IMU pre-integration in the lidar or camera coordinate system is:

[0113]

[0114] The superscript O in the formula represents the state in the laser radar or camera coordinate system; They are the rotation component, velocity component and translation component of the IMU pre-integration in the laser radar or camera coordinate system; is the rotation matrix of the external parameters between the lidar or camera coordinate system and the IMU coordinate system;

[0115] IMU pre-integration error model Expressed as:

[0116]

[0117] wherein, respectively represent the components of the IMU pre-integration error model on rotation, velocity, translation, accelerometer bias and gyroscope bias; subscript I ij represents the IMU measurement; superscript g represents the state quantity in the world coordinate system; represents the rotation matrix of the IMU measurement in the global coordinate system of the i-th frame; and respectively represent the IMU velocity quantity in the global coordinate system of the i-th frame and the j-th frame; and respectively represent the IMU translation quantity in the global coordinate system of the i-th frame and the j-th frame; X i , X j are the poses of the i-th frame and the j-th frame, respectively;

[0118] Step 3, using the IMU measurement of step two to perform distortion removal processing on the laser point cloud, and using a fixed region point cloud smoothness calculation method to extract the features of the distortion-removed point cloud;

[0119] Step 3-1, IMU measurement removes point cloud distortion; since the laser radar completes a frame of scanning during platform motion, there is a difference in sampling time for different points, resulting in distortion of the point cloud; using the high-frequency acceleration and angular velocity data provided by the IMU, time and space compensation is performed on each laser point to unify it to the same reference coordinate system, thereby achieving distortion removal processing; using the IMU measurement, pre-integration result and short-term prediction value in the laser radar coordinate system, a continuous motion trajectory within the laser radar scanning time is obtained, and according to the linear interpolation method and the laser radar scanning point sampling time, a corresponding motion matrix is obtained; then the distortion points are converted to the current laser radar coordinate system through inverse transformation of the motion matrix, to form a unified time point cloud without distortion;

[0120] Step 3-2, point cloud feature extraction based on the smoothness of the near-neighbor region point cloud of a fixed measurement distance; smoothness estimation is performed on the distortion-removed point cloud, so as to distinguish the planar feature points in the point cloud; existing methods only use the 5 points before and after the target point in the horizontal direction to calculate the smoothness, and the change in laser measurement distance causes different scales of point cloud smoothness, so it is impossible to form a unified smoothness description in the environment scanning; the method limits the nearest points of the extracted feature points to a fixed region within a certain distance threshold, and the laser points that are closer have more near-neighbor points, and the points that are farther away search fewer near-neighbor points, so that the smoothness of all points in the current frame scanning is maintained at the same scale; the point cloud smoothness calculation is:

[0121]

[0122] wherein C(·) is a mathematical expression of the point cloud smoothness, subscript n l represents the serial number of the point in the current scanning point cloud; superscript L represents the observation target of the laser radar; Represents a lidar point The ranging range of express nth l The distance measurement range of the nearest neighbor points; r blind and r max They are the near blind area of ​​the return point and the maximum measurement distance; D i is the laser point in the i-th frame laser scanning A set of consecutive points of nearest neighbors; D i The number of consecutive points in the curve segment; H and h are the curve segment length and horizontal angular resolution respectively; in the subsequent feature extraction module, only points with fixed area curvature will be used for the second stage of feature extraction and tracking; in the present invention, H = 0.35m, h = 0.2°, r blind =2m,r max =25m;

[0123] On this basis, if the smoothness of the target point is greater than the preset threshold τ=5, the point is set as an edge point, otherwise it is set as a plane point; the present invention records the feature point set of the i-th laser radar scan as F i , the edge feature point set is recorded as The plane feature point set is recorded as The superscripts e and p represent edge features and plane features, respectively. There are few vertical landmarks in the star map environment, so the ground feature point cloud mainly composed of plane points is extracted.

[0124] Step 4: The camera captures the image, extracts the visual corner features, and projects the laser point cloud of the continuous frames in step 3 onto the image to complete the depth value association between the laser point cloud and the visual corner features. The depth value association points are then filtered out using the reprojection consistency verification method across time-series images.

[0125] Step 4-1: The camera captures the image and uses the Shi-Tomasi algorithm to quickly extract sparse corner features with large grayscale gradients in the image;

[0126] Step 4-2: Depth value association between laser point cloud and visual corner features, such as Figure 5The multi-frame laser radar scanning data in the continuous time window is fused into a dense point cloud under the same coordinate system; through the external parameter transformation matrix between the laser radar and the camera, the three-dimensional point cloud is projected into the camera coordinate system; then the point cloud is further projected to the image plane by using the camera intrinsic parameter matrix, forming a dense depth point distribution; then, the projected point cloud is down-sampled and indexed in a two-dimensional K-D tree manner; all points are projected onto a unit sphere with the camera center as the origin, and each point is represented by its polar coordinate angle for accelerated search; the visual features extracted from the image are back-projected onto the unit sphere, and the three nearest neighbor laser radar points are searched on the sphere; by checking the geometric consistency of the three points in three-dimensional space, the validity of the association is judged, and the occlusion or cross-object situation is excluded; for the features that pass the validity check, a line of sight from the camera center to the visual feature is constructed, and the intersection of the line of sight and the plane formed by the three radar points is calculated, and the intersection distance is the depth value of the visual feature; for the point features without successful association depth, a supplementary estimation based on the parallax information is adopted;

[0127] Step 4-3, depth value outlier elimination based on cross-time image re-projection consistency verification method; extract observable same-name three-dimensional feature points in continuous multiple image frames and record their initial depth assignment; re-project these three-dimensional points to other frame image planes and calculate the re-projection error between them and the real image feature points; for each group of simultaneously observable three-dimensional points, the feature points in the frame are projected to the frame through the relative pose transformation to evaluate whether the projection error falls within the preset threshold; if the error is out of limit, the depth value corresponding to the frame is judged as abnormal; all depth assignments form a Gaussian distribution centered on the average depth, and by removing the abnormal projection points that exceed the projection error threshold, the feature points with good re-projection consistency are retained; if the number of valid depths of a certain feature is less than the minimum threshold, the depth is not used for fusion; if the observation number requirement is met, the average value of the depth after removing the outliers is used as the final depth result;

[0128] Step 5, based on the extraction of visual corner point features, KLT optical flow tracking method is used to match the corner points in adjacent image frames to obtain stable tracking feature point pairs; the three-dimensional feature points tracked in the previous frame are projected into the current frame image coordinate system based on camera pose estimation, and are corresponded with the visual corner point features extracted in the current frame; a visual re-projection error model between the projected points and the actual observed points is established:

[0129]

[0130] where r c (·) is the mathematical expression of the point feature projection model; and indicates the observation of the cth 3D point in the ith frame and the jth frame, is 2D corner features projected on the i-th image plane; π(·) is the optical projection function, which projects image features from the camera coordinate system to the image plane; b1 and b2 are two orthogonal bases of the tangent plane;

[0131] Step 6, nearest neighbor point matching is performed on the point cloud features in step three, and a point-to-random surface point cloud matching error model is constructed; in order to solve the problem of irregular surface in the unstructured environment of the star table and the invalidity of the traditional plane hypothesis, a point-to-random surface Mahalanobis distance model is proposed to improve the matching accuracy between laser point clouds; the point-to-random surface distance model is as shown in Figure 6 The local point cloud neighborhood is modeled as a Gaussian surface with a mean value and uncertainty, and the residual error of the point and the surface is expressed in the form of Mahalanobis distance, so as to realize fine modeling and registration of the non-flat surface;

[0132] For the target point in the i-th frame of laser scanning, its neighborhood N a laser points are extracted; due to the existence of laser radar measurement noise and cumulative error, the position of the target point has uncertainty, denoted as an uncertainty matrix The uncertainty matrix is the inverse of the covariance matrix W, and the covariance matrix can be calculated as:

[0133]

[0134] The covariance matrix can be calculated from the difference between the neighboring points and the Gaussian mean value, wherein The Gaussian mean value formed by the neighborhood is usually set to the three-dimensional position of the target point; is a summation symbol, and n' represents the serial number of the neighboring point, q n′ is the neighboring point of ; the inverse matrix W -1 characterizes the uncertainty of the point to the Gaussian surface, and is defined as the weight matrix of the Mahalanobis distance measurement;

[0135] For the target point i from the i-th frame of local map M , the subscript ma represents the neighboring point in the map, and the matching error of can be expressed as Then the residual error between the point cloud features in the i-th frame of scanning and the target random surface can be defined as:

[0136]

[0137] where r pg(·) is the mathematical representation of the point-to-random surface Mahalanobis distance; N is the number of points to be matched in the i-th frame scan; n represents the serial number of the point to be matched;

[0138] the inverse matrix of the covariance W -1 is composed of the eigenvector matrix N pg and the diagonal matrix Λ pg ; therefore, by decomposing the matrix W -1 , the point-to-random surface distance is converted to:

[0139]

[0140] where diag(·) is the mathematical representation of the diagonal matrix, λ1, λ2, λ3 are the descending order eigenvalues of the matrix W -1 , and ε1, ε2, ε3 are the eigenvectors corresponding to the eigenvalues λ1, λ2, λ3; the superscript T is the transpose operation of the matrix and the vector;

[0141] Then, the point-to-random surface distance model is converted to the point-to-surface cost function, which is the sum of the directionally weighted Euclidean distance squares:

[0142]

[0143] which represents the weighted projection error on multiple direction vectors, and is used to measure the spatial deviation between the matched points and the random surface;

[0144] Step 7, combine the IMU pre-integrated error model in step 2, the visual re-projection error model in step 5, and the point cloud matching error model of the point-to-random surface in step 6 to build the target function of the multi-modal information fusion odometer:

[0145]

[0146] where N s is the set of frame serial numbers; O c is the set of visual observation feature points; r0 is the prior error of the initial state; represents the square of the 2-norm; is the value of the independent variable X i that makes the function reach the minimum value;

[0147] By using the factor graph optimization method to solve the target function, the catalog patroler pose at the i-th frame time can be obtained.

[0148] Step 8, repeat steps 2 to 7 to estimate the continuous frame pose transformation and complete the construction of the multi-modal information fusion odometer.

[0149] In summary, the multi-modal information fusion odometry method for star catalog positioning provided in this example fuses monocular camera, lidar and inertial measurement unit (IMU) data to achieve high-precision and robust pose estimation. First, through joint extrinsic calibration of lidar-camera-IMU and reconstruction of lidar scans based on image timestamps, time and space synchronization of multiple source sensors is completed. Subsequently, for the IMU observation information between consecutive image frames and point cloud frames, a pre-integration error model is constructed, and the IMU predicted motion state is used to achieve laser point cloud de-warping, while robust point cloud geometric features are extracted based on the smoothness of the near-neighbor region with a fixed measurement distance. On this basis, the time-series laser point cloud is projected onto the image plane to complete the depth value association with the visual corner point features, and a re-projection consistency verification mechanism across time-series images is introduced to eliminate outliers. For star catalog unstructured terrain features, a point-to-random surface Mahalanobis distance model is further constructed for point cloud matching to achieve accurate point cloud registration under non-flat ground conditions. Finally, the IMU pre-integration error, visual re-projection error and laser point cloud registration error are combined to construct a unified multi-modal optimization objective function, and a graph optimization method is used to continuously estimate and optimize the pose of the rover, thereby significantly improving the positioning accuracy and system robustness in complex environments.

[0150] Embodiment two:

[0151] The present application provides a kind of multi-modal information fusion odometry construction system for star catalog positioning, for realizing the multi-modal information fusion odometry construction method for star catalog positioning described in the first aspect, it includes:

[0152] Sensor space-time synchronization module is used for the space joint calibration of monocular camera, lidar and inertial measurement unit, and receives camera image data, lidar point cloud data and inertial measurement data, reconstructs laser radar point cloud using the timestamp of camera image, realizes the time synchronization of camera image and laser radar point cloud;

[0153] First construction module is used for the frame processing of camera image, obtains continuous visual image frame, carries out pre-integration to the IMU measurement value between adjacent two image frames in continuous visual image frame, obtains IMU short-term prediction value, the relative change of IMU short-term prediction value and the nearest neighbor IMU measurement value of two image frames is compared, and IMU pre-integration error model is established;

[0154] Feature extraction module is used for laser point cloud motion compensation de-warping using IMU pre-integration result, and the geometric features of de-warping laser point cloud are extracted based on the smoothness calculation method of the near-neighbor region with fixed measurement distance;

[0155] The depth value association module is used to extract visual corner features from camera images, project the dedistorted continuous frame point cloud into the camera image coordinate system, associate the geometric features of the laser point cloud with the depth values ​​of the visual corner features, and filter outliers from the associated depth values ​​by verifying the consistency of reprojection across time-series images.

[0156] The second construction module is used to match adjacent frame images using a descriptor algorithm based on the visual corner feature to obtain the corresponding tracking feature, project the tracking feature to the image coordinate system where the current visual corner feature is located, and establish a visual reprojection error model between the projection point and the visual corner point;

[0157] The third construction module is used to perform nearest neighbor matching on the geometric features of the point cloud and build a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface;

[0158] The multimodal fusion positioning module is used to combine the IMU pre-integration error model, the visual reprojection error model, and the point cloud matching error model from point to random surface to construct the objective function of the multimodal information fusion odometry and calculate the position and pose of the rover through the incremental graph optimization solver.

[0159] The iterative optimization module is used for repeated iterations to achieve continuous pose transformation estimation and complete the construction of a multimodal information fusion odometry.

[0160] Example 3:

[0161] This embodiment provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, the multimodal information fusion odometry construction method for star catalog positioning described in the first aspect is adopted.

[0162] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device can also include input and output devices, network access devices and buses, etc.

[0163] Furthermore, the processor may adopt a central processing unit (CPU). Of course, depending on the actual usage, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. may also be adopted. The general-purpose processor may adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0164] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended that the scope of the application be limited to the detailed description contained herein or the specific examples given herein, but rather that the scope of the application be determined by the appended claims, and their equivalents.

[0165] To those skilled in the art, the above-mentioned terms in the present application can be understood according to the specific meaning in the specific context. When an element is referred to as being "assembled to", "mounted to", "fixed to" or "disposed to" another element, it can be directly on the other element or there can be an intermediate element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there can be an intermediate element. The terms "vertical", "horizontal", "upper", "lower", "left", "right", and similar expressions used herein are for illustrative purposes only and are not intended to be the only implementation.

[0166] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

[0167] In the description of the specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Claims

1. A method for constructing a multimodal information fusion odometry for star catalog positioning, characterized in that: The method is applied to a star chart cruiser equipped with a monocular camera, a lidar, and an inertial measurement unit, and includes the following steps: S1. Perform spatial joint calibration of the monocular camera, lidar, and inertial measurement unit, receive camera image data, lidar point cloud data, and inertial measurement data, reconstruct the lidar point cloud using the camera image timestamp, and achieve time synchronization between the camera image and the lidar point cloud. S2. Frame the camera image to obtain continuous visual image frames, pre-integrate the IMU measurement values ​​between two adjacent image frames in the continuous visual image frames, obtain the IMU short-term prediction value, compare the relative change of the IMU short-term prediction value with the nearest neighbor IMU measurement value of the two image frames, and establish an IMU pre-integration error model; S3. Using the IMU pre-integration result of step S2, the laser point cloud is subjected to motion compensation dedistortion, and the geometric features of the dedistorted laser point cloud are extracted based on the neighborhood smoothness calculation method with a fixed measurement distance; S4. Extract visual corner features from the camera image, project the dedistorted continuous frame point cloud output from step S3 into the camera image coordinate system, associate the geometric features of the laser point cloud with the depth values ​​of the visual corner features, and perform outlier filtering on the associated depth values ​​by verifying the consistency of reprojection across time-series images; S5. Based on the visual corner feature in step S4, use the descriptor algorithm to match adjacent frame images to obtain the corresponding tracking feature, project the tracking feature to the image coordinate system where the current visual corner feature is located, and establish a visual reprojection error model between the projected point and the visual corner point; S6. Perform nearest neighbor matching on the geometric features of the point cloud in step S3 and construct a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface; S7. Combine the IMU pre-integration error model in step S2, the visual reprojection error model in step S5, and the point-to-random surface point cloud matching error model in step S6 to construct the objective function of the multimodal information fusion odometry, and calculate the position and pose of the rover using the incremental graph optimization solver. S8. Repeat steps S2 to S7 to achieve continuous pose transformation estimation and complete the construction of the multimodal information fusion odometry.

2. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 1, characterized in that: In step S1, spatial joint calibration is performed on the monocular camera, lidar, and inertial measurement unit. The camera image data, lidar point cloud data, and inertial measurement data are received. The lidar point cloud is reconstructed using the timestamp of the camera image to achieve time synchronization between the camera image and the lidar point cloud. The details are as follows: Step S1-1. Implement the intrinsic calibration of the monocular camera, the extrinsic calibration of the camera and lidar, and the extrinsic calibration of the camera and IMU respectively, and achieve accurate fusion of camera, lidar, and IMU data in a unified coordinate system, as follows: Camera intrinsic parameter calibration: Use the OpenCV calibration tool to generate a checkerboard calibration plate pattern. Based on this, use a monocular camera to observe the calibration plate image, establish the optical projection function of the calibration plate corner points, and use Zhang Zhengyou camera calibration method to solve the camera's intrinsic parameter matrix and distortion coefficient; Camera and LiDAR extrinsic parameter calibration: Use the Autoware open-source calibration tool to estimate the extrinsic parameters between a single camera and LiDAR. Use a calibration plate or 3D target with spatial features, simultaneously acquire image and point cloud data, extract the characteristic corner points corresponding to the calibration plate or 3D target, establish an observation constraint function for the characteristic corner points, and solve the change matrix between the camera and LiDAR to achieve camera and LiDAR extrinsic parameter calibration. External parameter calibration of the camera and IMU: The IMU and camera are calibrated using the Kalibr tool and motion registration method. The IMU is rigidly mounted on the camera bracket, and a segment of data containing motion information is synchronously recorded. The angular velocity and acceleration data of the IMU and the camera's position change are extracted, and the external parameters from the IMU to the camera are solved through optimization. After the IMU and camera are calibrated, the transformation relationship from IMU to LiDAR is indirectly obtained by knowing the external parameters from the camera to LiDAR, thereby completing the spatial joint calibration of the camera, LiDAR and IMU. Step S1-2. Reconstruct the LiDAR point cloud using the timestamp of the camera image to achieve time synchronization between the camera image and the LiDAR point cloud, as follows: Taking advantage of the continuous scanning characteristics of the rotating lidar, each frame of the lidar scan is divided into multiple scanning segments according to the camera frequency, and the scanning segments are spliced ​​in time series to generate a high-frequency reconstructed scan aligned with the camera time. After reconstruction, the lidar point cloud acquisition frequency is improved and kept synchronized with the camera image acquisition frequency.

3. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 2, characterized in that: In step S2, the camera image is framed to obtain continuous visual image frames. The IMU measurement values ​​between two adjacent image frames in the continuous visual image frames are pre-integrated to obtain the IMU short-term prediction value. The relative changes of the IMU short-term prediction value and the nearest neighbor IMU measurement values ​​of the two image frames are compared to establish the IMU pre-integration error model, which is as follows: Step S2-1. The IMU pre-integration between two adjacent image frames i and j is as follows: In the formula They are the rotation component, velocity component and translation component of the IMU pre-integrated quantity respectively; R i and R j are the rotation matrices of the IMU measurements closest to image frame i and image frame j respectively; v i and v j Indicates the speed measured by IMU, p i and p j represents the translation measured by the IMU; Δt represents the time interval between image frame i and image frame j; k represents the sequence number of the IMU measurement between image frame i and image frame j; w k and a k are the angular velocity and translational acceleration measured by the IMU at the kth frame; g represents the gravity vector; Δv ik , ΔR ik They represent the velocity change and rotation change measured by the IMU at the kth frame relative to the ith frame respectively; and are the zero bias of the accelerometer and gyroscope in the IMU at image frame i, respectively, which are modeled as random noise; Step S2-2. The short-term prediction value of the IMU pre-integration in the lidar or camera coordinate system is: The superscript O in the formula represents the state in the laser radar or camera coordinate system; They are the rotation component, velocity component and translation component of the IMU pre-integration in the laser radar or camera coordinate system; is the rotation matrix of the external parameters between the lidar or camera coordinate system and the IMU coordinate system; IMU pre-integration error model Expressed as: in, Respectively represent the components of the IMU pre-integration error model in rotation, velocity, translation, accelerometer bias, and gyroscope bias; subscript I ij Represents IMU measurement; the superscript g represents the state quantity in the world coordinate system; Represents the rotation matrix of the IMU measurement in the global coordinate system of the i-th frame; and Represent the IMU velocity in the global coordinate system of the i-th frame and the j-th frame respectively; and Respectively represent the IMU translation in the global coordinate system of the i-th frame and the j-th frame; X i , X j are the poses of the i-th frame and the j-th frame respectively.

4. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 3, characterized in that: In step S3, the laser point cloud is subjected to motion compensation dedistortion using the IMU pre-integration result of step S2, and the geometric features of the dedistorted laser point cloud are extracted based on the neighborhood smoothness calculation method with a fixed measurement distance, as follows: Step S3-1. Using the high-frequency acceleration and angular velocity data provided by the IMU, perform spatiotemporal compensation on each laser point, unifying them to the same reference coordinate system to achieve dedistortion. The continuous motion trajectory within the lidar scan time is obtained using the IMU measurements and pre-integration results from step S2, along with the short-term predictions in the lidar coordinate system. The corresponding motion matrix is ​​obtained based on the linear interpolation method and the lidar scan point sampling time. The distorted points are then converted to the current lidar coordinate system through the inverse transformation of the motion matrix, forming an undistorted point cloud at the same time. Step S3-2. Extracting point cloud features based on the smoothness of the point cloud in the neighboring area at a fixed measurement distance; Smoothness estimation of the dedistorted point cloud is performed to distinguish the planar feature points in the point cloud; The points adjacent to the feature points to be extracted are limited to a fixed area with a certain distance threshold. Laser points that are close have more neighboring points, while points that are far away have fewer neighboring points. This keeps the smoothness of all points in the current frame scan at the same scale. The point cloud smoothness is calculated as: Where C(·) is the mathematical expression of point cloud smoothness, subscript n l Indicates the serial number of the point in the current scan point cloud; the superscript L indicates the observation target of the lidar; Indicates the inth lidar point The ranging range; express nth l The distance measurement range of the nearest neighbor points; r blind and r max They are the near blind area of ​​the return point and the maximum measurement distance; D i is the nth laser point in the i-th frame laser scan The set of consecutive points of the nearest neighbors; |D i |=180H / πhr n D i The number of consecutive points in the curve; H and h are the curve segment length and horizontal angular resolution respectively; On this basis, if the smoothness of the target point is greater than the preset threshold τ = 5, the point is set as an edge point, otherwise it is set as a plane point; The feature point set of the i-th lidar scan is recorded as F i , the edge feature point set is recorded as The plane feature point set is recorded as The superscripts e and p represent edge features and plane features, respectively.

5. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 4, characterized in that: In step S4, the visual corner features of the camera image are extracted, the dedistorted continuous frame point cloud output from step S3 is projected into the camera image coordinate system, the depth values ​​of the laser point cloud geometric features and the visual corner features are associated, and the associated depth values ​​are filtered out by verifying the consistency of reprojection across time-series images, as follows: Step S4-1. Use the Shi-Tomasi algorithm to quickly extract sparse corner features with large grayscale gradients in the image; Step S4-2: Depth value association between laser point cloud and visual corner feature: The multi-frame LiDAR scan data within a continuous time window is fused into a dense point cloud in the same coordinate system. The 3D point cloud is projected to the camera coordinate system using the extrinsic parameter transformation matrix between the LiDAR and the camera. The point cloud is then further projected onto the image plane using the camera intrinsic parameter matrix to form a dense depth point distribution. The projected point cloud is then downsampled and indexed using a two-dimensional KD tree. All points are projected onto a unit sphere with the camera center as the origin, and each point is represented by its polar coordinate angle to accelerate the search. The visual features extracted from the image are back-projected onto the unit sphere, and the three nearest neighbor lidar points are searched on the sphere. The geometric consistency of these three lidar points in three-dimensional space is checked to determine the validity of the association, eliminating occlusion or cross-object situations. For visual features that pass the validity check, the line of sight from the camera center to the visual feature is constructed, and its intersection with the plane formed by the three lidar points is calculated. The distance between the intersection points is the depth value of this visual feature. For point features that are not successfully associated with depth, triangulation based on disparity information is used for supplementary estimation; Step S4-3: Depth value outlier filtering based on the reprojection consistency verification method across time-series images: Extract the 3D feature points with the same name that can be observed in multiple consecutive image frames, record their initial depth assignments, reproject these 3D points to the image planes of other frames, and calculate the reprojection error between them and the real image feature points; For each set of image frames that can simultaneously observe 3D points, the feature points in one frame are projected onto other frames through relative pose transformation, and the projection error is evaluated to see if it falls within a preset threshold. If the error exceeds the limit, the depth value corresponding to the current frame is judged to be abnormal. All depth assignments form a Gaussian distribution centered on the average depth. By removing abnormal projection points that exceed the projection error threshold, feature points with good reprojection consistency are retained. If the effective depth number of a feature is lower than the minimum threshold, this depth will not be used for fusion; if the observation quantity requirement is met, the average depth after removing the outliers is used as the final depth result.

6. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 5, characterized in that: In step S5, based on the visual corner feature in step S4, the descriptor algorithm is used to match adjacent frame images to obtain the corresponding tracking feature, and the tracking feature is projected to the image coordinate system where the current visual corner feature is located. A visual reprojection error model is established between the projection point and the visual corner point, as follows: Based on the extracted visual corner features, the KLT optical flow tracking method is used to match the corner points in adjacent image frames to obtain stable tracking feature point pairs. The 3D feature points tracked in the previous frame are projected into the current frame image coordinate system based on the camera pose estimation and correspond to the visual corner features extracted in the current frame. The visual reprojection error model between the projection point and the actual observation point is established as follows: where r c (·) is the mathematical expression of the point feature projection model; and Refers to observing the cth 3D point in the i-th frame and the j-th frame, yes The 2D corner feature obtained by projecting onto the image plane of the i-th frame; π(·) is the optical projection function, which is used to project the image feature points from the camera coordinate system to the image plane; b1 and b2 are Two orthogonal bases of tangent planes.

7. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 6, characterized in that: In step S6, the geometric features of the point cloud in step S3 are matched with the nearest neighbor points, and a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface is constructed, as follows: Step S6-1. Model the local point cloud neighborhood as a Gaussian surface with mean and uncertainty, and express the residual error between the point and the surface in the form of Mahalanobis distance, so as to achieve fine modeling and registration of non-flat surfaces in the unstructured environment of the star catalog; For the target point in the laser scan of the i-th frame Extract N in its neighborhood a laser points, due to the existence of laser radar measurement noise and cumulative error, The position of is uncertain, which is recorded as the uncertainty matrix The uncertainty matrix is ​​the inverse matrix of the covariance matrix W, which can be calculated as: The covariance matrix is ​​calculated from the difference between the neighboring points and the Gaussian mean, where The Gaussian mean of the neighborhood is usually set to The three-dimensional position of is the summation symbol, n′ represents The sequence number of the neighbor point, q n′ for The neighboring points of -1 Characterizes the uncertainty of the point to the Gaussian surface, defined as the weight matrix of the Mahalanobis distance metric; Step S6-2. For the local map M from the i-th frame i Points to be registered The subscript ma represents the neighboring points in the map, which is The matching error is expressed as Then the residual between the point cloud feature and the target random surface in the i-th frame scan is defined as: where r pg (·) is the mathematical representation of the Mahalanobis distance from a point to a random surface; N is the number of points to be matched in the i-th frame scan; The inverse covariance matrix W -1 By the eigenvector matrix N pg and the diagonal matrix Λ pg Therefore, by decomposing the matrix W -1 , convert the distance from a point to a random surface into: Where diag(·) is the mathematical representation of a diagonal matrix, λ1,λ2,λ3 are the matrices W -1 The descending eigenvalues ​​of , ε1, ε2, ε3 are the eigenvectors corresponding to the eigenvalues ​​λ1, λ2, λ3; the superscript T is the transpose operation of the matrix and vector; Step S6-3. Convert the point-to-random surface distance model into a point-to-surface cost function, which is the sum of squared Euclidean distances weighted by direction: That is, it is expressed as a weighted projection error on multiple direction vectors, which is used to measure the degree of spatial deviation between the matching point and the random surface.

8. The method for constructing a multimodal information fusion odometer for star catalog positioning according to claim 7, characterized in that: In step S7, the objective function of the multimodal information fusion odometry is constructed as follows: where N s is a set of frame numbers; c is the set of visual observation feature points; r0 is the prior error of the initial state; represents the square of the 2-norm; The independent variable X that makes the function reach its minimum value i The value of By solving the objective function using factor graph optimization, the position of the star rover at the i-th frame can be obtained.

9. A multimodal information fusion odometer construction system for star chart positioning, used to implement a multimodal information fusion odometer construction method for star chart positioning according to any one of claims 1 to 8, characterized in that: include: The sensor spatiotemporal synchronization module is used to perform spatial joint calibration for the monocular camera, lidar, and inertial measurement unit. It receives camera image data, lidar point cloud data, and inertial measurement data, reconstructs the lidar point cloud using the camera image timestamp, and achieves time synchronization between the camera image and the lidar point cloud. The first building module is used to perform frame processing on the camera image to obtain continuous visual image frames, pre-integrate the IMU measurement values ​​between two adjacent image frames in the continuous visual image frames, obtain the IMU short-term prediction value, compare the relative change of the IMU short-term prediction value with the nearest neighbor IMU measurement values ​​of the two image frames, and establish an IMU pre-integration error model; The feature extraction module is used to perform motion compensation dedistortion on the laser point cloud using the IMU pre-integration results, and extract the geometric features of the dedistorted laser point cloud based on the neighborhood smoothness calculation method with a fixed measurement distance; The depth value association module is used to extract visual corner features from camera images, project the dedistorted continuous frame point cloud into the camera image coordinate system, associate the geometric features of the laser point cloud with the depth values ​​of the visual corner features, and filter outliers from the associated depth values ​​by verifying the consistency of reprojection across time-series images. The second construction module is used to match adjacent frame images using a descriptor algorithm based on the visual corner feature to obtain the corresponding tracking feature, project the tracking feature to the image coordinate system where the current visual corner feature is located, and establish a visual reprojection error model between the projection point and the visual corner point; The third construction module is used to perform nearest neighbor matching on the geometric features of the point cloud and build a point cloud matching error model based on the Mahalanobis distance from the point to the Gaussian surface; The multimodal fusion positioning module is used to combine the IMU pre-integration error model, the visual reprojection error model, and the point cloud matching error model from point to random surface to construct the objective function of the multimodal information fusion odometry and calculate the position and pose of the rover through the incremental graph optimization solver. The iterative optimization module is used for repeated iterations to achieve continuous pose transformation estimation and complete the construction of a multimodal information fusion odometry.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method for constructing a multimodal information fusion odometer for star catalog positioning according to any one of claims 1 to 8 is adopted.

Images