Image Feature Matching Method and System for Multi-Sensor Fusion Navigation

Abstract

The present invention relates to the technical field of image feature analysis, and specifically includes an image feature matching method and system for multi-sensor fusion navigation. The method includes: using inertial navigation data to assist in calibrating image information and point cloud information to determine dynamic calibration features; capturing a continuous frame motion attitude compensation search area, extracting key feature points and projecting and matching them to generate an initial set; based on the dynamic calibration features, combining the dynamic calibration features with a joint optimization objective to iteratively update the external parameter matrix, and verifying to obtain an image feature matching set, which solves the technical problem that image feature matching is easily affected by motion blur and perspective changes, and the accuracy of image feature matching is limited. It realizes capturing the motion attitude between continuous frames, compensating the search area of image feature matching, reducing the image feature matching error caused by motion, combining the joint optimization objective, iteratively updating the external parameter matrix, gradually approaching the optimal solution, and improving the technical effects of image feature matching accuracy and stability.

Description

Technical Field

[0001] The present invention relates to the technical field of image feature analysis, and particularly to an image feature matching method and system for multi-sensor fusion navigation. Background Art

[0002] Multi-sensor image fusion refers to appropriately processing images of the same target or scene collected by multiple sensors to generate a new image that is more suitable for human eye perception or subsequent computer processing. Each sensor is designed to adapt to a specific environment and application range, and there are both redundancy and complementarity between images obtained by sensors with different characteristics or viewpoints. By fusing them, the reliability of the system and the utilization efficiency of image information can be effectively improved.

[0003] In navigation and positioning systems, by combining data from multiple sensors (such as visual sensors, inertial sensors, GPS, lidar, etc.), the deficiencies of a single sensor can be compensated, and the overall performance of the system can be improved. However, in terms of image feature matching, due to differences in the imaging principles, resolutions, viewpoints, etc. of different sensors, it is difficult to extract and match image features.

[0004] In summary, in the prior art, there are technical problems that image feature matching is easily affected by motion blur and viewpoint changes, and the accuracy of image feature matching is limited. Summary of the Invention

[0005] The present application provides an image feature matching method and system for multi-sensor fusion navigation, aiming to solve the technical problems in the prior art that image feature matching is easily affected by motion blur and viewpoint changes, and the accuracy of image feature matching is limited.

[0006] In view of the above problems, the technical solution of the present application is as follows:

[0007] On the one hand, the present application provides an image feature matching method for multi-sensor fusion navigation. The method includes: using inertial navigation data to assist in calibrating the image information uploaded by an image monitoring device to determine a first dynamic calibration feature; using inertial navigation data to assist in calibrating the point cloud information uploaded by a lidar to determine a second dynamic calibration feature; capturing the motion posture between consecutive frames to compensate the search area for image feature matching; extracting key feature points in the search area and performing projection matching with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set; based on the first dynamic calibration feature and the second dynamic calibration feature, combined with a joint optimization objective, iteratively update the extrinsic parameter matrix; using the updated extrinsic parameter matrix to verify the initial feature matching set to obtain an image feature matching set.

[0008] On the other hand, the present application provides an image feature matching system for multi-sensor fusion navigation. The system includes: an auxiliary calibration module for using inertial navigation data to assist in calibrating the image information uploaded by an image monitoring device to determine a first dynamic calibration feature; using inertial navigation data to assist in calibrating the point cloud information uploaded by a lidar to determine a second dynamic calibration feature; a motion attitude capture module for capturing the motion attitude between consecutive frames to compensate the search area for image feature matching; a projection matching module for extracting key feature points in the search area and performing projection matching with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set; an iterative update module for iteratively updating the external parameter matrix based on the first dynamic calibration feature, the second dynamic calibration feature, and in combination with a joint optimization objective; and a verification module for using the updated external parameter matrix to verify the initial feature matching set to obtain an image feature matching set.

[0009] In summary, one or more technical solutions provided in the present application achieve capturing the motion attitude between consecutive frames, reducing the image feature matching error caused by motion, and further iteratively updating the external parameter matrix through the first dynamic calibration feature and the second dynamic calibration feature, in combination with a joint optimization objective, to gradually approach the optimal solution, improving the technical effects of image feature matching accuracy and stability. Description of the Drawings

[0010] Figure 1 is a schematic flowchart of an image feature matching method for multi-sensor fusion navigation provided by the present application;

[0011] Figure 2 is a schematic structural diagram of an image feature matching system for multi-sensor fusion navigation provided by the present application.

[0012] Description of the reference numerals: auxiliary calibration module M100, motion attitude capture module M200, projection matching module M300, iterative update module M400, verification module M500. Detailed Description of the Embodiments

[0013] Embodiment 1

[0014] The present application will be specifically described below with reference to the drawings. As Figure 1 shown, the present application provides an image feature matching method for multi-sensor fusion navigation. The method includes:

[0015] S1: Using inertial navigation data to assist in calibrating the image information uploaded by an image monitoring device to determine a first dynamic calibration feature; using inertial navigation data to assist in calibrating the point cloud information uploaded by a lidar to determine a second dynamic calibration feature; S2: Capturing the motion attitude between consecutive frames to compensate the search area for image feature matching.

[0016] Specifically, inertial navigation data refers to the information about the movement of an object provided by an inertial measurement unit (IMU), including parameters such as acceleration and angular velocity, which can reflect the movement state and attitude changes of the object; auxiliary calibration refers to using inertial navigation data to calibrate the output data of image monitoring devices and lidars to ensure the spatio-temporal consistency between different sensor data; the dynamic calibration feature refers to adjusting and optimizing the calibration results of sensor data in real time through inertial navigation data in a dynamic environment to adapt to environmental changes; capturing the movement attitude between consecutive frames refers to extracting the attitude change information of an object during movement by analyzing consecutive image frames or point cloud data.

[0017] In the process of multi-sensor fusion for mine tunneling navigation, image monitoring devices and lidars respectively provide two-dimensional images and three-dimensional point cloud data. However, these data may be inconsistent in time and space. By using inertial navigation data to assist in calibrating image information and point cloud information, the data of different sensors can be aligned to the same coordinate system, thereby determining the first dynamic calibration feature and the second dynamic calibration feature, effectively solving the spatio-temporal deviation problem between sensors and improving the accuracy of data fusion.

[0018] At the same time, capturing the movement attitude between consecutive frames and compensating the search area for image feature matching can dynamically adjust the range of feature matching and reduce the matching errors caused by motion blur or perspective changes. For example, in the mine tunneling navigation scenario, the movement of mine tunneling equipment causes the offset of image feature points. By capturing the movement attitude and compensating the search area, it can ensure that feature points can still be accurately matched in a dynamic environment, thereby improving the accuracy and stability of the mine tunneling navigation system.

[0019] S3: Extract key feature points in the search area and perform projection matching with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set; S4: Based on the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization objective, iteratively update the extrinsic parameter matrix; S5: Use the updated extrinsic parameter matrix to verify the initial feature matching set to obtain an image feature matching set.

[0020] Specifically, key feature points refer to points that are unique and stable in image or point cloud data, used to describe information such as the shape and position of an object, and are repeatable under different perspectives or times; projection matching refers to projecting the extracted key feature points from their original coordinate system (such as an image coordinate system or a lidar coordinate system) to the unified coordinate system of an inertial navigation system (INS), and finding the corresponding relationships of these points in different sensor data through a matching algorithm; the joint optimization objective refers to integrating the errors of multiple sensor data (such as the reprojection error of an image and the spatial matching error of a point cloud) into an optimization framework, and optimizing the external parameter matrix between sensors by minimizing these errors; the external parameter matrix refers to a matrix that describes the relative position and attitude between two sensors, used to convert the data of one sensor to the coordinate system of another sensor.

[0021] In the process of multi-sensor fusion for mine tunneling navigation, extracting key feature points is the basis for realizing the matching of image and point cloud data. By extracting key feature points in the search area and projecting them into the inertial navigation coordinate system, the data of different sensors can be unified into the same reference framework, ensuring the corresponding relationships of the feature points in different sensor data; based on the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization objective, the external parameter matrix is iteratively updated. By minimizing the image reprojection error and the point cloud spatial matching error, the relative position and attitude relationship between sensors are gradually optimized, thereby improving the accuracy of multi-sensor fusion. For example, in the mine tunneling navigation scenario, the movement of the mine tunneling equipment will cause dynamic changes in sensor data. By jointly optimizing the external parameter matrix, the deviation between sensors can be corrected in real time to ensure the accuracy of the navigation system. The initial feature matching set is verified using the updated external parameter matrix, the accuracy and consistency of the initial feature matching set are verified, and the feature points with incorrect matches are removed, further improving the reliability and accuracy of the matching result, and effectively reducing the matching errors caused by sensor errors or environmental changes.

[0022] Furthermore, the method of this application also includes:

[0023] Setting the joint optimization objective, fusing the reprojection error corresponding to the image monitoring device and the spatial matching error corresponding to the lidar; modeling the reprojection error term and the spatial matching error term in the joint optimization objective as factor nodes in a factor graph, performing factor graph optimization, jointly setting the state variables of each factor node, and setting the external parameter matrix.

[0024] Specifically, the joint optimization objective refers to integrating the errors of multiple sensor data (such as the reprojection error of the image monitoring device and the spatial matching error of the lidar) into a unified optimization framework, and optimizing the extrinsic matrix between sensors by minimizing the sum of these errors; the reprojection error refers to the deviation between the actual projection point and the theoretical projection point when projecting a three-dimensional point onto a two-dimensional image plane in the image monitoring device; the spatial matching error refers to the spatial deviation between the point cloud data and the reference point cloud or model in the lidar; a factor graph is a graph model used to represent the constraint relationships between variables, where nodes represent variables or constraints, and edges represent the relationships between variables. Factor graph optimization is an optimization method based on the graph model, which solves the optimal values of variables by minimizing the errors of all constraint terms. Among them, the factor node is the constraint node in the factor graph, which is used to represent the error terms in the optimization objective (such as reprojection error and spatial matching error), and the state variable refers to the variable to be optimized, such as the extrinsic matrix between sensors.

[0025] In the process of multi-sensor fusion for mine tunneling navigation, setting the joint optimization objective is a key step to improve the system accuracy. By fusing the reprojection error of the image monitoring device and the spatial matching error of the lidar into an optimization objective, the error characteristics of the two types of sensor data can be considered simultaneously, so as to more comprehensively optimize the extrinsic matrix between sensors. Further, the reprojection error term and the spatial matching error term are modeled as factor nodes in the factor graph, and the state variables of each factor node are jointly optimized through the factor graph optimization method. This process utilizes the efficient optimization characteristics of the factor graph and can quickly solve the optimal extrinsic matrix. For example, in the mine tunneling navigation scenario, the movement and environmental changes of the mine tunneling equipment will cause dynamic deviations in the sensor data. Through factor graph optimization, the extrinsic matrix can be adjusted in real time to ensure the consistency of the image and point cloud data in the inertial navigation coordinate system, thereby improving the accuracy and stability of the mine tunneling navigation system.

[0026] Furthermore, the method of the present application further includes:

[0027] Performing timestamp synchronization on the image monitoring device and the lidar, and setting a time synchronization mechanism; based on the time synchronization mechanism, performing interpolation processing on asynchronous data to generate a synchronized data stream, and the synchronized data stream includes inertial navigation data.

[0028] Specifically, timestamp synchronization refers to aligning the data collected by different sensors in chronological order to ensure consistency in the time dimension. Timestamp synchronization is the basis for multi-sensor data fusion because the sampling frequencies and data generation times of different sensors may vary. The time synchronization mechanism refers to the specific methods or strategies used to achieve timestamp synchronization, such as calibrating the time bases of sensors through hardware synchronization signals or software algorithms. Asynchronous data refers to the problem of data desynchronization caused by inconsistent sampling times of sensors. Interpolation processing refers to inserting new data points between known data points through mathematical methods to generate a temporally continuous data stream. A synchronized data stream refers to the data of different sensors being aligned in time after time synchronization processing, enabling processing and analysis based on a unified time base.

[0029] In a mine tunneling navigation system, image monitoring devices and lidars usually have different sampling frequencies and data generation times, which leads to data asynchronization problems. To ensure the consistency of data in the time dimension, it is necessary to perform timestamp synchronization on these sensors and set corresponding time synchronization mechanisms. Further, calibrate the time bases of the sensors through hardware or software methods to ensure that data can be recorded with a unified timestamp. For asynchronous data caused by different sampling frequencies, an interpolation processing method is used to generate a temporally continuous synchronized data stream. For example, the sampling frequency of the lidar may be higher than that of the image sensor. In this case, corresponding lidar data can be generated at the time points of the image data through interpolation, thereby generating a synchronized data stream containing inertial navigation data to ensure that the data of different sensors are aligned in time and effectively reduce the errors caused by data asynchronization, providing a reliable basis for subsequent feature extraction, matching, and fusion.

[0030] Furthermore, capture the motion postures between consecutive frames and compensate the search area for image feature matching. The method of the present application includes:

[0031] According to a preset sliding window, determine the image information and point cloud information from the th frame to the th frame, where N is a positive integer divisible by 2; based on the image information and point cloud information from the th frame to the th frame, determine the first motion transformation relationship feature and the second motion transformation relationship feature between adjacent frames; according to the first motion transformation relationship feature and the second motion transformation relationship feature, compensate the search area for image feature matching.

[0032] Specifically, a preset sliding window refers to a time or frame window with a fixed size set when processing continuous frame data, which is used to extract and analyze the image information and point cloud information within the window. The size of the sliding window is usually determined according to the specific application scenario and the sampling frequency of the sensor; the first motion transformation relationship feature and the second motion transformation relationship feature refer to the features describing the motion changes between adjacent frames extracted by analyzing the image information and point cloud information between continuous frames, such as rotation angle, translation vector, or motion trajectory, etc.; the search area for compensating image feature matching refers to dynamically adjusting the search range during image feature matching according to the extracted motion transformation relationship features, so as to reduce the offset or loss of feature points caused by motion and improve the matching efficiency and accuracy.

[0033] During the multi-sensor fusion mine tunneling navigation process, in order to process the dynamic changes between continuous frames, a sliding window method is used to extract and analyze the image and point cloud data within a certain time range. Further, according to the preset sliding window, determine the image information and point cloud information from the th frame to the th frame, where is a positive integer divisible by 2 for symmetric analysis; based on the data of continuous frames, further calculate the motion transformation relationship features between adjacent frames. For example, by comparing the image feature points of adjacent frames or the key points in the point cloud, the first motion transformation relationship feature and the second motion transformation relationship feature are extracted. The first motion transformation relationship feature and the second motion transformation relationship feature can reflect the attitude changes of the sensor during the motion process; according to these motion transformation relationship features, dynamically compensate the search area for image feature matching. For example, if it is detected that the sensor has a large translation or rotation during the motion process, the search area will be adjusted accordingly to ensure that the feature points can be correctly matched at the new position, reducing the matching errors caused by motion blur or perspective changes and improving the robustness and efficiency of feature matching.

[0034] Furthermore, based on the image information and point cloud information from the th frame to the th frame, determine the first motion transformation relationship feature between adjacent frames. The method of this application includes:

[0035] Based on the image information and point cloud information of the th frame and the th frame, determine the relative motion; based on the image information and point cloud information of the th frame and the th frame, determine the relative motion; traverse the image information and point cloud information from the th frame to the th frame, and pass through the Relative motion, the relative motion to the relative motion, determine the first motion transformation relationship feature.

[0036] Specifically, relative motion refers to the motion change of a sensor or a target object calculated by comparing image information or point cloud information between consecutive frames, including a translation vector and a rotation angle; the first motion transformation relationship feature refers to a comprehensive motion feature obtained by analyzing multiple relative motions (such as the relative motion, the relative motion,..., the relative motion), which is used to describe the overall motion trend and change law of the sensor within a sliding window.

[0037] In the process of mine tunneling navigation with multi-sensor fusion, to accurately capture the dynamic motion of the sensor, it is necessary to extract motion features by analyzing the relative motion between consecutive frames. Further, based on the image and point cloud information of the frame and the relative motion is calculated; based on the image and point cloud information of the frame and the relative motion is calculated; by traversing all the image and point cloud data from the frame to the frame, analyzing the image and point cloud data frame by frame from the frame to the frame, integrating all the relative motion information from to , determine the first motion transformation relationship feature, which can reflect the overall motion trend of the sensor within the entire sliding window, such as the average motion speed, the cumulative rotation angle, etc. For example, in the mine tunneling navigation scenario, the motion of the mine tunneling equipment causes the rapid translation and rotation of the sensor. By calculating and integrating the relative motion, the search area for image feature matching is dynamically adjusted, reducing the matching error caused by motion blur or perspective change.

[0038] Furthermore, based on the image information and point cloud information from the frame to the

[0039] frame, determine the second motion transformation relationship feature between adjacent frames. The method of this application includes: Based on the image information and point cloud information of the frame and the image information and point cloud information of the frame, determine the Relative motion; traversing the frame to the image information and point cloud information of the frame, and through the relative motion, the relative motion to the relative motion, determine the second motion transformation relationship feature.

[0040] Specifically, the relative motion and the relative motion refer to the motion changes of the sensor or the target object calculated by analyzing the image information and point cloud information between adjacent frames (such as the frame and the frame, the frame and the frame), including translation and rotation, etc.; the second motion transformation relationship feature refers to the comprehensive motion feature obtained by integrating the relative motions between multiple adjacent frames (such as the relative motion, the relative motion,..., the relative motion), which is used to describe another motion trend or change rule of the sensor within the entire sliding window.

[0041] In the process of multi-sensor fusion mine tunneling navigation, in order to capture the dynamic behavior of the sensor more comprehensively, it is necessary to analyze the motion changes between consecutive frames from different angles. Specifically, first, based on the frame and the frame of image and point cloud information, calculate the relative motion; then, based on the frame and the frame of image and point cloud information, calculate the relative motion; by traversing all the image and point cloud data from the frame to the frame, analyze the image and point cloud data frame by frame from the frame to the frame, integrate all the relative motion information from to , determine the second motion transformation relationship feature. This feature can provide another perspective on the sensor's motion. By calculating and integrating these relative motions, the sensor's state can be predicted more accurately, thereby dynamically adjusting the search area of image feature matching, reducing the matching error caused by motion blur or perspective change, improving the accuracy and efficiency of feature matching, and enhancing the adaptability of the mine tunneling navigation system to complex dynamic environments.

[0042] Distinctive, the first motion transformation relationship feature corresponds to each odd frame to the corresponding even frame of the same motion process; the second motion transformation relationship feature corresponds to each even frame to the corresponding odd frame of the same motion process. Preferably, to more comprehensively capture the dynamic behavior of the sensor, the motion transformation relationship between consecutive frames is analyzed from two directions, that is, from odd frames to even frames and from even frames to odd frames. Bidirectional analysis can provide more complete motion information and improve the accuracy and robustness of feature matching.

[0043] Furthermore, according to the first motion transformation relationship feature and the second motion transformation relationship feature, the search area for compensating image feature matching is as follows. The method of the present application includes:

[0044] Based on the first motion transformation relationship feature, the first motion transformation matrix from the th frame to the th frame is determined by cumulative motion transformation ; based on the second motion transformation relationship feature, the second motion transformation matrix from the th frame to the th frame is determined by cumulative motion transformation ; based on the preset sliding window, a dynamic search window is configured through the first motion transformation matrix and the second motion transformation matrix .

[0045] Specifically, cumulative motion transformation refers to sequentially combining the relative motion transformations between multiple consecutive frames to obtain the overall motion transformation from the initial frame to the target frame, which is achieved by matrix multiplication and is used to describe the overall motion state of the sensor over a period of time; the first motion transformation matrix is a matrix accumulated based on the first motion transformation relationship feature (from odd frames to even frames) and is used to describe the motion change from the th frame to the th frame; the second motion transformation matrix is a matrix accumulated based on the second motion transformation relationship feature (from even frames to odd frames) and is used to describe the motion change from the th frame to the th frame; the dynamic search window refers to the feature matching search area dynamically adjusted according to the motion state of the sensor, and its size and shape can be updated in real time according to the motion transformation matrix to adapt to the dynamic changes of the sensor.

[0046] In the process of mine tunneling navigation with multi-sensor fusion, accurate feature matching is the key to achieving high-precision positioning and navigation. Preferably, in order to more accurately compensate for the movement of the sensor and optimize the search area of feature matching, it is necessary to generate a motion transformation matrix based on the extracted motion transformation relationship features, and configure a dynamic search window accordingly. Further, based on the first motion transformation relationship feature (the relative motion from odd frames to even frames), by accumulating these relative motion transformations, calculate the overall motion transformation from the frame to the frame, and determine the first motion transformation matrix from the frame to the frame , where the first motion transformation relationship feature includes the frame to the frame corresponding relative motion, the frame to the frame corresponding relative motion to the frame to the frame corresponding relative motion. By multiplying these relative motions through matrix multiplication, the first motion transformation matrix is obtained; further, based on the second motion transformation relationship feature (the relative motion from even frames to odd frames), by accumulating these relative motion transformations, calculate the overall motion transformation matrix from the frame to the frame, and determine the second motion transformation matrix from the frame to the frame , where the second motion transformation relationship feature includes the frame and the frame corresponding relative motion, the frame and the frame corresponding relative motion to the frame to the frame corresponding relative motion. By multiplying these relative motions through matrix multiplication, the second motion transformation matrix is obtained; the first motion transformation matrix and the second motion transformation matrix are complementary.

[0047] Based on the preset sliding window range, through the first motion transformation matrix and the second motion transformation matrix , dynamically adjust the search area for feature matching to ensure that the feature matching process can adapt to the dynamic changes of the sensor, reduce the matching errors caused by motion blur or perspective changes. Preferably, through two-way analysis and dynamic adjustment of the search window, it can better adapt to the changes in the complex dynamic environment, reduce the risk of false matching, and ensure the reliability of the feature matching result.

[0048] Furthermore, extract key feature points in the search area. The method of the present application further includes:

[0049] Based on the key feature points, configure feature descriptors in the search area; use the key feature points and feature descriptors to generate an optical flow vector; set a first local search range according to the optical flow vector.

[0050] Specifically, a feature descriptor refers to a vector or matrix used to describe the local area information around a key feature point, which can capture information such as the texture and shape of the feature point, making the feature point comparable in different images or point cloud data. Feature descriptors include SIFT, SURF, ORB, etc.; an optical flow vector refers to a vector that describes the motion direction and speed of a key feature point between consecutive frames, usually obtained by calculating the position change of the feature point in adjacent frames. The optical flow reflects the motion trajectory of the feature point on the image plane; the first local search range refers to the initial search area for feature matching determined based on the optical flow vector, which is used to find the matching points corresponding to the key feature points in the current frame in subsequent frames.

[0051] During the multi-sensor fusion mine tunneling navigation process, in order to improve the efficiency and accuracy of feature matching, it is necessary to further describe and analyze the key feature points. Specifically, based on the extracted key feature points, generate feature descriptors for each key feature point in the search area. By analyzing the local area around the key feature point, extract its texture, shape and other information to form a descriptor that can uniquely identify the feature point. For example, in image data, the SIFT or ORB algorithm can be used to generate descriptors for each key feature point for quick identification and matching in subsequent frames.

[0052] Using key feature points and their feature descriptors, calculate the motion changes of key feature points between consecutive frames to generate optical flow vectors. The optical flow vectors reflect the motion directions and speeds of the feature points, providing motion clues for subsequent feature matching. For example, by calculating the position changes of key feature points in adjacent frames, their motion trajectories on the image plane can be obtained, thereby generating optical flow vectors; according to the directions and magnitudes of the optical flow vectors, determine the possible positions of the key feature points in subsequent frames and set the first local search range. The optical flow vectors provide the motion direction and speed information of the key feature points, so the size and position of the search range can be dynamically adjusted based on this information. For example, if the optical flow vector indicates that the feature point is moving rapidly in the horizontal direction, the search range will be expanded accordingly in the horizontal direction to ensure that the feature point can be correctly matched at the new position.

[0053] Set the first local search range to reduce the search space for feature matching, improve the matching efficiency and accuracy. By using the motion information provided by the optical flow vectors, the positions of key feature points in subsequent frames can be quickly located, thereby reducing the matching errors caused by motion blur or perspective changes. For example, in the scenario of mine tunneling navigation, the rapid movement of mine tunneling equipment may cause the feature points to move rapidly in the image. By setting the first local search range, it can be ensured that the feature points are correctly matched at the new position, thereby improving the real-time performance and reliability of the navigation system.

[0054] Furthermore, the method of this application further includes:

[0055] Based on the direction of the optical flow vector, determine the predicted position coordinates of the key feature point in the next frame; based on the magnitude of the optical flow vector, adjust the search step size of the first local search range, and update the first local search range with the predicted position coordinates as the center.

[0056] Specifically, the predicted position coordinates refer to the positions where the key feature points may appear in the next frame image deduced according to the directions and magnitudes of the optical flow vectors, which are estimates based on the motion trends of the feature points in the current frame; the search step size refers to the distance that the search algorithm moves each time during the feature matching process. The size of the search step size affects the search efficiency and accuracy: a larger step size can speed up the search, but may miss the target feature points; a smaller step size can improve the matching accuracy, but will increase the computational amount; updating the first local search range refers to redefining the search area for feature matching according to the predicted position coordinates and the adjusted search step size to adapt to the motion changes of the feature points.

[0057] During the process of multi-sensor fusion for mine tunneling navigation, in order to further improve the efficiency and accuracy of feature matching, it is necessary to dynamically adjust the search strategy according to the direction and magnitude of the optical flow vector. Specifically, according to the direction of the optical flow vector, calculate the predicted position coordinates of the key feature points in the next frame. The direction of the optical flow vector reflects the movement trend of the feature points. Therefore, by moving the position of the feature points in the current frame along the direction of the optical flow vector, an approximate position in the next frame can be obtained. For example, if the direction of the optical flow vector is towards the upper right, then the predicted position coordinates will shift towards the upper right.

[0058] Based on the magnitude of the optical flow vector (i.e., the movement speed of the feature points), dynamically adjust the search step size of the first local search range. The larger the magnitude of the optical flow vector, the faster the movement speed of the feature points. Therefore, it is necessary to appropriately increase the search step size to cover a larger area; conversely, if the magnitude is smaller, the search step size can be reduced to improve the matching accuracy. For example, if the magnitude of the optical flow vector is large, the search step size can be increased to 1.5 times the original; if the magnitude is small, it can remain unchanged or be appropriately reduced.

[0059] Centered on the predicted position coordinates, redefine the first local search range according to the adjusted search step size. This process ensures that the search area can closely follow the movement trend of the feature points, thereby improving the success rate and efficiency of feature matching. For example, if the predicted position coordinates are in the upper right corner of the image, the search range will expand around this position and adjust its size according to the search step size; by dynamically adjusting the search strategy, reduce the matching errors caused by the rapid movement or changing movement direction of the feature points. For example, in the mine tunneling navigation scenario, when the mine tunneling equipment is advancing rapidly or encountering complex geological structures, the feature points may move quickly. By adjusting the search step size and updating the search range, it can ensure that the feature points are correctly matched at the new position, thereby improving the real-time performance and robustness of the mine tunneling navigation system.

[0060] Furthermore, based on the magnitude of the optical flow vector, adjust the search step size of the first local search range. The method of the present application includes:

[0061] The adjustment amount of the search step size of the first local search range ; where used to characterize the adjustment amount of the search step size of the first local search range, S base refers to the preset reference step size value in image feature matching, used to characterize the magnitude of the optical flow vector, used to characterize the maximum value of the magnitude of the optical flow vector, is an adjustment coefficient, used to control the influence degree of the optical flow vector magnitude on the search step size, and the value range is [0, 1].

[0062] Specifically, the adjustment amount of the search step size of the first local search range refers to the increment or decrement of the search step size dynamically adjusted according to the magnitude of the optical flow vector during the feature matching process, which is used to optimize the search efficiency and matching accuracy; S base refers to the preset reference step size value in image feature matching, which is used to control the movement of the first local search range during feature point matching, and the magnitude of the optical flow vector refers to the length of the optical flow vector, which reflects the movement speed of the feature point on the image plane; the maximum magnitude of the optical flow vector refers to the maximum length reached by the optical flow vector in the current scene, which is used to normalize the magnitude of the optical flow vector; the adjustment coefficient is a parameter used to control the influence degree of the optical flow vector magnitude on the search step size, and its value range is [0, 1]. The closer the adjustment coefficient is to 1, the greater the influence of the optical flow vector magnitude on the search step size; the closer it is to 0, the smaller the influence.

[0063] During the multi-sensor fusion mine tunneling navigation process, in order to adapt to the movement changes of feature points, it is necessary to dynamically adjust the search step size according to the magnitude of the optical flow vector. The adjustment amount of the search step size of the first local search range ; Add the calculated adjustment amount to the original search step size to obtain a new search step size. By dynamically adjusting the search step size, the search efficiency and matching accuracy can be balanced. When the movement speed of the feature point is relatively fast, increasing the search step size can expand the search range and improve the success rate of matching; when the movement speed of the feature point is relatively slow, reducing the search step size can improve the matching accuracy and adapt to the movement changes of the feature point.

[0064] Furthermore, the method of the present application further includes:

[0065] Based on the key feature points, configure multi-scale levels in the search area; set feature matching pairs according to the multi-scale levels and feature descriptors; set a second local search range according to the spatial consistency of the feature matching pairs.

[0066] Specifically, the multi-scale level refers to multiple different-resolution or scale representations constructed for key feature points within the search area. Through multi-scale analysis, the details and global information of feature points can be captured at different levels, enhancing the robustness of feature matching; the feature matching pair refers to the pair of matching feature points found based on feature descriptors between different frames, and these matching pairs are used to describe the corresponding relationship of feature points between consecutive frames; the spatial consistency refers to the similarity or regularity of the spatial positions of feature matching pairs. If the spatial position change trends of multiple feature matching pairs are similar, they are considered to have spatial consistency; the second local search range refers to the search area further optimized based on the spatial consistency of feature matching pairs, which is used to more accurately locate the matching position of feature points.

[0067] In the process of multi-sensor fusion for mine tunneling navigation, in order to improve the accuracy and robustness of feature matching, it is necessary to construct a multi-scale hierarchy within the search area and set up feature matching pairs and optimize the search range based on this. Specifically, based on the key feature points, a multi-scale hierarchy is constructed within the search area. By analyzing the feature points at different scales, local details and global information can be captured simultaneously. For example, the pyramid structure can be used to downsample the image to generate multiple image layers with different resolutions, and each layer corresponds to a scale. At each scale, the feature descriptors will be recalculated to adapt to the feature point representation at different scales; according to the multi-scale hierarchy and feature descriptors, matching feature point pairs are found between different frames. By performing matching at multiple scales, the reliability of the matching can be improved. For example, if a matching pair is found at a coarse scale, these matching pairs can be further verified and refined at a fine scale, thereby improving the matching accuracy.

[0068] According to the spatial consistency of the feature matching pairs, the second local search range is determined. If multiple feature matching pairs show consistency in spatial positions, the search range can be narrowed and concentrated around these matching pairs, further reducing the search space, improving the matching efficiency, and reducing the possibility of false matches; through multi-scale analysis and spatial consistency constraints, the matching positions of the feature points can be more accurately located, improving the accuracy of feature matching.

[0069] Furthermore, the method of this application further includes:

[0070] According to the confidence of the feature matching pairs, the feature matching pairs with a confidence lower than the preset confidence threshold are eliminated to determine the remaining feature matching pairs; based on the motion consistency of the feature matching pairs, the search step size of the second local search range is dynamically adjusted, and with the feature matching pair with the highest spatial consistency among the remaining feature matching pairs as the center, the second local search range is updated.

[0071] Specifically, the confidence is a quantitative index indicating the reliability or matching quality of the feature matching pair. The higher the confidence, the higher the reliability of the feature matching pair; the confidence threshold is a preset threshold used to screen the feature matching pairs with a confidence higher than this threshold, thereby eliminating unreliable matching pairs; the motion consistency refers to the similarity in the motion direction and speed of the feature matching pairs. If the motion directions and speeds of multiple matching pairs are similar, they are considered to have a high motion consistency; dynamically adjusting the search step size means adjusting the search step size of the second local search range in real time according to the motion consistency of the feature matching pairs to adapt to the motion changes of the feature points.

[0072] During the multi-sensor fusion mine tunneling navigation process, in order to further improve the accuracy and reliability of feature matching, it is necessary to screen and optimize feature matching pairs and dynamically adjust the search range. Specifically, according to the confidence of feature matching pairs, the matching pairs with confidence lower than the preset threshold are eliminated, and the matching pairs with higher confidence are retained, effectively removing mis-matched feature points and improving the reliability of the matching result. For example, if the confidence threshold is set to 0.8, only the matching pairs with confidence higher than 0.8 will be retained; based on the motion consistency of the remaining feature matching pairs, the search step size of the second local search range is dynamically adjusted. If the motion consistency of the feature matching pairs is high, the search step size can be appropriately reduced to improve the matching accuracy; if the consistency is low, the search step size can be appropriately increased to expand the search range. For example, if the motion directions and speeds of the feature matching pairs are very similar, the search step size can be reduced to 0.5 times the original; if the consistency is low, it can be increased to 1.5 times the original.

[0073] Centering on the matching pair with the highest spatial consistency among the remaining feature matching pairs, the second local search range is redefined to ensure that the search range is always concentrated around the most reliable feature points, thereby further improving the accuracy and efficiency of the matching. For example, if a certain feature matching pair has the highest spatial consistency, the search range will be expanded around this matching pair and optimized according to the adjusted search step size; by screening and optimizing feature matching pairs, the matching positions of feature points are more accurately located, improving the accuracy and robustness of feature matching.

[0074] Furthermore, based on the motion consistency of feature matching pairs, the search step size of the second local search range is dynamically adjusted. The method of the present application includes:

[0075] The adjustment amount of the search step size of the second local search range ; where is used to characterize the adjustment amount of the search step size of the second local search range, and are adjustment coefficients used to balance the weights of confidence and motion consistency, is the confidence of the i-th feature matching pair, with a value range of [0, 1], M is the total number of remaining feature matches, is the motion consistency angle of the i-th feature matching pair; is used to quantify motion consistency.

[0076] Specifically, the adjustment amount of the search step size of the second local search range refers to the increment or decrement of the search step size dynamically adjusted according to the confidence and motion consistency of feature matching pairs during the feature matching process. This adjustment amount is used to optimize the search efficiency and matching accuracy; and is an adjustment coefficient, which is a parameter used to control the influence degree of confidence and motion consistency on the search step size. Its value range is [0, 1]. The closer the adjustment coefficient is to 1, the greater the influence of confidence and motion consistency on the search step size; the closer it is to 0, the smaller the influence. is the confidence of the i-th feature matching pair, which is a quantitative index of the reliability or matching quality of the i-th feature matching pair. The higher the confidence, the higher the reliability of the feature matching pair; the motion consistency angle refers to the similarity of the i-th feature matching pair in the motion direction. If the motion directions of multiple matching pairs are similar, it is considered to have a high motion consistency; quantifying motion consistency refers to converting the motion consistency angle into a numerical form for easy calculation and comparison.

[0077] During the multi-sensor fusion mine tunneling navigation process, in order to adapt to the motion changes of feature points and improve the accuracy of feature matching, it is necessary to dynamically adjust the search step size according to the confidence and motion consistency of feature matching pairs. Specifically, the adjustment amount of the search step size of the second local search range ; Add the calculated adjustment amount to the original search step size to obtain a new search step size. By dynamically adjusting the search step size, the search efficiency and matching accuracy can be balanced. When the motion directions and speeds of feature points are relatively consistent, reducing the search step size can improve the matching accuracy; when the motion directions and speeds of feature points change greatly, increasing the search step size can expand the search range, improve the success rate of matching, adapt to the motion changes of feature points, more accurately locate the matching position of feature points, and improve the robustness and adaptability of the mine tunneling navigation system.

[0078] Furthermore, the method of the present application further includes:

[0079] Nest the first local search range within a dynamic search window and set the first nested combination; nest the second local search range within the dynamic search window and set the second nested combination; through the first nested combination and the second nested combination, globally constrain the dynamic search window.

[0080] Specifically, the nested combination means embedding a local search range (such as the first local search range or the second local search range) into a larger search window (such as the dynamic search window) and constraining and optimizing the search process through a combined method. The purpose of the nested combination is to coordinate the search strategy at different levels to improve the matching efficiency and accuracy; the global constraint means imposing limiting conditions on the entire dynamic search window through the nested combination method to ensure that the search process maintains consistency and reliability within the global scope. The global constraint can prevent the search range from being too large or too small, thereby avoiding false matching or missed matching.

[0081] During the multi-sensor fusion mine tunneling navigation process, in order to further optimize the search strategy for feature matching, it is necessary to nest the local search range into a dynamic search window and coordinate the search process through global constraints. Specifically, the first local search range is nested into the dynamic search window to form the first nested combination. The first local search range is usually adjusted based on the direction and magnitude of the optical flow vector and is applicable to feature points with fast movement. Through the nested combination, it can be ensured that the first local search range searches within the range of the dynamic search window, and at the same time, the information of the optical flow vector is used to improve the search efficiency. The second local search range is nested into the dynamic search window to form the second nested combination. The second local search range is usually adjusted based on the spatial consistency and confidence of the feature matching pairs and is applicable to scenarios that require higher-precision matching. Through the nested combination, the search range can be further optimized within the range of the dynamic search window to improve the matching accuracy.

[0082] Through the first nested combination and the second nested combination, global constraints are imposed on the dynamic search window. The role of global constraints is to ensure that within the dynamic search window, the search process takes into account both feature points with fast movement (through the first local search range) and the requirements for high-precision matching (through the second local search range). For example, if the first local search range expands due to a large magnitude of the optical flow vector, the second local search range can be adjusted through spatial consistency constraints to ensure that the search range is neither too large nor too small.

[0083] Through nested combinations and global constraints, on the one hand, the two strategies of the first nested combination and the second nested combination are coordinated within the dynamic search window, and the search range is gradually narrowed within the dynamic search window, reducing unnecessary calculations, thereby improving the efficiency of feature matching, ensuring that the search process is both efficient and accurate. On the other hand, global constraints can prevent the search range from becoming unreasonable due to local adjustments, thereby avoiding false matches or missed matches, ensuring the consistency and reliability of feature matching in the global range, improving the accuracy and robustness of feature matching, and ensuring the efficient operation of the mine tunneling navigation system.

[0084] In summary, the beneficial effects of the embodiments of this application are:

[0085] By using inertial navigation data to assist in calibrating the image information uploaded by the image monitoring device and the point cloud information uploaded by the lidar, the first dynamic calibration feature and the second dynamic calibration feature are determined; the motion postures between consecutive frames are captured to compensate the search area for image feature matching; key feature points are extracted in the search area and projected and matched with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set; based on the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization objective, the external parameter matrix is iteratively updated; using the updated external parameter matrix to verify the initial feature matching set to obtain the image feature matching set. The present application provides an image feature matching method and system for multi-sensor fusion navigation, realizes capturing the motion postures between consecutive frames, reduces the image feature matching error caused by motion, and further, through the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization objective, iteratively updates the external parameter matrix to gradually approach the optimal solution, improving the technical effects of the image feature matching accuracy and stability.

[0086] Embodiment 2

[0087] Based on the same inventive concept as the image feature matching method for multi-sensor fusion navigation in the foregoing embodiment, as Figure 2 shown, the embodiment of the present application provides an image feature matching system for multi-sensor fusion navigation, wherein the system includes:

[0088] An auxiliary calibration module M100, configured to use inertial navigation data to assist in calibrating the image information uploaded by the image monitoring device to determine the first dynamic calibration feature; use inertial navigation data to assist in calibrating the point cloud information uploaded by the lidar to determine the second dynamic calibration feature.

[0089] A motion posture capture module M200, configured to capture the motion postures between consecutive frames to compensate the search area for image feature matching.

[0090] A projection matching module M300, configured to extract key feature points in the search area and perform projection matching with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set.

[0091] An iterative update module M400, configured to iteratively update the external parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization objective.

[0092] A verification module M500, configured to use the updated external parameter matrix to verify the initial feature matching set to obtain the image feature matching set.

[0093] Further, the iterative update module M400 is further configured to execute the following method:

[0094] Set a joint optimization objective by fusing the reprojection error corresponding to the image monitoring device and the spatial matching error corresponding to the lidar.

[0095] Model the reprojection error term and the spatial matching error term in the joint optimization objective as factor nodes in a factor graph, perform factor graph optimization, combine the state variables of each factor node, and set the extrinsic parameter matrix.

[0096] Furthermore, the iterative update module M400 is also used to execute the following method:

[0097] Synchronize the timestamps of the image monitoring device and the lidar, and set a time synchronization mechanism.

[0098] Based on the time synchronization mechanism, perform interpolation processing on asynchronous data to generate a synchronous data stream, where the synchronous data stream includes inertial navigation data.

[0099] Furthermore, the motion pose capture module M200 is used to execute the following method:

[0100] According to a preset sliding window, determine the image information and point cloud information from the -th frame to the -th frame, where N is a positive integer divisible by 2.

[0101] Based on the image information and point cloud information from the -th frame to the -th frame, determine the first motion transformation relationship feature and the second motion transformation relationship feature between adjacent frames.

[0102] According to the first motion transformation relationship feature and the second motion transformation relationship feature, compensate the search area for image feature matching.

[0103] Furthermore, the motion pose capture module M200 is also used to execute the following method:

[0104] Based on the image information and point cloud information of the -th frame and the -th frame, determine the relative motion.

[0105] Based on the image information and point cloud information of the -th frame and the -th frame, determine the relative motion.

[0106] Traverse the image information and point cloud information from the -th frame to the -th frame, and through the relative motion and the relative motion to the Determine the first motion transformation relationship feature based on the relative motion.

[0107] Furthermore, the motion attitude capture module M200 is further configured to execute the following method:

[0108] Based on the image information and point cloud information of the th frame and the th frame, determine the relative motion.

[0109] Based on the image information and point cloud information of the th frame and the th frame, determine the relative motion.

[0110] Traverse the image information and point cloud information from the th frame to the th frame, and determine the second motion transformation relationship feature through the relative motion, the relative motion to the relative motion.

[0111] Furthermore, the motion attitude capture module M200 is further configured to execute the following method:

[0112] Based on the first motion transformation relationship feature, determine the first motion transformation matrix from the th frame to the th frame through cumulative motion transformation .

[0113] Based on the second motion transformation relationship feature, determine the second motion transformation matrix from the th frame to the th frame through cumulative motion transformation .

[0114] Based on the preset sliding window, configure a dynamic search window through the first motion transformation matrix and the second motion transformation matrix .

[0115] Furthermore, the projection matching module M300 is further configured to execute the following method:

[0116] Configure feature descriptors in the search area based on the key feature points.

[0117] Generate an optical flow vector using the key feature points and the feature descriptors.

[0118] Set a first local search range according to the optical flow vector.

[0119] Further, the projection matching module M300 is further configured to execute the following method:

[0120] Based on the direction of the optical flow vector, determine the predicted position coordinates of the key feature points in the next frame.

[0121] Based on the magnitude of the optical flow vector, adjust the search step size of the first local search range, and update the first local search range with the predicted position coordinates as the center.

[0122] Further, the projection matching module M300 is further configured to execute the following method:

[0123] The adjustment amount of the search step size of the first local search range .

[0124] Wherein, is used to characterize the adjustment amount of the search step size of the first local search range, S base refers to the preset reference step size value in image feature matching, is used to characterize the magnitude of the optical flow vector, is used to characterize the maximum value of the magnitude of the optical flow vector, is an adjustment coefficient used to control the influence degree of the optical flow vector magnitude on the search step size, and the value range is [0, 1].

[0125] Further, the projection matching module M300 is further configured to execute the following method:

[0126] Based on the key feature points, configure multi-scale levels in the search area.

[0127] According to the multi-scale levels and feature descriptors, set feature matching pairs.

[0128] According to the spatial consistency of the feature matching pairs, set the second local search range.

[0129] Further, the projection matching module M300 is further configured to execute the following method:

[0130] According to the confidence level of the feature matching pairs, eliminate the feature matching pairs with a confidence level lower than the preset confidence threshold, and determine the remaining feature matching pairs.

[0131] Based on the motion consistency of the feature matching pairs, dynamically adjust the search step size of the second local search range, and update the second local search range with the feature matching pair with the highest spatial consistency among the remaining feature matching pairs as the center.

[0132] Further, the projection matching module M300 is further configured to execute the following method:

[0133] Adjustment amount of the search step size of the second local search range 。

[0134] Wherein, is used to represent the adjustment amount of the search step size of the second local search range, and are adjustment coefficients, used to balance the weights of confidence and motion consistency, is the confidence of the i-th feature matching pair, and its value range is [0, 1], M is the total number of remaining feature matches, is the motion consistency angle of the i-th feature matching pair; is used to quantify the motion consistency.

[0135] Furthermore, the projection matching module M300 is further used to execute the following method:

[0136] Nest the first local search range within a dynamic search window and set the first nested combination.

[0137] Nest the second local search range within a dynamic search window and set the second nested combination.

[0138] Globally constrain the dynamic search window through the first nested combination and the second nested combination.

[0139] In summary, any step can be stored as computer instructions or programs in an unrestricted computer memory and can be called and recognized by an unrestricted computer processor, and no redundant restrictions are imposed here.

[0140] Furthermore, the above technical solution only reflects the preferred technical solution of the technical solution of the embodiments of the present application. Some changes that those skilled in the art may make to some parts thereof all reflect the principles of the novel embodiments of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application.

Claims

1. An image feature matching method for multi-sensor fusion navigation, characterized in that: The method comprises: Using the inertial navigation data to assist in calibrating the image information uploaded by the image monitoring device, and determining the first dynamic calibration feature; using the inertial navigation data to assist in calibrating the point cloud information uploaded by the laser radar, and determining the second dynamic calibration feature; Capture the motion gestures between consecutive frames and compensate the search area for image feature matching; Extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set; Iteratively updating the extrinsic parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature in combination with a joint optimization objective; Using the updated extrinsic parameter matrix, verifying the initial feature matching set to obtain an image feature matching set; Among them, capturing the motion posture between consecutive frames and compensating the search area for image feature matching include: According to the preset sliding window, determine the Frame to The image information and point cloud information of the frame, where N is a positive integer divisible by 2; Based on Frame to The image information and point cloud information of the frames are used to determine the first motion transformation relationship characteristics and the second motion transformation relationship characteristics between adjacent frames; Compensating a search area for image feature matching according to the first motion transformation relationship feature and the second motion transformation relationship feature; Among them, based on Frame to The image information and point cloud information of the frame are used to determine the first motion transformation relationship characteristics between adjacent frames, including: Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determining the first motion transformation relationship characteristics; Among them, based on Frame to The image information and point cloud information of the frame are used to determine the second motion transformation relationship characteristics between adjacent frames, including: Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determining the characteristics of the second motion transformation relationship; Wherein, according to the first motion transformation relationship feature and the second motion transformation relationship feature, the search area for compensating image feature matching includes: Based on the first motion transformation relationship feature, the first Frame to The first motion transformation matrix of the frame ; Based on the second motion transformation relationship feature, the first Frame to The second motion transformation matrix of the frame ; Based on the preset sliding window, through the first motion transformation matrix , the second motion transformation matrix , configure the dynamic search window.

2. The image feature matching method for multi-sensor fusion navigation according to claim 1, characterized in that: Setting a joint optimization target to fuse the reprojection error corresponding to the image monitoring device and the spatial matching error corresponding to the laser radar; The reprojection error term and the spatial matching error term in the joint optimization objective are modeled as factor nodes in a factor graph, the factor graph is optimized, and the state variables of each factor node are combined to set the external parameter matrix.

3. The image feature matching method for multi-sensor fusion navigation according to claim 2, characterized in that: The method further comprises: Performing time stamp synchronization on the image monitoring device and the laser radar, and setting a time synchronization mechanism; Based on the time synchronization mechanism, the asynchronous data is interpolated to generate a synchronous data stream, wherein the synchronous data stream includes inertial navigation data.

4. The image feature matching method for multi-sensor fusion navigation according to claim 1, characterized in that: Extracting key feature points in the search area, the method further includes: Based on the key feature points, configuring a feature descriptor in the search area; Generate an optical flow vector using the key feature points and feature descriptors; A first local search range is set according to the optical flow vector.

5. The image feature matching method for multi-sensor fusion navigation according to claim 4, characterized in that: Determining the predicted position coordinates of the key feature point in the next frame based on the direction of the optical flow vector; Based on the magnitude of the optical flow vector, the search step size of the first local search range is adjusted, and the first local search range is updated with the predicted position coordinates as the center.

6. The image feature matching method for multi-sensor fusion navigation according to claim 5, characterized in that: Based on the magnitude of the optical flow vector, adjusting the search step size of the first local search range, the method comprising: The adjustment amount of the search step length of the first local search range ; in, The adjustment amount for characterizing the search step length of the first local search range, S base Refers to the pre-set reference step value in image feature matching. Used to characterize the magnitude of the optical flow vector, Used to characterize the maximum amplitude of the optical flow vector, It is an adjustment coefficient used to control the influence of the optical flow vector amplitude on the search step size, and its value range is [0, 1].

7. The image feature matching method for multi-sensor fusion navigation according to claim 6, characterized in that: The method comprises: Based on the key feature points, configuring a multi-scale hierarchy in the search area; According to the multi-scale level and the feature descriptor, setting feature matching pairs; A second local search range is set according to the spatial consistency of the feature matching pairs.

8. The image feature matching method for multi-sensor fusion navigation according to claim 7, characterized in that: According to the confidence of the feature matching pairs, the feature matching pairs with a confidence lower than a preset threshold are eliminated to determine the remaining feature matching pairs; Based on the motion consistency of the feature matching pairs, the search step length of the second local search range is dynamically adjusted, and the second local search range is updated with the feature matching pair with the highest spatial consistency among the remaining feature matching pairs as the center.

9. The image feature matching method for multi-sensor fusion navigation according to claim 8, characterized in that: Based on the motion consistency of the feature matching pair, dynamically adjusting the search step length of the second local search range, the method comprises: The adjustment amount of the search step length of the second local search range ; in, The adjustment amount for characterizing the search step length of the second local search range, and is an adjustment factor used to balance the weights of confidence and motion consistency, is the confidence of the ith feature match pair, ranging from [0, 1], M is the total number of remaining feature matches, is the motion consistency angle of the i-th feature matching pair; Used to quantify motion consistency.

10. The image feature matching method for multi-sensor fusion navigation according to claim 9, characterized in that: Nesting the first local search range in the dynamic search window to set a first nested combination; Nesting the second local search range in the dynamic search window to set a second nested combination; The dynamic search window is globally constrained by the first nested combination and the second nested combination.

11. Image feature matching system for multi-sensor fusion navigation, characterized in that: The system is used to implement the image feature matching method for multi-sensor fusion navigation according to any one of claims 1 to 10, comprising: An auxiliary calibration module is used to use the inertial navigation data to perform auxiliary calibration on the image information uploaded by the image monitoring device to determine the first dynamic calibration feature; and use the inertial navigation data to perform auxiliary calibration on the point cloud information uploaded by the laser radar to determine the second dynamic calibration feature; Motion gesture capture module, used to capture the motion gesture between consecutive frames and compensate the search area for image feature matching; A projection matching module is used to extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set; An iterative updating module, configured to iteratively update an external parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature in combination with a joint optimization objective; A verification module, used to verify the initial feature matching set using the updated extrinsic parameter matrix to obtain an image feature matching set; Among them, the motion gestures between consecutive frames are captured, and the search area for image feature matching is compensated. The system is used to perform the following method steps: According to the preset sliding window, determine the Frame to The image information and point cloud information of the frame, where N is a positive integer divisible by 2; Based on Frame to The image information and point cloud information of the frames are used to determine the first motion transformation relationship characteristics and the second motion transformation relationship characteristics between adjacent frames; Compensating a search area for image feature matching according to the first motion transformation relationship feature and the second motion transformation relationship feature; Among them, based on Frame to The image information and point cloud information of the frame are used to determine the first motion transformation relationship characteristics between adjacent frames, and the system is used to perform the following method steps: Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determining the first motion transformation relationship characteristics; Among them, based on Frame to The image information and point cloud information of the frame are used to determine the second motion transformation relationship characteristics between adjacent frames, and the system is used to perform the following method steps: Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determining the characteristics of the second motion transformation relationship; Wherein, according to the first motion transformation relationship feature and the second motion transformation relationship feature, the search area for compensating image feature matching is used to perform the following method steps: Based on the first motion transformation relationship feature, the first Frame to The first motion transformation matrix of the frame ; Based on the second motion transformation relationship feature, the first Frame to The second motion transformation matrix of the frame ; Based on the preset sliding window, through the first motion transformation matrix , the second motion transformation matrix , configure the dynamic search window.

Images