Image feature matching method and system for multi-sensor fusion navigation

Abstract

The invention relates to the technical field related to image feature analysis, in particular to an image feature matching method and system for multi-sensor fusion navigation, and the method comprises the following steps: calibrating image information and point cloud information in an auxiliary manner by using inertial navigation data, and determining dynamic calibration features; capturing a continuous frame motion attitude compensation search area, extracting key feature points, and performing projection matching to generate an initial set; based on the dynamic calibration features, the dynamic calibration features and the joint optimization target are combined to iteratively update the external parameter matrix, and verification is performed to obtain the image feature matching set, so that the technical problems that image feature matching is easily influenced by motion blur and view angle change and the image feature matching precision is limited are solved, the motion attitude between continuous frames is captured, and the accuracy of image feature matching is improved. The method has the technical effects that a search area for image feature matching is compensated, image feature matching errors caused by movement are reduced, the optimal solution is gradually approached by combining a joint optimization target and iteratively updating an external parameter matrix, and the image feature matching precision and stability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field related to image feature analysis, and in particular to an image feature matching method and system for multi-sensor fusion navigation. Background Art

[0002] Multi-sensor image fusion refers to the appropriate processing of images of the same target or scene collected by multiple sensors to produce a new image that is more suitable for human eye perception or subsequent computer processing. Each sensor is designed to adapt to a specific environment and scope of application. Images acquired by sensors with different characteristics or different viewpoints are both redundant and complementary. By fusing them, the reliability of the system and the efficiency of image information utilization can be effectively improved.

[0003] In navigation and positioning systems, by combining data from multiple sensors (such as visual sensors, inertial sensors, GPS, lidar, etc.), the shortcomings of a single sensor can be compensated and the overall performance of the system can be improved. However, in terms of image feature matching, due to differences in imaging principles, resolutions, and viewing angles of different sensors, image feature extraction and matching are difficult.

[0004] In summary, in the prior art, there are technical problems that image feature matching is easily affected by motion blur and perspective changes, and the image feature matching accuracy is limited. Summary of the invention

[0005] The present application provides an image feature matching method and system for multi-sensor fusion navigation, aiming to solve the technical problems in the prior art that image feature matching is easily affected by motion blur and perspective change, and the image feature matching accuracy is limited.

[0006] In view of the above problems, the technical solution to implement this application is: On the one hand, the present application provides an image feature matching method for multi-sensor fusion navigation, wherein the method includes: using inertial navigation data to assist in the calibration of image information uploaded by an image monitoring device to determine a first dynamic calibration feature; using inertial navigation data to assist in the calibration of point cloud information uploaded by a lidar to determine a second dynamic calibration feature; capturing motion gestures between consecutive frames to compensate for a search area for image feature matching; extracting key feature points in the search area, and performing projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set; based on the first dynamic calibration feature and the second dynamic calibration feature, combined with a joint optimization objective, iteratively updating an extrinsic parameter matrix; using the updated extrinsic parameter matrix to verify the initial feature matching set to obtain an image feature matching set.

[0007] On the other hand, the present application provides an image feature matching system for multi-sensor fusion navigation, wherein the system includes: an auxiliary calibration module, used to use inertial navigation data to perform auxiliary calibration on image information uploaded by an image monitoring device to determine a first dynamic calibration feature; use inertial navigation data to perform auxiliary calibration on point cloud information uploaded by a lidar to determine a second dynamic calibration feature; a motion posture capture module, used to capture motion postures between consecutive frames and compensate for a search area for image feature matching; a projection matching module, used to extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set; an iterative update module, used to iteratively update an extrinsic parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature in combination with a joint optimization objective; a verification module, used to verify the initial feature matching set using the updated extrinsic parameter matrix to obtain an image feature matching set.

[0008] In summary, one or more technical solutions provided in the present application realize the capture of motion postures between consecutive frames, reduce the image feature matching error caused by motion, and further through the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization goal, iteratively update the external parameter matrix, gradually approach the optimal solution, thereby improving the technical effect of image feature matching accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flow chart of an image feature matching method for multi-sensor fusion navigation is provided for this application; Figure 2 A structural schematic diagram of an image feature matching system for multi-sensor fusion navigation is provided for this application.

[0010] Description of the accompanying drawings: auxiliary calibration module M100, motion posture capture module M200, projection matching module M300, iterative update module M400, verification module M500. DETAILED DESCRIPTION

[0011] Embodiment 1 The present application is described in detail below with reference to the accompanying drawings. Figure 1 As shown, the present application provides an image feature matching method for multi-sensor fusion navigation, wherein the method comprises: S1: Use inertial navigation data to assist in the calibration of image information uploaded by the image monitoring device to determine the first dynamic calibration feature; use inertial navigation data to assist in the calibration of point cloud information uploaded by the lidar to determine the second dynamic calibration feature; S2: Capture the motion posture between consecutive frames and compensate for the search area for image feature matching.

[0012] Specifically, inertial navigation data refers to the information about the motion of an object provided by the inertial measurement unit (IMU), including parameters such as acceleration and angular velocity, which can reflect the motion state and posture changes of the object; auxiliary calibration refers to the use of inertial navigation data to calibrate the output data of image monitoring equipment and lidar to ensure the temporal and spatial consistency between different sensor data; dynamic calibration features refer to the real-time adjustment and optimization of the calibration results of sensor data through inertial navigation data in a dynamic environment to adapt to environmental changes; capturing the motion posture between consecutive frames refers to extracting the posture change information of the object during the motion process by analyzing consecutive image frames or point cloud data.

[0013] In the process of mine excavation navigation with multi-sensor fusion, image monitoring equipment and lidar provide two-dimensional images and three-dimensional point cloud data respectively, but these data may be inconsistent in time and space. By using inertial navigation data to assist in the calibration of image information and point cloud information, the data of different sensors can be aligned to the same coordinate system, thereby determining the first dynamic calibration feature and the second dynamic calibration feature, effectively solving the problem of temporal and spatial deviations between sensors and improving the accuracy of data fusion.

[0014] At the same time, capturing the motion posture between consecutive frames and compensating the search area for image feature matching can dynamically adjust the range of feature matching and reduce matching errors caused by motion blur or changes in perspective. For example, in the mine excavation navigation scenario, the movement of the mine excavation equipment causes the offset of image feature points. By capturing the motion posture and compensating the search area, it can ensure that the feature points can still be accurately matched in a dynamic environment, thereby improving the accuracy and stability of the mine excavation navigation system.

[0015] S3: extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set; S4: based on the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization goal, iteratively update the extrinsic parameter matrix; S5: use the updated extrinsic parameter matrix to verify the initial feature matching set to obtain the image feature matching set.

[0016] Specifically, key feature points refer to points that are unique and stable in image or point cloud data, which are used to describe the shape, position and other information of an object, and are repeatable under different viewing angles or times; projection matching refers to projecting the extracted key feature points from their original coordinate system (such as the image coordinate system or the lidar coordinate system) to the unified coordinate system of the inertial navigation system (INS), and finding the correspondence between these points in different sensor data through a matching algorithm; the joint optimization goal refers to integrating the errors of multiple sensor data (such as image reprojection error, spatial matching error of point cloud) into an optimization framework, and optimizing the extrinsic parameter matrix between sensors by minimizing these errors; the extrinsic parameter matrix refers to a matrix that describes the relative position and posture between two sensors, and is used to convert the data of one sensor to the coordinate system of another sensor.

[0017] In the process of mine excavation navigation with multi-sensor fusion, extracting key feature points is the basis for matching image and point cloud data. By extracting key feature points in the search area and projecting them into the inertial navigation coordinate system, the data of different sensors can be unified into the same reference frame, ensuring the correspondence between feature points in different sensor data. Based on the first dynamic calibration feature and the second dynamic calibration feature, the external parameter matrix is ​​iteratively updated in combination with the joint optimization target. By minimizing the image reprojection error and the point cloud space matching error, the relative position and posture relationship between sensors are gradually optimized, thereby improving the accuracy of multi-sensor fusion. For example, in the mine excavation navigation scenario, the movement of the mine excavation equipment will cause dynamic changes in sensor data. By jointly optimizing the external parameter matrix, the deviation between sensors can be corrected in real time to ensure the accuracy of the navigation system. The updated external parameter matrix is ​​used to verify the initial feature matching set, verify the accuracy and consistency of the initial feature matching set, and eliminate the feature points with incorrect matching, further improving the reliability and accuracy of the matching results, and effectively reducing the matching errors caused by sensor errors or environmental changes.

[0018] Furthermore, the present application method also includes: A joint optimization target is set to integrate the reprojection error corresponding to the image monitoring device and the spatial matching error corresponding to the lidar; the reprojection error term and the spatial matching error term in the joint optimization target are modeled as factor nodes in a factor graph, and the factor graph is optimized. The state variables of each factor node are combined to set the external parameter matrix.

[0019] Specifically, the joint optimization goal refers to integrating the errors of multiple sensor data (such as the reprojection error of image monitoring equipment and the spatial matching error of lidar) into a unified optimization framework, and optimizing the extrinsic parameter matrix between sensors by minimizing the sum of these errors; reprojection error refers to the deviation between the actual projection point and the theoretical projection point when projecting a three-dimensional point onto a two-dimensional image plane in an image monitoring device; spatial matching error refers to the spatial deviation between point cloud data and a reference point cloud or model in lidar; factor graph is a graphical model for representing the constraint relationship between variables, where nodes represent variables or constraints and edges represent the relationship between variables. Factor graph optimization is an optimization method based on a graphical model that solves the optimal value of a variable by minimizing the errors of all constraint terms. Factor nodes are constraint nodes in the factor graph, which are used to represent error terms in the optimization objective (such as reprojection error and spatial matching error). State variables refer to variables that need to be optimized, such as the extrinsic parameter matrix between sensors.

[0020] In the process of mine excavation navigation with multi-sensor fusion, setting a joint optimization target is a key step to improve the accuracy of the system. By fusing the reprojection error of the image monitoring equipment and the spatial matching error of the lidar into one optimization target, the error characteristics of the two sensor data can be considered at the same time, thereby more comprehensively optimizing the extrinsic parameter matrix between the sensors. Furthermore, the reprojection error term and the spatial matching error term are modeled as factor nodes in the factor graph, and the state variables of each factor node are jointly optimized through the factor graph optimization method. This process utilizes the efficient optimization characteristics of the factor graph and can quickly solve the optimal extrinsic parameter matrix. For example, in the mine excavation navigation scenario, the movement of the mine excavation equipment and environmental changes will cause dynamic deviations in the sensor data. Through factor graph optimization, the extrinsic parameter matrix can be adjusted in real time to ensure the consistency of the image and point cloud data in the inertial navigation coordinate system, thereby improving the accuracy and stability of the mine excavation navigation system.

[0021] Furthermore, the present application method also includes: The image monitoring device and the laser radar are synchronized by time stamps, and a time synchronization mechanism is set; based on the time synchronization mechanism, the asynchronous data is interpolated to generate a synchronous data stream, and the synchronous data stream includes inertial navigation data.

[0022] Specifically, timestamp synchronization refers to aligning data collected by different sensors in chronological order to ensure consistency in the time dimension. Timestamp synchronization is the basis of multi-sensor data fusion, because the sampling frequency and data generation time of different sensors may be different; time synchronization mechanism refers to the specific method or strategy used to achieve timestamp synchronization, such as calibrating the time base of the sensor through hardware synchronization signals or software algorithms; asynchronous data refers to the problem of data asynchrony caused by inconsistent sensor sampling times; interpolation processing refers to inserting new data points between known data points through mathematical methods to generate a temporally continuous data stream; synchronous data stream means that after time synchronization processing, the data of different sensors are aligned in time and can be processed and analyzed with a unified time base.

[0023] In the mine excavation navigation system, image monitoring equipment and lidar usually have different sampling frequencies and data generation times, which leads to data asynchrony problems. In order to ensure the consistency of data in the time dimension, it is necessary to synchronize the timestamps of these sensors and set up a corresponding time synchronization mechanism. Furthermore, the time base of the sensor is calibrated by hardware or software methods to ensure that data can be recorded with a unified timestamp. For asynchronous data caused by different sampling frequencies, an interpolation processing method is used to generate a temporally continuous synchronous data stream. For example, the sampling frequency of the lidar may be higher than that of the image sensor. At this time, the corresponding lidar data can be generated at the time point of the image data through interpolation, thereby generating a synchronous data stream containing inertial navigation data, ensuring that the data of different sensors are aligned in time, effectively reducing the errors caused by data asynchrony, and providing a reliable foundation for subsequent feature extraction, matching and fusion.

[0024] Furthermore, the motion gestures between consecutive frames are captured, and the search area for compensating image feature matching is compensated. The method of the present application includes: According to the preset sliding window, determine the Frame to The image information and point cloud information of the frame, wherein N is a positive integer divisible by 2; based on the Frame to The image information and point cloud information of the frame are used to determine the first motion transformation relationship feature and the second motion transformation relationship feature between adjacent frames; and the search area for image feature matching is compensated according to the first motion transformation relationship feature and the second motion transformation relationship feature.

[0025] Specifically, the preset sliding window refers to a fixed-size time or frame window set when processing continuous frame data, which is used to extract and analyze the image information and point cloud information within the window. The size of the sliding window is usually determined according to the specific application scenario and the sampling frequency of the sensor; the first motion transformation relationship feature and the second motion transformation relationship feature refer to the features that describe the motion changes between adjacent frames, such as rotation angle, translation vector or motion trajectory, extracted by analyzing the image information and point cloud information between continuous frames; the search area for compensating image feature matching refers to dynamically adjusting the search range during image feature matching according to the extracted motion transformation relationship features, so as to reduce the offset or loss of feature points caused by motion and improve matching efficiency and accuracy.

[0026] In the process of mine excavation navigation with multi-sensor fusion, in order to deal with the dynamic changes between consecutive frames, the sliding window method is used to extract and analyze the image and point cloud data within a certain time range. Furthermore, according to the preset sliding window, the image and point cloud data from the first frame are determined. Frame to The image information and point cloud information of the frame, where is a positive integer divisible by 2 to facilitate symmetry analysis; based on the data of continuous frames, the motion transformation relationship features between adjacent frames are further calculated. For example, by comparing the image feature points of adjacent frames or the key points in the point cloud, the first motion transformation relationship features and the second motion transformation relationship features are extracted. The first motion transformation relationship features and the second motion transformation relationship features can reflect the posture changes of the sensor during the movement; according to these motion transformation relationship features, the search area of ​​the image feature matching is dynamically compensated. For example, if it is detected that the sensor has undergone a large translation or rotation during the movement, the search area will be adjusted accordingly to ensure that the feature points can be correctly matched at the new position, reduce the matching errors caused by motion blur or changes in viewing angle, and improve the robustness and efficiency of feature matching.

[0027] Furthermore, based on Frame to The image information and point cloud information of the frame are used to determine the first motion transformation relationship characteristics between adjacent frames. The method of the present application includes: Based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion; based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion; traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determine the first motion transformation relationship characteristics.

[0028] Specifically, relative motion refers to the motion change of the sensor or target object calculated by comparing image information or point cloud information between consecutive frames, including translation vector and rotation angle; the first motion transformation relationship feature .... Relative motion, Relative motion, ..., The comprehensive motion features obtained by the sensor (relative motion) are used to describe the overall motion trend and change law of the sensor within the sliding window.

[0029] In the process of mine navigation with multi-sensor fusion, in order to accurately capture the dynamic motion of the sensor, it is necessary to extract motion features by analyzing the relative motion between consecutive frames. Frame and Frame image and point cloud information, calculate the first Relative motion; based on Frame and Frame image and point cloud information, calculate the first Relative motion; by traversing the Frame to All images and point cloud data of the frame are analyzed frame by frame from the Frame to Frame images and point cloud data, integrating arrive The first motion transformation relationship feature is determined based on all relative motion information of the sensor in the entire sliding window. The first motion transformation relationship feature can reflect the overall motion trend of the sensor in the entire sliding window, such as the average motion speed, the cumulative rotation angle, etc. For example, in the mine excavation navigation scenario, the movement of the mine excavation equipment causes the rapid translation and rotation of the sensor. By calculating and integrating the relative motion, the search area for image feature matching is dynamically adjusted to reduce the matching error caused by motion blur or perspective change.

[0030] Furthermore, based on Frame to The image information and point cloud information of the frame are used to determine the second motion transformation relationship characteristics between adjacent frames. The method of the present application includes: Based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion; based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion; traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determine the second motion transformation relationship characteristics.

[0031] Specifically, Relative motion and Relative motion is the motion obtained by analyzing adjacent frames (such as Frame and Frame, Frame and The second motion transformation relationship feature refers to the image information and point cloud information between multiple adjacent frames (such as the first frame), and the calculated motion changes of the sensor or target object between these frames, including translation and rotation. Relative motion, Relative motion, ..., The comprehensive motion characteristics obtained by the sensor (relative motion) are used to describe another motion trend or change law of the sensor in the entire sliding window.

[0032] In the mine excavation navigation process of multi-sensor fusion, in order to more comprehensively capture the dynamic behavior of the sensor, it is necessary to analyze the motion changes between consecutive frames from different angles. Specifically, firstly, based on the Frame and Frame image and point cloud information, calculate the first Relative motion; then based on Frame and Frame image and point cloud information, calculate the first Relative motion; by traversing the Frame to All images and point cloud data of the frame are analyzed frame by frame from the Frame to Frame images and point cloud data, integrating arrive All relative motion information of the image is collected to determine the second motion transformation relationship feature, which can provide another perspective on the sensor movement. By calculating and integrating these relative motions, the sensor state can be predicted more accurately, thereby dynamically adjusting the search area for image feature matching, reducing matching errors caused by motion blur or perspective changes, improving the accuracy and efficiency of feature matching, and enhancing the adaptability of the mine excavation navigation system to complex dynamic environments.

[0033] Distinctively, the first motion transformation relationship feature corresponds to each odd frame to the corresponding even frame of the same motion process; the second motion transformation relationship feature corresponds to each even frame to the corresponding odd frame of the same motion process. Preferably, in order to more comprehensively capture the dynamic behavior of the sensor, the motion transformation relationship between consecutive frames is analyzed from two directions, namely, from odd frames to even frames and from even frames to odd frames. The two-way analysis can provide more complete motion information and improve the accuracy and robustness of feature matching.

[0034] Furthermore, according to the first motion transformation relationship feature and the second motion transformation relationship feature, the search area for compensating image feature matching, the method of the present application includes: Based on the first motion transformation relationship feature, the first Frame to The first motion transformation matrix of the frame Based on the second motion transformation relationship feature, determine the first Frame to The second motion transformation matrix of the frame Based on the preset sliding window, through the first motion transformation matrix , the second motion transformation matrix , configure the dynamic search window.

[0035] Specifically, the cumulative motion transformation refers to the combination of the relative motion transformations between multiple consecutive frames in sequence to obtain the overall motion transformation from the initial frame to the target frame, which is achieved by matrix multiplication and is used to describe the overall motion state of the sensor over a period of time; the first motion transformation matrix It is a matrix accumulated based on the first motion transformation relationship feature (odd frames to even frames), which is used to describe the transition from the first Frame to Motion change of frame; second motion transformation matrix It is a matrix accumulated based on the second motion transformation relationship feature (even frames to odd frames), which is used to describe the transition from the first Frame to The dynamic search window refers to the feature matching search area that is dynamically adjusted according to the motion state of the sensor, and its size and shape can be updated in real time according to the motion transformation matrix to adapt to the dynamic changes of the sensor.

[0036] In the process of mine navigation with multi-sensor fusion, accurate feature matching is the key to achieve high-precision positioning and navigation. Preferably, in order to more accurately compensate for the motion of the sensor and optimize the search area for feature matching, it is necessary to generate a motion transformation matrix based on the extracted motion transformation relationship features, and configure the dynamic search window accordingly. Further, based on the first motion transformation relationship feature (relative motion from odd frames to even frames), by accumulating these relative motion transformations, calculate the relative motion transformations from the first frame to the second frame. Frame to The overall motion transformation of the frame determines the Frame to The first motion transformation matrix of the frame , wherein the first motion transformation relationship feature includes Frame to The frame corresponding to Relative motion, Frame to The frame corresponding to Relative movement to Frame to The frame corresponding to Relative motion, these relative motions are accumulated by matrix multiplication to obtain the first motion transformation matrix Further, based on the second motion transformation relationship feature (relative motion from even frames to odd frames), by accumulating these relative motion transformations, the relative motion from the first frame to the second frame is calculated. Frame to The overall motion transformation matrix of the frame is determined by Frame to The second motion transformation matrix of the frame , wherein the second motion transformation relationship feature includes Frame and The frame corresponding to Relative motion, Frame and The frame corresponding to Relative movement to Frame to The frame corresponding to Relative motion, these relative motions are accumulated by matrix multiplication to obtain the second motion transformation matrix ; The first motion transformation matrix , the second motion transformation matrix To complement each other.

[0037] Based on the preset sliding window range, the first motion transformation matrix , the second motion transformation matrix , dynamically adjust the search area for feature matching to ensure that the feature matching process can adapt to the dynamic changes of the sensor and reduce matching errors caused by motion blur or changes in viewing angle. Preferably, through bidirectional analysis and dynamic adjustment of the search window, it can better adapt to changes in complex dynamic environments, reduce the risk of mismatching, and ensure the reliability of feature matching results.

[0038] Furthermore, in extracting key feature points in the search area, the method of the present application further includes: Based on the key feature points, feature descriptors are configured in the search area; optical flow vectors are generated using the key feature points and feature descriptors; and a first local search range is set according to the optical flow vectors.

[0039] Specifically, a feature descriptor refers to a vector or matrix used to describe the information of the local area around the key feature point. It can capture the texture, shape and other information of the feature point, so that the feature point is comparable in different images or point cloud data. Feature descriptors include SIFT, SURF, ORB, etc.; an optical flow vector refers to a vector that describes the direction and speed of movement of a key feature point between consecutive frames. It is usually obtained by calculating the position change of the feature point in adjacent frames. The optical flow reflects the movement trajectory of the feature point on the image plane; the first local search range refers to the initial search area for feature matching determined based on the optical flow vector, which is used to find matching points corresponding to the key feature points of the current frame in subsequent frames.

[0040] In the process of mine excavation navigation with multi-sensor fusion, in order to improve the efficiency and accuracy of feature matching, it is necessary to further describe and analyze the key feature points. Specifically, based on the extracted key feature points, a feature descriptor is generated for each key feature point in the search area. By analyzing the local area around the key feature point, its texture, shape and other information are extracted to form a descriptor that can uniquely identify the feature point. For example, in image data, SIFT or ORB algorithm can be used to generate a descriptor for each key feature point so that it can be quickly identified and matched in subsequent frames.

[0041] Using key feature points and their feature descriptors, the motion changes of key feature points between consecutive frames are calculated to generate optical flow vectors. The optical flow vectors reflect the motion direction and speed of the feature points and provide motion clues for subsequent feature matching. For example, by calculating the position changes of key feature points in adjacent frames, their motion trajectories on the image plane can be obtained, thereby generating optical flow vectors; according to the direction and size of the optical flow vectors, the possible positions of the key feature points in subsequent frames are determined, and the first local search range is set. The optical flow vectors provide the motion direction and speed information of the key feature points, so the size and position of the search range can be dynamically adjusted based on this information. For example, if the optical flow vector indicates that the feature point moves quickly in the horizontal direction, the search range will be expanded in the horizontal direction accordingly to ensure that the feature point can be correctly matched at the new position.

[0042] The first local search range is set to reduce the search space for feature matching and improve matching efficiency and accuracy. By utilizing the motion information provided by the optical flow vector, the position of key feature points in subsequent frames can be quickly located, thereby reducing matching errors caused by motion blur or changes in perspective. For example, in a mine excavation navigation scenario, the rapid movement of mine excavation equipment may cause feature points to move rapidly in the image. By setting the first local search range, it can be ensured that the feature points are correctly matched at the new position, thereby improving the real-time and reliability of the navigation system.

[0043] Furthermore, the present application method also includes: Based on the direction of the optical flow vector, the predicted position coordinates of the key feature point in the next frame are determined; based on the amplitude of the optical flow vector, the search step size of the first local search range is adjusted, and the first local search range is updated with the predicted position coordinates as the center.

[0044] Specifically, predicting the position coordinates refers to calculating the possible position of the key feature points in the next frame of the image based on the direction and amplitude of the optical flow vector, which is an estimate based on the motion trend of the feature points in the current frame; the search step size refers to the distance the search algorithm moves each time during the feature matching process. The size of the search step size affects the efficiency and accuracy of the search: a larger step size can speed up the search, but may miss the target feature points; a smaller step size can improve the matching accuracy, but will increase the amount of calculation; updating the first local search range refers to redefining the search area for feature matching based on the predicted position coordinates and the adjusted search step size to adapt to the movement changes of the feature points.

[0045] In the process of mine excavation navigation with multi-sensor fusion, in order to further improve the efficiency and accuracy of feature matching, it is necessary to dynamically adjust the search strategy according to the direction and amplitude of the optical flow vector. Specifically, according to the direction of the optical flow vector, the predicted position coordinates of the key feature points in the next frame are calculated. The direction of the optical flow vector reflects the movement trend of the feature points. Therefore, the feature point position of the current frame can be moved along the direction of the optical flow vector to obtain its approximate position in the next frame. For example, if the direction of the optical flow vector is to the upper right, then the predicted position coordinates will be offset to the upper right.

[0046] Based on the amplitude of the optical flow vector (i.e., the movement speed of the feature points), the search step size of the first local search range is dynamically adjusted. The larger the amplitude of the optical flow vector, the faster the movement speed of the feature points. Therefore, the search step size needs to be appropriately increased to cover a larger area. Conversely, if the amplitude is small, the search step size can be reduced to improve the matching accuracy. For example, if the amplitude of the optical flow vector is large, the search step size can be increased to 1.5 times the original value. If the amplitude is small, it can remain unchanged or be appropriately reduced.

[0047] With the predicted position coordinates as the center, the first local search range is redefined according to the adjusted search step size. This process ensures that the search area can closely follow the movement trend of the feature points, thereby improving the success rate and efficiency of feature matching. For example, if the predicted position coordinates are in the upper right corner of the image, the search range will be expanded around this position and its size will be adjusted according to the search step size. By dynamically adjusting the search strategy, the matching error caused by the rapid movement of feature points or changes in the direction of movement is reduced. For example, in the mine excavation navigation scenario, the feature points may move rapidly when the mine excavation equipment is advancing rapidly or encountering complex geological structures. By adjusting the search step size and updating the search range, it can be ensured that the feature points are correctly matched at the new position, thereby improving the real-time and robustness of the mine excavation navigation system.

[0048] Furthermore, based on the amplitude of the optical flow vector, the search step size of the first local search range is adjusted. The method of the present application includes: The adjustment amount of the search step length of the first local search range ;in, The adjustment amount for characterizing the search step length of the first local search range, S base Refers to the pre-set reference step value in image feature matching. Used to characterize the magnitude of the optical flow vector, Used to characterize the maximum amplitude of the optical flow vector, It is an adjustment coefficient used to control the influence of the optical flow vector amplitude on the search step size, and its value range is [0, 1].

[0049] Specifically, the adjustment amount of the search step length of the first local search range is It refers to the process of dynamically adjusting the increment or decrement of the search step size according to the amplitude of the optical flow vector during the feature matching process, which is used to optimize the search efficiency and matching accuracy; S base Refers to the preset reference step value in image feature matching, which is used to control the movement of the first local search range when matching feature points, the amplitude of the optical flow vector It refers to the length of the optical flow vector, which reflects the speed of the feature point on the image plane; the maximum amplitude of the optical flow vector It refers to the maximum length of the optical flow vector in the current scene, which is used to normalize the amplitude of the optical flow vector; adjustment coefficient It is a parameter used to control the influence of the optical flow vector amplitude on the search step size. Its value range is [0, 1]. The closer the adjustment coefficient is to 1, the greater the influence of the optical flow vector amplitude on the search step size; the closer it is to 0, the smaller the influence.

[0050] In the process of mine excavation navigation with multi-sensor fusion, in order to adapt to the movement changes of feature points, it is necessary to dynamically adjust the search step size according to the amplitude of the optical flow vector. The adjustment amount of the search step size in the first local search range is ; Add the calculated adjustment amount to the original search step to obtain a new search step. By dynamically adjusting the search step, the search efficiency and matching accuracy can be balanced. When the feature points move faster, increasing the search step can expand the search range and improve the matching success rate. When the feature points move slower, reducing the search step can improve the matching accuracy and adapt to the movement changes of the feature points.

[0051] Furthermore, the present application method also includes: Based on the key feature points, a multi-scale level is configured in the search area; according to the multi-scale level and feature descriptors, a feature matching pair is set; and according to the spatial consistency of the feature matching pair, a second local search range is set.

[0052] Specifically, the multi-scale hierarchy refers to multiple representations of different resolutions or scales constructed for key feature points in the search area. Through multi-scale analysis, the details and global information of feature points can be captured at different levels, thereby enhancing the robustness of feature matching. Feature matching pairs refer to matching feature point pairs found based on feature descriptors between different frames. These matching pairs are used to describe the correspondence between feature points in consecutive frames. Spatial consistency refers to the similarity or regularity of feature matching pairs in spatial position. If the spatial position change trends of multiple feature matching pairs are similar, they are considered to have spatial consistency. The second local search range refers to the search area further optimized based on the spatial consistency of feature matching pairs, which is used to more accurately locate the matching positions of feature points.

[0053] In the process of mine excavation navigation with multi-sensor fusion, in order to improve the accuracy and robustness of feature matching, it is necessary to construct a multi-scale hierarchy in the search area, and set feature matching pairs and optimize the search range based on this. Specifically, based on key feature points, a multi-scale hierarchy is constructed in the search area. By analyzing feature points at different scales, local details and global information can be captured at the same time. For example, a pyramid structure can be used to downsample the image to generate multiple image layers with different resolutions, each layer corresponding to a scale. At each scale, the feature descriptor will be recalculated to adapt to the feature point representation at different scales; according to the multi-scale hierarchy and feature descriptors, matching feature point pairs are found between different frames. By matching at multiple scales, the reliability of matching can be improved. For example, if matching pairs are found at a coarse scale, these matching pairs can be further verified and refined at a fine scale, thereby improving matching accuracy.

[0054] According to the spatial consistency of the feature matching pairs, the second local search range is determined. If multiple feature matching pairs show consistency in spatial position, the search range can be narrowed and concentrated around these matching pairs, further reducing the search space, improving matching efficiency, and reducing the possibility of mismatching; through multi-scale analysis and spatial consistency constraints, the matching position of feature points can be more accurately located, thereby improving the accuracy of feature matching.

[0055] Furthermore, the present application method also includes: According to the confidence of the feature matching pairs, the feature matching pairs below a preset confidence threshold are eliminated to determine the remaining feature matching pairs; based on the motion consistency of the feature matching pairs, the search step length of the second local search range is dynamically adjusted, and the second local search range is updated with the feature matching pair with the highest spatial consistency among the remaining feature matching pairs as the center.

[0056] Specifically, confidence refers to the reliability of a feature matching pair or a quantitative indicator of matching quality. The higher the confidence, the higher the reliability of the feature matching pair. The confidence threshold is a preset threshold used to screen feature matching pairs with a confidence higher than the threshold, thereby eliminating unreliable matching pairs. Motion consistency refers to the similarity of motion direction and speed of feature matching pairs. If the motion directions and speeds of multiple matching pairs are similar, they are considered to have high motion consistency. Dynamic adjustment of the search step size refers to adjusting the search step size of the second local search range in real time according to the motion consistency of the feature matching pairs to adapt to the motion changes of the feature points.

[0057] In the process of mine excavation navigation with multi-sensor fusion, in order to further improve the accuracy and reliability of feature matching, it is necessary to screen and optimize the feature matching pairs, and dynamically adjust the search range. Specifically, according to the confidence of the feature matching pairs, the matching pairs with confidence lower than the preset threshold are eliminated, and the matching pairs with higher confidence are retained, so as to effectively remove the feature points of mismatching and improve the reliability of the matching results. For example, if the confidence threshold is set to 0.8, only the matching pairs with confidence higher than 0.8 will be retained; based on the motion consistency of the remaining feature matching pairs, the search step of the second local search range is dynamically adjusted. If the motion consistency of the feature matching pairs is high, the search step can be appropriately reduced to improve the matching accuracy; if the consistency is low, the search step can be appropriately increased to expand the search range. For example, if the motion direction and speed of the feature matching pairs are very similar, the search step can be reduced to 0.5 times of the original; if the consistency is low, it can be increased to 1.5 times of the original.

[0058] The second local search range is redefined with the matching pair with the highest spatial consistency among the remaining feature matching pairs as the center to ensure that the search range is always concentrated around the most reliable feature points, thereby further improving the matching accuracy and efficiency. For example, if a feature matching pair has the highest spatial consistency, the search range will be expanded around the matching pair and optimized according to the adjusted search step size; by screening and optimizing feature matching pairs, the matching position of the feature points can be more accurately located, thereby improving the accuracy and robustness of feature matching.

[0059] Furthermore, based on the motion consistency of the feature matching pair, the search step length of the second local search range is dynamically adjusted. The method of the present application includes: The adjustment amount of the search step length of the second local search range ;in, The adjustment amount for characterizing the search step length of the second local search range, and is an adjustment factor used to balance the weights of confidence and motion consistency, is the confidence of the ith feature match pair, ranging from [0, 1], M is the total number of remaining feature matches, is the motion consistency angle of the i-th feature matching pair; Used to quantify motion consistency.

[0060] Specifically, the adjustment amount of the search step length of the second local search range is It means that during the feature matching process, the increment or decrement of the search step size is dynamically adjusted according to the confidence and motion consistency of the feature matching pair. This adjustment is used to optimize the search efficiency and matching accuracy. and is an adjustment coefficient, which is a parameter used to control the influence of confidence and motion consistency on the search step length. Its value range is [0, 1]. The closer the adjustment coefficient is to 1, the greater the influence of confidence and motion consistency on the search step length; the closer it is to 0, the smaller the influence; is the confidence of the i-th feature matching pair, which is a quantitative indicator of the reliability or matching quality of the i-th feature matching pair. The higher the confidence, the higher the reliability of the feature matching pair. Refers to the similarity of the motion direction of the i-th feature matching pair. If the motion directions of multiple matching pairs are similar, they are considered to have high motion consistency; Quantify motion consistency It refers to converting the motion consistency angle into numerical form for easy calculation and comparison.

[0061] In the process of mine excavation navigation with multi-sensor fusion, in order to adapt to the motion changes of feature points and improve the accuracy of feature matching, it is necessary to dynamically adjust the search step size according to the confidence and motion consistency of the feature matching pair. Specifically, the adjustment amount of the search step size of the second local search range is ; Add the calculated adjustment amount to the original search step to obtain a new search step; By dynamically adjusting the search step, the search efficiency and matching accuracy can be balanced. When the movement direction and speed of the feature points are relatively consistent, reducing the search step can improve the matching accuracy; when the movement direction and speed of the feature points change greatly, increasing the search step can expand the search range, improve the matching success rate, adapt to the movement changes of the feature points, more accurately locate the matching position of the feature points, and improve the robustness and adaptability of the mine excavation navigation system.

[0062] Furthermore, the present application method also includes: The first local search range is nested in the dynamic search window to set a first nested combination; the second local search range is nested in the dynamic search window to set a second nested combination; the dynamic search window is globally constrained by the first nested combination and the second nested combination.

[0063] Specifically, nested combination refers to embedding a local search range (such as the first local search range or the second local search range) into a larger search window (such as a dynamic search window), and constraining and optimizing the search process through combination. The purpose of nested combination is to coordinate search strategies at different levels to improve matching efficiency and accuracy; global constraint refers to imposing restrictions on the entire dynamic search window through nested combination to ensure that the search process remains consistent and reliable on a global scale. Global constraint can prevent the search range from being too large or too small, thereby avoiding mismatches or missed matches.

[0064] In the process of mine excavation navigation with multi-sensor fusion, in order to further optimize the search strategy of feature matching, it is necessary to nest the local search range into the dynamic search window and coordinate the search process through global constraints. Specifically, the first local search range is nested into the dynamic search window to form a first nested combination. The first local search range is usually adjusted based on the direction and amplitude of the optical flow vector, which is suitable for fast-moving feature points. Through the nested combination, it can be ensured that the first local search range is searched within the range of the dynamic search window, and the information of the optical flow vector is used to improve the search efficiency; the second local search range is nested into the dynamic search window to form a second nested combination. The second local search range is usually adjusted based on the spatial consistency and confidence of the feature matching pair, which is suitable for scenarios requiring higher precision matching. Through the nested combination, the search range can be further optimized within the range of the dynamic search window to improve the matching accuracy.

[0065] Through the first nested combination and the second nested combination, the dynamic search window is globally constrained. The role of the global constraint is to ensure that within the dynamic search window, the search process takes into account both fast-moving feature points (through the first local search range) and the need for high-precision matching (through the second local search range). For example, if the first local search range is enlarged due to the large amplitude of the optical flow vector, the second local search range can be adjusted through spatial consistency constraints to ensure that the search range is not too large or too small.

[0066] Through nested combinations and global constraints, on the one hand, the two strategies of the first nested combination and the second nested combination are coordinated within the dynamic search window, and the search range is gradually narrowed within the dynamic search window, reducing unnecessary calculations, thereby improving the efficiency of feature matching and ensuring that the search process is both efficient and accurate. On the other hand, global constraints can prevent the search range from becoming unreasonable due to local adjustments, thereby avoiding mismatches or missed matches, ensuring that feature matching maintains consistency and reliability on a global scale, improving the accuracy and robustness of feature matching, and ensuring the efficient operation of the mine excavation navigation system.

[0067] In summary, the beneficial effects of the embodiments of the present application are: The image information uploaded by the image monitoring device and the point cloud information uploaded by the laser radar are assisted in the calibration using inertial navigation data to determine the first dynamic calibration feature and the second dynamic calibration feature; the motion posture between consecutive frames is captured to compensate the search area for image feature matching; key feature points are extracted in the search area, and projection matching is performed with the image information and the point cloud information in the inertial navigation coordinate system to generate an initial feature matching set; based on the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization goal, the extrinsic parameter matrix is ​​iteratively updated; the initial feature matching set is verified using the updated extrinsic parameter matrix to obtain the image feature matching set. The present application provides an image feature matching method and system for multi-sensor fusion navigation, thereby achieving the capture of motion posture between consecutive frames and reducing the image feature matching error caused by motion. Furthermore, through the first dynamic calibration feature and the second dynamic calibration feature, combined with the joint optimization goal, the extrinsic parameter matrix is ​​iteratively updated to gradually approach the optimal solution, thereby improving the technical effect of image feature matching accuracy and stability.

[0068] Embodiment 2 Based on the same inventive concept as the image feature matching method for multi-sensor fusion navigation in the aforementioned embodiment, Figure 2 As shown, an embodiment of the present application provides an image feature matching system for multi-sensor fusion navigation, wherein the system includes: The auxiliary calibration module M100 is used to use the inertial navigation data to perform auxiliary calibration on the image information uploaded by the image monitoring device to determine the first dynamic calibration feature; and use the inertial navigation data to perform auxiliary calibration on the point cloud information uploaded by the laser radar to determine the second dynamic calibration feature.

[0069] The motion posture capture module M200 is used to capture the motion posture between consecutive frames and compensate the search area of ​​image feature matching.

[0070] The projection matching module M300 is used to extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set.

[0071] The iterative updating module M400 is used to iteratively update the extrinsic parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature in combination with a joint optimization objective.

[0072] The verification module M500 is used to verify the initial feature matching set using the updated extrinsic parameter matrix to obtain an image feature matching set.

[0073] Furthermore, the iterative update module M400 is also used to execute the following method: A joint optimization goal is set to fuse the reprojection error corresponding to the image monitoring device and the spatial matching error corresponding to the laser radar.

[0074] The reprojection error term and the spatial matching error term in the joint optimization objective are modeled as factor nodes in a factor graph, the factor graph is optimized, and the state variables of each factor node are combined to set the external parameter matrix.

[0075] Furthermore, the iterative update module M400 is also used to execute the following method: The image monitoring device and the laser radar are timestamped and a time synchronization mechanism is set.

[0076] Based on the time synchronization mechanism, the asynchronous data is interpolated to generate a synchronous data stream, wherein the synchronous data stream includes inertial navigation data.

[0077] Furthermore, the motion posture capture module M200 is used to execute the following method: According to the preset sliding window, determine the Frame to The image information and point cloud information of the frame, where N is a positive integer divisible by 2.

[0078] Based on Frame to The image information and point cloud information of the frame are used to determine the first motion transformation relationship characteristics and the second motion transformation relationship characteristics between adjacent frames.

[0079] A search area for image feature matching is compensated according to the first motion transformation relationship feature and the second motion transformation relationship feature.

[0080] Furthermore, the motion posture capturing module M200 is also used to execute the following method: Based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion.

[0081] Based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion.

[0082] Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determine the first motion transformation relationship characteristics.

[0083] Furthermore, the motion posture capturing module M200 is also used to execute the following method: Based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion.

[0084] Based on Frame and The image information and point cloud information of the frame are used to determine the Relative motion.

[0085] Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determine the second motion transformation relationship characteristics.

[0086] Furthermore, the motion posture capturing module M200 is also used to execute the following method: Based on the first motion transformation relationship feature, the first Frame to The first motion transformation matrix of the frame .

[0087] Based on the second motion transformation relationship feature, the first Frame to The second motion transformation matrix of the frame .

[0088] Based on the preset sliding window, through the first motion transformation matrix , the second motion transformation matrix , configure the dynamic search window.

[0089] Furthermore, the projection matching module M300 is also used to perform the following method: Based on the key feature points, a feature descriptor is configured in the search area.

[0090] An optical flow vector is generated using the key feature points and feature descriptors.

[0091] A first local search range is set according to the optical flow vector.

[0092] Furthermore, the projection matching module M300 is also used to perform the following method: Based on the direction of the optical flow vector, the predicted position coordinates of the key feature point in the next frame are determined.

[0093] Based on the magnitude of the optical flow vector, the search step size of the first local search range is adjusted, and the first local search range is updated with the predicted position coordinates as the center.

[0094] Furthermore, the projection matching module M300 is also used to perform the following method: The adjustment amount of the search step length of the first local search range .

[0095] in, The adjustment amount for characterizing the search step length of the first local search range, S base Refers to the pre-set reference step value in image feature matching. Used to characterize the magnitude of the optical flow vector, Used to characterize the maximum amplitude of the optical flow vector, It is an adjustment coefficient used to control the influence of the optical flow vector amplitude on the search step size, and its value range is [0, 1].

[0096] Furthermore, the projection matching module M300 is also used to perform the following method: Based on the key feature points, a multi-scale hierarchy is configured in the search area.

[0097] According to the multi-scale levels and feature descriptors, feature matching pairs are set.

[0098] A second local search range is set according to the spatial consistency of the feature matching pairs.

[0099] Furthermore, the projection matching module M300 is also used to perform the following method: According to the confidence of the feature matching pairs, feature matching pairs with a confidence level below a preset threshold are eliminated to determine the remaining feature matching pairs.

[0100] Based on the motion consistency of the feature matching pairs, the search step length of the second local search range is dynamically adjusted, and the second local search range is updated with the feature matching pair with the highest spatial consistency among the remaining feature matching pairs as the center.

[0101] Furthermore, the projection matching module M300 is also used to perform the following method: The adjustment amount of the search step length of the second local search range .

[0102] in, The adjustment amount for characterizing the search step length of the second local search range, and is an adjustment factor used to balance the weights of confidence and motion consistency, is the confidence of the ith feature match pair, ranging from [0, 1], M is the total number of remaining feature matches, is the motion consistency angle of the i-th feature matching pair; Used to quantify motion consistency.

[0103] Furthermore, the projection matching module M300 is also used to perform the following method: The first local search range is nested in the dynamic search window to set a first nested combination.

[0104] The second local search range is nested in the dynamic search window to set a second nested combination.

[0105] The dynamic search window is globally constrained by the first nested combination and the second nested combination.

[0106] In summary, any step can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor, and no unnecessary restrictions are made here.

[0107] Furthermore, the above-mentioned technical scheme only reflects the preferred technical scheme of the technical scheme of the embodiment of the present application. Some changes that may be made to some parts thereof by technicians in this technical field all reflect the novel principles of the embodiment of the present application. Obviously, technicians in this field can make various changes and modifications to the present application without departing from the scope of the present application.

Claims

1. An image feature matching method for multi-sensor fusion navigation, characterized in that: The method comprises: Using the inertial navigation data to assist in calibrating the image information uploaded by the image monitoring device, and determining the first dynamic calibration feature; using the inertial navigation data to assist in calibrating the point cloud information uploaded by the laser radar, and determining the second dynamic calibration feature; Capture the motion gestures between consecutive frames and compensate the search area for image feature matching; Extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set; Iteratively updating the extrinsic parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature in combination with a joint optimization objective; The updated extrinsic parameter matrix is ​​used to verify the initial feature matching set to obtain an image feature matching set.

2. The image feature matching method for multi-sensor fusion navigation according to claim 1, characterized in that: Setting a joint optimization target to fuse the reprojection error corresponding to the image monitoring device and the spatial matching error corresponding to the laser radar; The reprojection error term and the spatial matching error term in the joint optimization objective are modeled as factor nodes in a factor graph, the factor graph is optimized, and the state variables of each factor node are combined to set the external parameter matrix.

3. The image feature matching method for multi-sensor fusion navigation according to claim 2, characterized in that: The method further comprises: Performing time stamp synchronization on the image monitoring device and the laser radar, and setting a time synchronization mechanism; Based on the time synchronization mechanism, the asynchronous data is interpolated to generate a synchronous data stream, wherein the synchronous data stream includes inertial navigation data.

4. The image feature matching method for multi-sensor fusion navigation according to claim 1, characterized in that: Capturing motion gestures between consecutive frames and compensating for a search area for image feature matching, the method comprises: According to the preset sliding window, determine the Frame to The image information and point cloud information of the frame, where N is a positive integer divisible by 2; Based on Frame to The image information and point cloud information of the frames are used to determine the first motion transformation relationship characteristics and the second motion transformation relationship characteristics between adjacent frames; A search area for image feature matching is compensated according to the first motion transformation relationship feature and the second motion transformation relationship feature.

5. The image feature matching method for multi-sensor fusion navigation according to claim 4, characterized in that: Based on Frame to The method comprises: Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determine the first motion transformation relationship characteristics.

6. The image feature matching method for multi-sensor fusion navigation according to claim 5, characterized in that: Based on Frame to The method comprises: Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Based on Frame and The image information and point cloud information of the frame are used to determine the relative motion; Traverse the Frame to The image information and point cloud information of the frame are Relative motion, Relative movement to Relative motion, determine the second motion transformation relationship characteristics.

7. The image feature matching method for multi-sensor fusion navigation according to claim 4, characterized in that: Compensating a search area for image feature matching according to the first motion transformation relationship feature and the second motion transformation relationship feature, the method comprising: Based on the first motion transformation relationship feature, the first Frame to The first motion transformation matrix of the frame ; Based on the second motion transformation relationship feature, the first Frame to The second motion transformation matrix of the frame ; Based on the preset sliding window, through the first motion transformation matrix , the second motion transformation matrix , configure the dynamic search window.

8. The image feature matching method for multi-sensor fusion navigation according to claim 7, characterized in that: Extracting key feature points in the search area, the method further includes: Based on the key feature points, configuring a feature descriptor in the search area; Generate an optical flow vector using the key feature points and feature descriptors; A first local search range is set according to the optical flow vector.

9. The image feature matching method for multi-sensor fusion navigation according to claim 8, characterized in that: Determining the predicted position coordinates of the key feature point in the next frame based on the direction of the optical flow vector; Based on the magnitude of the optical flow vector, the search step size of the first local search range is adjusted, and the first local search range is updated with the predicted position coordinates as the center.

10. The image feature matching method for multi-sensor fusion navigation according to claim 9, characterized in that: Based on the magnitude of the optical flow vector, adjusting the search step size of the first local search range, the method comprising: The adjustment amount of the search step length of the first local search range ; in, The adjustment amount for characterizing the search step length of the first local search range, S base Refers to the pre-set reference step value in image feature matching. Used to characterize the magnitude of the optical flow vector, Used to characterize the maximum amplitude of the optical flow vector, It is an adjustment coefficient used to control the influence of the optical flow vector amplitude on the search step size, and its value range is [0, 1].

11. The image feature matching method for multi-sensor fusion navigation according to claim 8, characterized in that: The method comprises: Based on the key feature points, configuring a multi-scale hierarchy in the search area; According to the multi-scale level and the feature descriptor, setting feature matching pairs; A second local search range is set according to the spatial consistency of the feature matching pairs.

12. The image feature matching method for multi-sensor fusion navigation according to claim 11, characterized in that: According to the confidence of the feature matching pairs, the feature matching pairs with a confidence lower than a preset threshold are eliminated to determine the remaining feature matching pairs; Based on the motion consistency of the feature matching pairs, the search step length of the second local search range is dynamically adjusted, and the second local search range is updated with the feature matching pair with the highest spatial consistency among the remaining feature matching pairs as the center.

13. The image feature matching method for multi-sensor fusion navigation according to claim 12, characterized in that: Based on the motion consistency of the feature matching pair, dynamically adjusting the search step length of the second local search range, the method comprises: The adjustment amount of the search step length of the second local search range ; in, The adjustment amount for characterizing the search step length of the second local search range, and is an adjustment factor used to balance the weights of confidence and motion consistency, is the confidence of the ith feature match pair, ranging from [0, 1], M is the total number of remaining feature matches, is the motion consistency angle of the i-th feature matching pair; Used to quantify motion consistency.

14. The image feature matching method for multi-sensor fusion navigation according to claim 11, characterized in that: Nesting the first local search range in the dynamic search window to set a first nested combination; Nesting the second local search range in the dynamic search window to set a second nested combination; The dynamic search window is globally constrained by the first nested combination and the second nested combination.

15. An image feature matching system for multi-sensor fusion navigation, characterized in that: The system is used to implement the image feature matching method for multi-sensor fusion navigation according to any one of claims 1 to 14, and comprises: An auxiliary calibration module is used to use the inertial navigation data to perform auxiliary calibration on the image information uploaded by the image monitoring device to determine the first dynamic calibration feature; and use the inertial navigation data to perform auxiliary calibration on the point cloud information uploaded by the laser radar to determine the second dynamic calibration feature; Motion gesture capture module, used to capture the motion gesture between consecutive frames and compensate the search area for image feature matching; A projection matching module is used to extract key feature points in the search area, and perform projection matching with the image information and the point cloud information in an inertial navigation coordinate system to generate an initial feature matching set; An iterative updating module, configured to iteratively update an external parameter matrix based on the first dynamic calibration feature and the second dynamic calibration feature in combination with a joint optimization objective; The verification module is used to verify the initial feature matching set using the updated extrinsic parameter matrix to obtain the image feature matching set.

Images