A heterogeneous-based unmanned aerial vehicle vision-inertial navigation system optimization and acceleration method and system

Abstract

The application provides a heterogeneous-based unmanned aerial vehicle vision-inertial navigation system optimization and acceleration method and system, comprising: S1 output of a reliable feature point set is a direct input object for initial scale estimation of S2. The feature points established by S1 become seed features for tracking of S3. S3 uses hardware-accelerated optical flow for tracking in subsequent frames based on the feature point positions determined by S1. The tracking results of S3 enter S4 for bidirectional optical flow consistency verification, and the feature points that fail to track are removed, and high-reliability feature trajectories are output. The feature trajectories verified by S4 are input to S5 feature management for dynamic density control and quality evaluation. The feature distribution managed by S5 is used as a candidate set for S6 feature selection, S6 selects an optimal subset based on information gain evaluation for back-end optimization. The selection strategy of S6 is adjusted by the adaptive mechanism of S7, and the overall system parameters are adjusted according to the calculation resource state of S8.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle inertial navigation, in particular to a heterogeneous unmanned aerial vehicle visual-inertial navigation system optimization and acceleration method and system. BACKGROUND

[0002] With the diversification of unmanned aerial vehicle application scenarios, the autonomous navigation capability of unmanned aerial vehicles is facing increasingly severe challenges, especially on resource-constrained unmanned aerial vehicle platforms. At present, the visual-inertial navigation system (VINS) has the following technical problems and shortcomings:

[0003] Low computational efficiency: although the VINS system based on nonlinear optimization (such as VINS-Mono and ORB-SLAM) has high accuracy and robustness, it has high computational complexity, and it is difficult to run in real time on resource-limited unmanned aerial vehicle platforms. Feature extraction and optical flow calculation modules occupy a large amount of computing resources, resulting in slow system response speed.

[0004] Poor environmental adaptability: the existing VINS system performs poorly in low-texture environments and fast motion scenarios, initialization is prone to failure, feature point tracking is unstable, and the system is limited in complex environments.

[0005] Unreasonable resource allocation: the traditional VINS system does not consider the difference in the contribution of feature points to positioning accuracy, and simply uses all tracked feature points for back-end optimization, resulting in waste of computing resources and low processing efficiency. SUMMARY

[0006] The technical problem to be solved by the present application is how to improve the performance of the unmanned aerial vehicle navigation system.

[0007] The present application solves the above technical problems by the following technical means:

[0008] The present application provides a heterogeneous unmanned aerial vehicle visual-inertial navigation system optimization and acceleration method, comprising the following steps:

[0009] S1. Fusion of optical flow and descriptor: track feature points between consecutive frames through optical flow algorithm to obtain dense motion information, extract descriptors for key points and match and verify the motion information, and then output a reliable feature point set;

[0010] S2. Initial scale estimation and optimization: use the set relationship of the reliable feature point set among multiple frames, combine with IMU pre-integration data, solve the scale factor through visual-inertial joint optimization, and construct motion geometric constraints;

[0011] S3. Feature tracking: based on the reliable feature point set output in step S1, identify key frames to form seed features for tracking, and track in subsequent frames based on the seed features;

[0012] S4. Bidirectional optical flow consistency check: based on the tracking results of step S3, perform bidirectional optical flow consistency check, and filter out high-reliability feature tracks according to a preset tracking error threshold;

[0013] S5. Feature management: based on the feature tracks output by step S4, perform dynamic density control and quality assessment of feature points, and output the managed feature points;

[0014] S6. Feature selection: based on information gain evaluation, calculate the contribution of each feature point output by step S5 to system state estimation, and select a subset with the largest amount of information for backend optimization;

[0015] S7. Adaptive feature screening: adaptive feature screening is performed on the subset selected in step S6, and the feature selection strategy is dynamically adjusted;

[0016] S8. Trade-off strategy of computing resources and precision: dynamically adjust system parameters according to system resources.

[0017] Further, the step S1 is to perform multi-level feature fusion on ARS45, first use hardware accelerated KLT optical flow algorithm for preliminary tracking, then verify through BRIEF descriptor matching, finally introduce geometric consistency constraint to remove outliers, and output 200 high-quality feature points per frame.

[0018] Further, the step S2 specifically comprises: obtaining the optimal scale and initial state by solving the following optimization objective function:

[0019]

[0020] Wherein the visual re-projection residual can be represented as:

[0021] The IMU pre-integration residual can be represented as:

[0022]

[0023] Wherein is the scale factor, is the system state variable, and are the visual re-projection residual and the IMU pre-integration residual, respectively, is a robust kernel function, is the covariance matrix of IMU measurement, is the image feature point observation value, is the camera projection function, is the camera rotation matrix, is the camera translation vector, These are the coordinates of the 3D landmark point. For the k-th IMU frame, rotate to the world coordinate system. Let k be the position of the k-th IMU frame in the world coordinate system. Let the velocity of the k-th IMU frame be in the world coordinate system. The pre-integral value for the position from frame k to frame (k+1) is... For time intervals, This represents the gravity vector in the world coordinate system. For the velocity pre-integral quantity from frame k to frame (k+1), Let k be the rotation quaternion from the world coordinate system to the k-th IMU frame. For the rotation pre-integral quantity from frame k to frame (k+1), This is the quaternion multiplication operator.

[0024] Furthermore, step S5 specifically includes:

[0025] (1) Dynamic density control: The feature point density is adaptively adjusted according to the scene complexity and system state; the density is appropriately reduced in feature-rich areas and the sampling density is increased in feature-sparse areas to achieve uniform feature distribution;

[0026] (2) Multi-scale feature selection: Extract features at different levels of the image pyramid to ensure that both near and far targets have sufficient feature coverage and improve the system’s adaptability to scale changes;

[0027] (3) Feature quality scoring mechanism: Taking into account factors such as corner response value, gradient distribution and tracking stability, a quality score is assigned to each feature point, and high-quality features are retained first;

[0028] (4) Predictive feature management: Combined with camera motion prediction, feature points that may leave the field of view are identified in advance, and new features are extracted in the corresponding areas in a timely manner to avoid feature shortage.

[0029] Furthermore, the characteristic is that step S6 specifically comprises:

[0030] Feature point information gain evaluation is based on the following mathematical model: For feature points Its information gain Defined as:

[0031]

[0032] in It is a matrix eigenvalues, It is the dimension of the state vector. For feature points The corresponding Jacobian matrix, To observe the noise covariance matrix, `trace()` represents the trace of the matrix.

[0033] Furthermore, step S7 dynamically adjusts the feature selection strategy:

[0034] (1) Scene complexity awareness: By evaluating the number, distribution and quality of feature points in the current scene, the feature selection threshold is automatically adjusted. In feature-rich environments, stricter screening criteria are adopted, and selection conditions are relaxed in feature-sparse scenes.

[0035] (2) Motion pattern adaptation: Adjust feature selection preferences according to the UAV's motion state;

[0036] (3) Multi-objective optimization strategy: The feature selection problem is modeled as a multi-objective optimization problem, considering both information gain maximization and feature distribution uniformity; a weighted method is used to transform the multi-objective problem into a single-objective optimization problem:

[0037]

[0038] Among them, uniform distribution measure Defined as:

[0039]

[0040] in, For the set of feature points, For the i-th feature point, For the information gain of feature points, Weighting coefficients for information gain and distribution uniformity (0≤ ≤1), The number of image partitions, The number of feature points in the k-th grid.

[0041] (4) Computing resource awareness and adjustment: Real-time monitoring of system computing load, automatically reducing the number of optimized feature points when computing resources are tight, and appropriately increasing feature points when resources are sufficient, to achieve adaptive resource allocation.

[0042] Furthermore, the trade-off strategy of S8 is as follows:

[0043] (1) Multi-level optimization strategy: The back-end optimization is divided into two stages: coarse optimization and fine optimization. Coarse optimization uses fewer feature points to converge quickly and provide an initial estimate. Fine optimization uses more feature points for fine adjustment when computing resources allow.

[0044] (2) Dynamic management of key frames: Based on information theory, a key frame selection strategy is designed to retain only key frames that contribute significantly to the system state estimation; a key frame redundancy evaluation mechanism is introduced to remove redundant historical key frames in a timely manner and control the optimization window size.

[0045] (3) Variable optimization window: The sliding window size is dynamically adjusted according to the system state and computing resources; in highly dynamic scenarios or when feature tracking is unstable, the window is appropriately enlarged to improve robustness; in stable scenarios, the window is reduced to reduce the computational burden;

[0046] (4) Accuracy assurance mechanism: Design an accuracy monitoring module to evaluate the uncertainty of system state estimation in real time; when the uncertainty exceeds the preset threshold, automatically increase the number of feature points and optimize the number of iterations to ensure that the system accuracy will not decrease significantly.

[0047] This invention also provides an optimization and acceleration system for a heterogeneous UAV visual inertial navigation system, comprising:

[0048] Optical flow and descriptor fusion module: Tracks feature points between consecutive frames using optical flow algorithm to obtain dense motion information, and extracts descriptors from key points and matches and verifies them with motion information to output a reliable set of feature points.

[0049] Initial scale estimation and optimization module: Utilizing the set relationship of reliable feature point sets across multiple frames, combined with IMU pre-integrated data, the scale factor is solved through vision-inertial joint optimization to construct motion geometric constraints;

[0050] Feature tracking module: Based on the reliable set of feature points output by the optical flow and descriptor fusion module, key frames are identified, seed features for tracking are formed, and tracking is performed in subsequent frames based on the seed features;

[0051] Bidirectional optical flow consistency verification module: Based on the tracking results of the feature tracking module, it performs bidirectional optical flow consistency verification and selects highly reliable feature trajectories according to the preset tracking error threshold;

[0052] Feature Management Module: Based on the feature trajectory output by the bidirectional optical flow consistency verification module, it performs dynamic density control and quality assessment of feature points, and outputs the managed feature points;

[0053] Feature selection module: Based on information gain evaluation, calculate the contribution of each feature point output by the feature management module to the system state estimation, and select the subset with the largest information content for backend optimization;

[0054] Adaptive Feature Selection Module: Performs adaptive feature selection on subsets and dynamically adjusts the feature selection strategy;

[0055] The module for balancing computational resources and accuracy dynamically adjusts system parameters based on system resources.

[0056] Furthermore, the optical flow and descriptor fusion module performs multi-level feature fusion on the ARS45. First, it uses the hardware-accelerated KLT optical flow algorithm for preliminary tracking, then verifies it through BRIEF descriptor matching, and finally introduces geometric consistency constraints to remove outliers, outputting 200 high-quality feature points per frame.

[0057] Furthermore, the initial scale estimation and optimization module specifically involves obtaining the optimal scale and initial state by solving the following objective function:

[0058]

[0059] The visual reprojection residual can be expressed as:

[0060] The IMU pre-integration residual can be expressed as:

[0061]

[0062] in As a scale factor, For system state variables, and These are the visual reprojection residual and the IMU pre-integration residual, respectively. For robust kernel functions, The covariance matrix of the IMU measurements. For image feature point observations, For camera projection function, For the camera rotation matrix, Let be the camera translation vector. These are the coordinates of the 3D landmark point. For the k-th IMU frame, rotate to the world coordinate system. Let k be the position of the k-th IMU frame in the world coordinate system. Let the velocity of the k-th IMU frame be in the world coordinate system. The pre-integral value for the position from frame k to frame (k+1) is... For time intervals, This represents the gravity vector in the world coordinate system. For the velocity pre-integral quantity from frame k to frame (k+1), Let k be the rotation quaternion from the world coordinate system to the k-th IMU frame. For the rotation pre-integral quantity from frame k to frame (k+1), This is the quaternion multiplication operator.

[0063] The advantages of this invention are:

[0064] 1. Initialization Method for Optical Flow and Descriptor Fusion: Existing technologies typically employ a serial "detection-description-matching" process, with optical flow tracking and descriptor matching operating independently and primarily executed on the CPU. Our approach uses a hardware-accelerated KLT optical flow algorithm for initial tracking in the first layer; the second layer verifies matching using BRIEF descriptors; and the third layer introduces geometric consistency constraints to eliminate outliers through fundamental matrix estimation. This multi-layered fusion strategy achieves a complementary advantage in optical flow efficiency and descriptor robustness, significantly improving the initialization success rate in low-texture environments and motion-blurred scenes.

[0065] 2. Feature point selection algorithm based on information gain: The algorithm evaluates and filters feature points based on their contribution to the system state estimation, selects the subset with the largest amount of information to participate in the backend optimization, and achieves the optimal balance between computing resources and positioning accuracy.

[0066] 3. Adaptive Feature Management Strategy: Combining scene complexity perception, motion pattern adaptation, and computing resource monitoring, the density, distribution, and selection criteria of feature points are dynamically adjusted to improve the robustness and efficiency of the system in complex environments. Attached Figure Description

[0067] Figure 1 This refers to the algorithm architecture corresponding to the method in the embodiments of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] like Figure 1 As shown in the figure, this embodiment provides a method for optimizing and accelerating a heterogeneous UAV visual inertial navigation system, which includes the following steps:

[0070] Step 1: Optical Flow and Descriptor Fusion

[0071] This embodiment proposes an initialization method combining optical flow tracking and descriptor matching. The core idea of ​​this method is to leverage the high efficiency of optical flow algorithms and the high robustness of descriptor matching to achieve a complementary advantage. Specifically, the system first tracks feature points across consecutive frames using a hardware-accelerated optical flow algorithm to acquire dense motion information; simultaneously, it extracts descriptors from key points and performs matching to verify the optical flow tracking results and eliminate erroneous matches. In this embodiment, dense motion information refers to the detailed motion vector field obtained at the image pixel level through the optical flow algorithm. Specifically, the optical flow algorithm, based on the assumption of gray-level invariance, calculates the motion velocity and direction of each pixel in the image across consecutive frames. Compared to traditional feature point matching, which can only obtain motion information from a few hundred sparse feature points, the optical flow algorithm can provide motion estimation for tens of thousands of pixels in the image, forming a dense motion vector field. This dense information helps to more accurately understand the overall motion pattern of the scene, especially providing more motion constraints in textured areas. Optical flow tracking is based on the assumption that the gray level of the same point on the image remains unchanged over a short period of time, i.e.:

[0072]

[0073] Performing a Taylor expansion and rearranging the equation, we obtain the Lucas-Kanade optical flow equation:

[0074]

[0075] in x, y For image plane coordinates, t Let Δx, Δy, and Δt be time, and Δt be the corresponding changes. For pixel motion speed (optical flow), , and These are the partial derivatives of the image in space and time, respectively.

[0076] The fusion strategy employs a hierarchical approach based on consistency checks: the first layer uses a hardware-accelerated KLT optical flow algorithm for initial tracking; the second layer performs matching verification using BRIEF descriptors; and the third layer introduces geometric consistency constraints, eliminating outliers through fundamental matrix estimation. This multi-layered fusion strategy significantly improves the robustness and accuracy of initialization, especially performing well in low-texture and motion-blurred scenes.

[0077] Step 2: Initial Scaling Estimation and Optimization

[0078] Monocular vision systems suffer from scale uncertainty, requiring IMU information for scale recovery. This embodiment improves upon the traditional VINS-Mono scale estimation method by proposing an initial scale estimation algorithm based on joint vision-inertial optimization. This algorithm constructs an optimization problem containing scale factors using IMU pre-integration results, and combines visual geometric constraints to obtain the optimal scale and initial state by solving the following objective function:

[0079]

[0080] The visual reprojection residual can be expressed as:

[0081] The IMU pre-integration residual can be expressed as:

[0082]

[0083] in As a scale factor, For system state variables, and These are the visual reprojection residual and the IMU pre-integration residual, respectively. For robust kernel functions, The covariance matrix of the IMU measurements. For image feature point observations, For camera projection function, For the camera rotation matrix, Let be the camera translation vector. These are the coordinates of the 3D landmark point. For the k-th IMU frame, rotate to the world coordinate system. Let k be the position of the k-th IMU frame in the world coordinate system. Let the velocity of the k-th IMU frame be in the world coordinate system. The pre-integral value for the position from frame k to frame (k+1) is... For time intervals, This represents the gravity vector in the world coordinate system. For the velocity pre-integral quantity from frame k to frame (k+1), Let k be the rotation quaternion from the world coordinate system to the k-th IMU frame. For the rotation pre-integral quantity from frame k to frame (k+1), This is the quaternion multiplication operator.

[0084] To improve optimization efficiency, the system adopts a stepwise refinement strategy, first performing a coarse scale estimation, and then performing fine adjustments through nonlinear optimization.

[0085] The main output of step two is the scaling factor. The initial state estimates and these results provide a correct scaling benchmark for the entire VINS system. The scaling factor estimated in step two affects subsequent state estimation and optimization processes, establishing a correct metric benchmark for the system.

[0086] Step 3: Feature Tracking

[0087] This embodiment proposes a feature tracking scheme based on hardware-accelerated optical flow to replace the traditional "detection-description-matching" process. The core idea of ​​this scheme is to perform comprehensive feature detection on key frames, while directly tracking existing feature points on non-key frames using optical flow methods, thus significantly reducing the frequency of feature extraction.

[0088] Keyframe identification can utilize average disparity, which is the average disparity of feature points calculated across consecutive frames. If the average disparity of the tracked features exceeds a preset threshold, the current frame is marked as a keyframe. This method ensures the system can capture significant motion changes, thereby improving tracking accuracy. Comprehensive feature detection is performed on keyframes, while existing feature points are directly tracked using optical flow methods on non-keyframes.

[0089] Adaptive Feature Detection: An improved FAST corner detection algorithm is used on keyframes, combined with image gradient information to evaluate the quality of feature points, ensuring that the feature points are evenly distributed and have high discriminative power.

[0090] The specific steps for obtaining image gradient information are: obtaining gradient input image → hardware acceleration unit → parallel gradient calculation → gradient information output. Feature point acquisition method:

[0091] 1. Hardware parallel computation of image gradients (I x , I y );

[0092] 2. Evaluate corner quality based on gradient magnitude and direction;

[0093] 3. FAST detection + gradient quality scoring → final feature points.

[0094] In this embodiment, the hardware accelerator utilizes the Lucas-Kanade optical flow algorithm, which is hardware-accelerated (the ARS45's built-in Image Signal Processor (ISP) handles image preprocessing, the Vector Processing Unit (VPU) is dedicated to feature extraction and optical flow calculation, multi-point parallel processing is achieved using the SIMD instruction set, and zero-copy data transfer is achieved through a DMA controller). This algorithm tracks feature points on a multi-scale image pyramid, effectively addressing large parallax and motion blur. The core formula for optical flow tracking is:

[0095]

[0096] in The optical flow vector is obtained by solving the following minimization problem:

[0097]

[0098] Iterative solution of the equation:

[0099]

[0100] in, Let be the grayscale value of the image at coordinates (x, y) in frame t. For image coordinates, Represents a pixel window surrounding a feature point. Let represent the partial derivatives of the image grayscale with respect to the x-direction, y-direction, and time t, respectively. The increment of the optical flow vector is determined iteratively as follows: n is the number of iterations.

[0101] It should be noted that the optical flow in step one is mainly used for initial feature point acquisition and verification, and is part of the "optical flow and descriptor fusion" process, with the aim of establishing a reliable set of feature points. The optical flow in step three is specifically used for feature point tracking, continuously tracking based on the feature points established in step one. The Lucas-Kanade optical flow algorithm is used to track feature points on a multi-scale image pyramid. Step one is used for feature point initialization, and step three is used for feature point tracking. Step one processes the original image frame, while step three processes tracking based on known feature points.

[0102] Step 4: Bidirectional optical flow consistency check and feature point periodization

[0103] This step implements forward and reverse optical flow calculations (reducing errors compared to traditional unidirectional calculations), and filters reliable points by tracking error thresholds, effectively eliminating occlusion and mismatched points. A feature point aging mechanism is introduced to track the lifecycle of each feature point, prioritizing the retention of feature points that have been stably tracked for a long time, thus improving system stability. (This system filters reliable points by setting tracking error thresholds, comprehensively considering the differences between forward and reverse optical flow calculations, enabling more accurate judgment of feature point reliability, effectively removing points with large errors, and thereby improving the accuracy of the entire navigation system.)

[0104] It should be noted that step three uses an optical flow algorithm to track feature points and outputs the tracked feature point positions. Step four performs a "bidirectional optical flow consistency check" on the tracking results of step three. The bidirectional optical flow calculation specifically involves calculating the forward optical flow (from the previous frame to the current frame) and the reverse optical flow (from the current frame back to the previous frame), comparing the consistency of the two calculation results, setting a tracking error threshold, eliminating feature points with poor consistency, and outputting a highly reliable feature trajectory.

[0105] Step 5: Balance the quantity and quality of feature points

[0106] The quantity and quality of feature points directly affect the accuracy and computational efficiency of the VINS system. This embodiment designs a dynamic feature management strategy to optimize the utilization of computational resources while ensuring positioning accuracy.

[0107] 1. Dynamic Density Control: The feature point density is adaptively adjusted based on scene complexity and system state. Density is appropriately reduced in feature-rich regions and increased in feature-sparse regions to achieve uniform feature distribution. Specifically, if (regional feature point density > high density threshold), the FAST corner detection threshold is increased; if (regional feature point density <= low density threshold), the FAST corner detection threshold is decreased.

[0108] 2. Multi-scale feature selection: Features are extracted at different levels of the image pyramid to ensure sufficient feature coverage for both near and far targets, improving the system's adaptability to scale changes. (Robust feature coverage for near and far targets is achieved through hierarchical detection and cross-level matching).

[0109] 3. Feature quality scoring mechanism: Taking into account factors such as corner response value, gradient distribution, and tracking stability, a quality score is assigned to each feature point, and high-quality features are retained first.

[0110] 4. Predictive Feature Management: By combining camera motion prediction, feature points that may leave the field of view can be identified in advance, and new features can be extracted in the corresponding areas in a timely manner to avoid feature shortage.

[0111] Eliminating mismatches can reduce computational load while ensuring sufficient coverage, thus guaranteeing the accuracy of the calculation results while reducing computational load.

[0112] Step five involves extracting features from the image pyramid at different levels to ensure feature coverage of near and far targets, achieving uniform feature distribution through "layered detection + cross-layer matching." Step three executes the Lucas-Kanade algorithm at multiple scales to perform multi-scale optical flow calculations on the image pyramid. While the pyramids have different purposes and functions, they can share the same core functionality as step three.

[0113] Step five, the feature management phase, is an ongoing process. The specific timing of its execution is as follows:

[0114] 1. During each frame processing: based on the current camera motion state (provided by IMU and visual odometry);

[0115] 2. Predict the camera pose for the next frame: using a uniform motion model or IMU pre-integration results;

[0116] 3. Project existing feature points: Project the current feature points to the predicted position in the next frame;

[0117] 4. Identify feature points that are about to leave the field of view: determine whether the projection position exceeds the image boundary;

[0118] 5. Extract new features in the corresponding region: Force the detection of new feature points in the region that is about to lose feature coverage.

[0119] Step Six: Feature Selection Criteria and Algorithm

[0120] The computational complexity of backend optimization in a VINS system is quadratic or cubic with the number of feature points. On resource-constrained UAV platforms, effectively managing the number of feature points involved in optimization is crucial. Traditional VINS-Mono systems simply use all tracked feature points for backend optimization, failing to consider the differences in the contribution of feature points to positioning accuracy and computational resource limitations. This embodiment proposes an information gain-based feature point selection algorithm. Its core is to evaluate the contribution of each feature point to system state estimation and select the subset with the highest information content to participate in backend optimization.

[0121] Feature point information gain evaluation is based on the following mathematical model: For feature points Its information gain Defined as:

[0122]

[0123] in It is a matrix eigenvalues, It is the dimension of the state vector. For feature points The corresponding Jacobian matrix, To observe the noise covariance matrix, `trace()` represents the trace of the matrix. This model comprehensively considers factors such as the number of observations of feature points, the diversity of observation angles, and reprojection errors, quantifying the contribution of feature points to the system accuracy.

[0124] In terms of algorithm implementation, a greedy strategy is adopted to progressively select feature points with the largest information gain until a preset threshold is reached or the information gain increment falls below a specified threshold (limiting the total number of feature points participating in the optimization). To improve efficiency, the feature selection process is designed to be incremental, evaluating newly added feature points only when each new frame arrives, avoiding a complete recalculation.

[0125] Step 7: Adaptive Feature Selection Mechanism

[0126] To address the dynamic changes in the scene and motion state during drone flight, this embodiment designs an adaptive feature selection mechanism to dynamically adjust the feature selection strategy:

[0127] 1. Scene Complexity Awareness: Automatically adjusts the feature selection threshold by evaluating the quantity, distribution, and quality of feature points in the current scene. More stringent selection criteria are applied in feature-rich environments, while selection conditions are relaxed in feature-sparse scenarios.

[0128] 2. Motion Pattern Adaptation: Adjust feature selection preferences based on the drone's motion state (stationary, constant speed, acceleration, rotation, etc.). For example, prioritize feature points in the center region of the image during rapid rotation; retain more distant feature points during rapid forward movement to maintain scale consistency.

[0129] 3. Multi-objective optimization strategy: The feature selection problem is modeled as a multi-objective optimization problem, simultaneously considering maximizing information gain and ensuring uniform feature distribution. A weighted approach is used to transform the multi-objective problem into a single-objective optimization problem.

[0130]

[0131] Among them, uniform distribution measure Defined as:

[0132]

[0133] in, For the set of feature points, For the i-th feature point, For the information gain of feature points, Weighting coefficients for information gain and distribution uniformity (0≤ ≤1), The number of image partitions, The number of feature points in the k-th grid.

[0134] 4. Computational Resource Awareness and Adjustment: Real-time monitoring of system computational load, automatically reducing the number of optimized feature points when computational resources are scarce, and appropriately increasing feature points when resources are sufficient, to achieve adaptive resource allocation.

[0135] In terms of real-time monitoring of system computational load, a multi-layered monitoring strategy may be adopted. First, processing time monitoring involves monitoring the processing time of each image frame and setting thresholds (e.g., 33ms for 30fps) to monitor the execution time of modules such as front-end visual processing and back-end optimization, while also monitoring the backlog in the image processing queue. Second, hardware resource monitoring involves obtaining CPU utilization through system APIs, monitoring available memory and memory fragmentation, monitoring the utilization and temperature of acceleration units, and, on drone platforms, monitoring power consumption changes. Additionally, monitoring system status indicators includes monitoring data access cache hit rate, multi-threaded system scheduling efficiency, and data transfer load between the CPU and acceleration units.

[0136] In setting standards for reducing optimized features, it's necessary to combine the information gain calculation and quality scoring mechanisms mentioned in the previous steps. Based on the information gain-based grading standard, feature points can be divided into three priorities: high-priority features with information gain greater than threshold A are always retained; medium-priority features with information gain between thresholds B and A are retained when resources are sufficient; and low-priority features with information gain less than threshold B are eliminated first when resources are scarce. Simultaneously, feature quality scoring standards must be considered, including factors such as tracking stability, number of observations, and geometric distribution. Feature points that have been tracked stably for a long time, feature points observed in multiple keyframes, and feature points that can guarantee a uniform distribution in the image should receive higher weights.

[0137] The dynamic adjustment strategy can implement different retention strategies based on the load level: retain 95% of candidate feature points when the load is light, retain 80% of high-quality feature points when the load is moderate, and retain only 60% of core feature points when the load is severe, prioritizing system real-time performance. The adaptive threshold mechanism in step seven can be designed to reduce the number of feature points by 20% when the processing time exceeds 1.2 times the real-time threshold, and increase the number of feature points by 10% when the processing time is less than 0.8 times the real-time threshold and resources are sufficient.

[0138] It should be noted that the difference between the automatic adjustment of the feature selection threshold in step seven and step five is as follows: the dynamic density control in step five adaptively adjusts the feature point density based on scene complexity and system state. It appropriately reduces the sampling density in feature-rich regions and increases the sampling density in feature-sparse regions to achieve a uniform feature distribution. The dynamic density control in step five occurs during the front-end feature detection stage, specifically performing dynamic density control and quality evaluation on the feature trajectories output in step four. The feature distribution managed in step five serves as the candidate set for feature selection in step six. Step six selects the optimal subset from this subset based on information gain evaluation for back-end system parameter optimization. Using the optimal subset selected in steps five and six for back-end system parameter optimization reduces computational load while improving optimization effectiveness.

[0139] In contrast, the scenario complexity awareness in step seven occurs during the feature selection phase of backend optimization. Its purpose is to select which existing feature points will participate in the optimization computation. It plays a role in the screening stage before backend optimization, balancing computational resources and accuracy by adjusting the feature selection threshold criteria. The target is the existing set of feature points, ultimately selecting a subset of features for optimization.

[0140] From a temporal perspective, the entire process is as follows: First, system initialization is performed (steps one and two), followed by consistency verification (step four), then density control in step five during feature tracking, and finally, information gain selection (step six) and adaptive filtering mechanism (step seven) are performed before backend optimization. Therefore, step seven is an adaptive enhancement based on step six and occurs during the backend optimization phase.

[0141] In summary, step five controls "how many feature points to produce" on the front end, while step seven controls "which feature points to use" on the back end. Both consider the complexity of the scenario, but serve different stages and goals of the system.

[0142] Step 8: Strategy for Balancing Computational Resources and Accuracy

[0143] On UAV platforms, balancing computational resources and positioning accuracy is a core challenge in system design. Step eight dynamically adjusts system parameters based on the subset selected in step six and in conjunction with system resources. Utilizing the selected feature subset in backend optimization can reduce computational load. Specifically, feature selection in step six relies on information gain evaluation, which quantifies the contribution of feature points to system state estimation and selects the feature subset with the highest information content. The information gain evaluation formula for feature points involves calculations of the Jacobian matrix, covariance matrix, etc., and the computational complexity and memory consumption increase exponentially with the number of feature points. Therefore, feature selection must consider not only information gain but also computational resource consumption, memory usage, and real-time requirements.

[0144] Backend optimization is performed using a nonlinear least squares method (Gauss-Newton method), aiming to update state variables by minimizing the objective function (including visual reprojection residuals and IMU pre-integration residuals). A selected subset of features is used as input to the optimization objective, and the system's state variables (such as camera pose and IMU bias) are updated through joint vision-inertial optimization (visual reprojection and IMU pre-integration residuals). During optimization, the information gain of feature points directly affects their weight in the optimization process. Feature points with high information gain will occupy a larger proportion in the optimization, improving the final estimation accuracy. Under resource constraints, the size of the optimization window or the number of optimization iterations is dynamically adjusted to avoid excessive consumption of computational resources. Feature points with higher accuracy are prioritized for backend optimization, while other low-contribution feature points may be excluded to reduce computational load.

[0145] By selecting only the subset of features with the highest information content for backend optimization, computational load and memory consumption are reduced, thereby improving system response speed and real-time performance. This strategy is particularly effective on resource-constrained embedded platforms, ensuring efficient system operation with limited hardware resources. Although the number of feature points involved in optimization is reduced, the system's localization accuracy is effectively guaranteed by selecting feature points with high information gain. In low-texture or dynamic environments, the system can avoid localization errors caused by insufficient feature points through a reasonable feature selection strategy.

[0146] In this embodiment, step eight proposes the following trade-off strategy:

[0147] 1. Multi-level optimization strategy: The backend optimization is divided into two stages: coarse optimization and fine optimization. Coarse optimization uses fewer feature points to converge quickly and provide an initial estimate; fine optimization, within the limits of computational resources, uses more feature points for fine-tuning, so as to make reasonable use of resources.

[0148] 2. Dynamic Keyframe Management: Based on information theory, a keyframe selection strategy is designed to retain only keyframes that significantly contribute to system state estimation. A keyframe redundancy evaluation mechanism is introduced to promptly remove redundant historical keyframes and control the optimization window size.

[0149] 3. Variable Optimization Window: The sliding window size is dynamically adjusted based on system status and computational resources. In highly dynamic scenarios or when feature tracking is unstable, the window is appropriately enlarged to improve robustness; in stable scenarios, the window is reduced to decrease computational burden.

[0150] 4. Accuracy Guarantee Mechanism: An accuracy monitoring module is designed to assess the uncertainty of system state estimation in real time. When the uncertainty exceeds a preset threshold, the number of feature points and the number of optimization iterations are automatically increased to ensure that the system accuracy does not decrease significantly.

[0151] Thus far, steps one through eight have completed the process from front-end feature selection to back-end system optimization. For ease of understanding, steps one through eight can be summarized as follows:

[0152] Step 1 → Step 2: The reliable feature point set output from Step 1 is the direct input for the initial scale estimation in Step 2. Specifically, Step 2 utilizes the geometric relationships of these feature points across multiple frames, combined with IMU pre-integrated data, to solve for the scale factor through joint vision-inertial optimization. Without the reliable feature points from Step 1, Step 2 cannot establish accurate geometric constraints.

[0153] The main outputs of step two are the scaling factor and the initial state estimate, which provide a correct scaling benchmark for the entire VINS system. Although step three directly uses the seed feature points provided in step one, the scaling factor estimated in step two will affect the subsequent state estimation and optimization process, establishing a correct metric benchmark for the system.

[0154] Step 1 → Step 3: The feature points established in Step 1 become the seed features for tracking in Step 3. Step 3 does not re-detect features, but rather uses hardware-accelerated optical flow for tracking in subsequent frames based on the feature point positions determined in Step 1.

[0155] Step 3 → Step 4: The tracking results from Step 3 are used in Step 4 for bidirectional optical flow consistency verification. Feature points that fail to track are removed, and highly reliable feature trajectories are output.

[0156] Step 4 → Step 5: The feature trajectories verified in Step 4 serve as input for feature management in Step 5, enabling dynamic density control and quality evaluation. This, in turn, controls the number of feature points extracted in Step 1.

[0157] Step 5 → Step 6: The feature distribution after management in Step 5 serves as the candidate set for feature selection in Step 6. Step 6 selects the optimal subset from this set based on information gain evaluation for backend optimization.

[0158] Step 6 → Step 7 → Step 8: Form a strategy optimization chain. The adaptive mechanism in Step 7 adjusts the selection strategy in Step 6. Step 8 adjusts the overall system parameters based on the feature subset selected in Step 6 and the computing resource status.

[0159] Compared to the traditional VINS-MONO implementation, the feature extraction and optical flow tracking method proposed in this embodiment achieves a 3.2-fold performance improvement on the UAV platform, reducing the average feature tracking time from 35ms to 11ms while maintaining the same level of tracking accuracy. Particularly in dynamic scenes, the improved optical flow tracking algorithm exhibits higher robustness, with a 18.7% increase in feature point tracking success rate.

[0160] This embodiment implements the above-described method using a heterogeneous computing architecture-based VINS system, which mainly consists of three core modules: front-end vision processing, back-end state estimation, and loop closure detection. Compared to the traditional VINS-Mono system, this system reconstructs the front-end, transmitting computationally intensive tasks such as feature extraction and optical flow calculation to a dedicated processing unit via a heterogeneous zero-copy data channel, while maintaining complex nonlinear optimization tasks executed on the main processor.

[0161] This embodiment employs the Coolchip ARS45 as its heterogeneous zero-copy data channel. The Coolchip ARS45, as a dedicated vision processor, integrates an image signal processor, a neural network acceleration unit, and a vector processing unit, typically used for general image processing or deep learning inference tasks. However, this embodiment features a heterogeneous computing interface designed specifically for the characteristics of the VINS algorithm, which is fundamentally different from traditional general-purpose hardware acceleration. This embodiment deeply integrates the ARS45 with the eight core steps of VINS, designing a three-layer acceleration interface architecture: the application algorithm layer is responsible for executing steps one through eight of the VINS algorithm; the hardware abstraction layer provides a unified vision processing interface; and the Coolchip ARS45 hardware driver adaptation layer implements the zero-copy data transmission mechanism.

[0162] The core principle of algorithm acceleration lies in its intelligent task separation strategy. The system allocates computationally intensive tasks such as optical flow and descriptor fusion (step 1), feature tracking (step 3), and bidirectional optical flow verification (step 4) to the ARS45's image signal processor and vector processing unit, while retaining logically complex tasks such as initial scale estimation (step 2) and feature selection based on information gain (step 6) for CPU execution. This task separation is not a simple load balancing; rather, it is based on a deep analysis of the computational characteristics of the VINS algorithm, achieving optimal utilization of computing resources. Through hardware and software co-design, computationally intensive tasks (feature extraction, optical flow calculation) are allocated to dedicated acceleration units, while complex nonlinear optimization tasks are retained for execution on the main processor, resulting in overall system acceleration. The Coolchip ARS45's zero-copy mechanism and lightweight data structure minimize data transmission overhead.

[0163] In this embodiment, step one, multi-level feature fusion, involves three sub-processes: the KLT optical flow algorithm, BRIEF descriptor matching, and geometric consistency constraints. Its execution time on the ARS45 must precisely match the vision-inertial joint optimization time on the CPU in step two to avoid any processing unit becoming a performance bottleneck. Step six, information gain evaluation, involves Jacobian matrix calculation, whose computational complexity is quadratically related to the number of feature points, requiring an accurate time prediction model to guide hardware resource allocation. Simultaneously, there are strict data dependencies between steps. The reliable feature point set output from step one is the direct input for initial scale estimation in step two. Step three performs tracking based on these seed feature points, and step four performs consistency checks on the tracking results. This temporal coordination of data flow necessitates the design of a precise buffer management mechanism.

[0164] The real-time constraint is further complicated by the requirement to ensure performance. The VINS system requires an image processing frequency of 30Hz, and all eight steps must be completed within 33ms per frame. The adaptive filtering mechanism in step seven needs to adjust the algorithm parameters in real time based on scene complexity and UAV motion status, while the resource trade-off strategy in step eight requires dynamic monitoring of the load status of the ARS45 and CPU to optimize the allocation of computing resources while ensuring positioning accuracy.

[0165] In terms of data transmission, the system employs the ARS45's DMA controller to implement a zero-copy mechanism. Image data is directly transferred from the camera sensor to the ARS45's memory, avoiding multiple copies of image data between the CPU and accelerator as in traditional solutions. After processing, feature point coordinates and descriptors are passed to the CPU via a shared memory region for subsequent state estimation and optimization calculations. The ARS45 includes a multi-level cache structure, which effectively reduces memory access latency and improves feature extraction throughput. The Lucas-Kanade optical flow algorithm in step three is executed in parallel on a multi-scale image pyramid, utilizing the VPU's vector computation capabilities to achieve efficient optical flow tracking, resulting in a processing speed improvement of more than 5 times compared to a pure CPU implementation.

[0166] The hardware configuration challenge in this embodiment lies in the coordination and scheduling between the ARS45 and the CPU, and the dynamic management of heterogeneous resources. Since the two processing units have different processing capabilities, memory architectures, and latency characteristics, a complex load balancing mechanism needs to be designed. When the ARS45 processes feature fusion in step one and feature tracking in step three, the CPU can execute IMU pre-integration in step two and feature selection in step six in parallel, but the timing consistency of the data flow and the correct handling of dependencies must be ensured. The adaptive mechanism formed in steps seven and eight requires real-time monitoring of the resource utilization of the two processing units and dynamic adjustment of algorithm parameters based on the UAV's flight status and scene complexity. The complexity of memory management lies in the need for unified management of the ARS45's on-chip memory, CPU system memory, and shared memory regions, while simultaneously handling memory fragmentation, access conflicts, and cache coherence issues.

[0167] The innovation of this embodiment lies primarily in the deep integration and reconstruction of the algorithm and hardware. Traditional VINS-Mono systems execute all modules sequentially on the CPU, while this embodiment fundamentally reconstructs the entire computational process, re-decomposing the eight steps and mapping them to heterogeneous processing units. This is not merely a simple task allocation; it requires redesigning data structures, memory layout, execution order, and error handling mechanisms. Steps one through eight form a complete closed-loop optimization system. The scene complexity perception in step seven adjusts the feature selection strategy in step six based on the UAV's motion state, and the computational resource perception adjustment in step eight dynamically controls the feature density distribution in step five. The system also features a robust fault-tolerance mechanism. When the ARS45 experiences a hardware failure, the functions of steps one through four automatically switch to CPU execution. Although performance decreases, this ensures system continuity and reliability. This fault-tolerance design for heterogeneous computing further increases the system's complexity and innovative value.

[0168] The specific process is as follows:

[0169] Data acquisition phase: The camera on the drone acquires 640×480 resolution images at a frequency of 30Hz, and the IMU provides inertial measurement data at a frequency of 170Hz. The two types of data are precisely synchronized through hardware timestamps.

[0170] Front-end hardware acceleration processing: Image data is directly transmitted to the Coolchip ARS45 hardware accelerator. Step one performs multi-level feature fusion on the ARS45. First, a hardware-accelerated KLT optical flow algorithm is used for initial tracking, then BRIEF descriptor matching is used for verification, and finally geometric consistency constraints are introduced to eliminate outliers, outputting approximately 200 high-quality feature points per frame. Step three, based on these seed feature points, uses the ARS45's SIMD processing unit to perform Lucas-Kanade optical flow tracking on a multi-scale image pyramid. Step four implements forward and reverse optical flow calculations, and eliminates mismatches through consistency checks to ensure the robustness of feature tracking.

[0171] Hardware-CPU Data Transfer: The zero-copy mechanism in step six ensures efficient transfer of feature point data from the ARS45 to the CPU main processor. DMA (Direct Memory Access) is used to avoid repeated copying of data between user space and kernel space, resulting in a data transfer latency of less than 1ms.

[0172] CPU logic optimization: Step two performs vision-inertial joint optimization on the CPU, using the feature point trajectories provided by the front end and the IMU pre-integration results to estimate the initial scale and system state by solving a nonlinear least squares problem. Step five implements dynamic feature management, adaptively adjusting the feature density according to scene complexity to ensure that feature points are evenly distributed in the image. Step six selects the feature subset that contributes most to positioning accuracy based on information gain theory, controlling the number of feature points participating in backend optimization to 100-200. Steps seven and eight form an adaptive adjustment mechanism, adjusting the algorithm parameters in real time according to the UAV's motion state and computing resources.

[0173] Navigation data output: The backend Bundle Adjustment optimizer performs nonlinear optimization based on the selected feature subset, outputting 6DOF pose and velocity estimates. The data update frequency is 30Hz, which meets the real-time navigation and control requirements of UAVs.

[0174] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system, characterized in that, Includes the following steps: S1. Optical Flow and Descriptor Fusion: Feature points are tracked between consecutive frames using an optical flow algorithm to obtain dense motion information. At the same time, descriptors are extracted from key points and matched and verified with motion information to output a reliable set of feature points. S2. Initial Scale Estimation and Optimization: Utilizing the set relationships of reliable feature point sets across multiple frames, combined with IMU pre-integrated data, the scale factor is solved through visual-inertial joint optimization to construct motion geometric constraints; S3. Feature Tracking: Based on a reliable set of feature points, key frames are identified, seed features for tracking are formed, and tracking is performed in subsequent frames based on the seed features; S4. Bidirectional optical flow consistency check: Based on the tracking results of S3, a bidirectional optical flow consistency check is performed, and high-reliability feature trajectories are selected according to the preset tracking error threshold. S5. Feature Management: Based on feature trajectories, perform dynamic density control and quality assessment of feature points, and output the managed feature points; S6. Feature selection: Based on information gain evaluation, calculate the contribution of each feature point output in step S5 to the system state estimation, and select the subset with the largest information content for back-end optimization; S7. Adaptive Feature Selection: Perform adaptive feature selection on the subset selected in step S6, and dynamically adjust the feature selection strategy. S8. Strategy for balancing computational resources and accuracy: Based on the subset selected in step S6, and in combination with system resources, dynamically adjust system parameters.

2. The method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system according to claim 1, characterized in that, Step S1 involves performing multi-level feature fusion on the ARS45. First, the hardware-accelerated KLT optical flow algorithm is used for preliminary tracking. Then, BRIEF descriptor matching is used for verification. Finally, geometric consistency constraints are introduced to remove outliers, and 200 high-quality feature points are output per frame.

3. The method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system according to claim 1, characterized in that, Step S2 specifically involves obtaining the optimal scale and initial state by solving the following objective function: The visual reprojection residual can be expressed as: The IMU pre-integration residual can be expressed as: in As a scale factor, For system state variables, and These are the visual reprojection residual and the IMU pre-integration residual, respectively. For robust kernel functions, The covariance matrix of the IMU measurements. For image feature point observations, For camera projection function, For the camera rotation matrix, Let be the camera translation vector. 3D landmark coordinates; For the k-th IMU frame, rotate to the world coordinate system. Let k be the position of the k-th IMU frame in the world coordinate system. Let the velocity of the k-th IMU frame be in the world coordinate system. The pre-integral value for the position from frame k to frame (k+1) is... For time intervals, This represents the gravity vector in the world coordinate system. For the velocity pre-integral quantity from frame k to frame (k+1), Let k be the rotation quaternion from the world coordinate system to the k-th IMU frame. For the rotation pre-integral quantity from frame k to frame (k+1), This is the quaternion multiplication operator.

4. The method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system according to claim 1, characterized in that, Step S5 specifically involves: (1) Dynamic density control: The feature point density is adaptively adjusted according to the scene complexity and system state; the density is appropriately reduced in feature-rich areas and the sampling density is increased in feature-sparse areas to achieve uniform feature distribution; (2) Multi-scale feature selection: Extract features at different levels of the image pyramid to ensure that there are sufficient feature coverage for both near and far targets, and improve the system’s adaptability to scale changes; (3) Feature quality scoring mechanism: Taking into account factors such as corner response value, gradient distribution and tracking stability, a quality score is assigned to each feature point, and high-quality features are retained first; (4) Predictive feature management: Combined with camera motion prediction, feature points that may leave the field of view are identified in advance, and new features are extracted in the corresponding areas in a timely manner to avoid feature shortage.

5. The method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system according to any one of claims 1 to 4, characterized in that, Step S6 specifically involves: Feature point information gain evaluation is based on the following mathematical model: For feature points Its information gain Defined as: in It is a matrix eigenvalues, It is the dimension of the state vector. For feature points The corresponding Jacobian matrix, To observe the noise covariance matrix, `trace()` represents the trace of the matrix.

6. The method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system according to any one of claims 1 to 4, characterized in that, The step S7 dynamically adjusts the feature selection strategy: (1) Scene complexity awareness: By evaluating the number, distribution and quality of feature points in the current scene, the feature selection threshold is automatically adjusted. In feature-rich environments, stricter screening criteria are adopted, and selection conditions are relaxed in feature-sparse scenes. (2) Motion pattern adaptation: Adjust feature selection preferences according to the UAV's motion state; (3) Multi-objective optimization strategy: The feature selection problem is modeled as a multi-objective optimization problem, considering both information gain maximization and feature distribution uniformity; a weighted method is used to transform the multi-objective problem into a single-objective optimization problem: Among them, uniform distribution measure Defined as: in, For the set of feature points, For the i-th feature point, For the information gain of feature points, These are the weighting coefficients for information gain and distribution uniformity. The number of image partitions, The number of feature points in the k-th grid; (4) Computing resource awareness and adjustment: Real-time monitoring of system computing load, automatically reducing the number of optimized feature points when computing resources are tight, and appropriately increasing feature points when resources are sufficient, to achieve adaptive resource allocation.

7. The method for optimizing and accelerating a heterogeneous UAV visual-inertial navigation system according to any one of claims 1 to 4, characterized in that, The trade-off strategy for S8 is as follows: (1) Multi-level optimization strategy: The back-end optimization is divided into two stages: coarse optimization and fine optimization. Coarse optimization uses fewer feature points to converge quickly and provide an initial estimate. Fine optimization uses more feature points for fine adjustment when computing resources allow. (2) Dynamic management of key frames: Based on information theory, a key frame selection strategy is designed to retain only key frames that contribute significantly to the system state estimation; a key frame redundancy evaluation mechanism is introduced to remove redundant historical key frames in a timely manner and control the optimization window size. (3) Variable optimization window: The sliding window size is dynamically adjusted according to the system state and computing resources; in highly dynamic scenarios or when feature tracking is unstable, the window is appropriately enlarged to improve robustness; in stable scenarios, the window is reduced to reduce the computational burden; (4) Accuracy assurance mechanism: Design an accuracy monitoring module to evaluate the uncertainty of system state estimation in real time; when the uncertainty exceeds the preset threshold, automatically increase the number of feature points and optimize the number of iterations to ensure that the system accuracy will not decrease significantly.

8. A heterogeneous UAV visual inertial navigation system optimization and acceleration system, characterized in that, include: Optical flow and descriptor fusion module: Tracks feature points between consecutive frames using optical flow algorithm to obtain dense motion information, and extracts descriptors from key points and matches and verifies them with motion information to output a reliable set of feature points. Initial scale estimation and optimization module: Utilizing the set relationship of reliable feature point sets across multiple frames, combined with IMU pre-integrated data, the scale factor is solved through vision-inertial joint optimization to construct motion geometric constraints; Feature tracking module: Based on the reliable set of feature points output by the optical flow and descriptor fusion module, key frames are identified, seed features for tracking are formed, and tracking is performed in subsequent frames based on the seed features; Bidirectional optical flow consistency verification module: Based on the tracking results of the feature tracking module, it performs bidirectional optical flow consistency verification and selects highly reliable feature trajectories according to the preset tracking error threshold; Feature Management Module: Based on the feature trajectory output by the bidirectional optical flow consistency verification module, it performs dynamic density control and quality assessment of feature points, and outputs the managed feature points; Feature selection module: Based on information gain evaluation, calculate the contribution of each feature point output by the feature management module to the system state estimation, and select the subset with the largest information content for backend optimization; Adaptive Feature Selection Module: Performs adaptive feature selection on subsets and dynamically adjusts the feature selection strategy; The module for balancing computational resources and accuracy: It dynamically adjusts system parameters based on the subset selected by the feature selection module and in combination with system resources.

9. The optimization and acceleration system for a heterogeneous UAV visual-inertial navigation system according to claim 8, characterized in that, The optical flow and descriptor fusion module performs multi-level feature fusion on the ARS45. First, it uses the hardware-accelerated KLT optical flow algorithm for preliminary tracking, then verifies it through BRIEF descriptor matching, and finally introduces geometric consistency constraints to remove outliers, outputting 200 high-quality feature points per frame.

10. The optimization and acceleration system for a heterogeneous UAV visual-inertial navigation system according to claim 8, characterized in that, The initial scale estimation and optimization module specifically obtains the optimal scale and initial state by solving the following objective function: The visual reprojection residual can be expressed as: The IMU pre-integration residual can be expressed as: in As a scale factor, For system state variables, and These are the visual reprojection residual and the IMU pre-integration residual, respectively. For robust kernel functions, The covariance matrix of the IMU measurements. For image feature point observations, For camera projection function, For the camera rotation matrix, Let be the camera translation vector. 3D landmark coordinates; For the k-th IMU frame, rotate to the world coordinate system. Let k be the position of the k-th IMU frame in the world coordinate system. Let the velocity of the k-th IMU frame be in the world coordinate system. The pre-integral value for the position from frame k to frame (k+1) is... For time intervals, This represents the gravity vector in the world coordinate system. For the velocity pre-integral quantity from frame k to frame (k+1), Let k be the rotation quaternion from the world coordinate system to the k-th IMU frame. For the rotation pre-integral quantity from frame k to frame (k+1), This is the quaternion multiplication operator.

Images