Method and system for optimizing and accelerating visual inertial navigation system of unmanned aerial vehicle based on heterogeneity

Abstract

The invention provides an optimization and acceleration method and system for a visual inertial navigation system of an unmanned aerial vehicle based on heterogeneity. The method comprises the following steps: S1, outputting a reliable feature point set which is a direct input object for performing initial scale estimation in S2; and the feature points established in the step S1 become seed features tracked in the step S3. And S3, based on the position of the feature point determined in the S1, performing tracking by using a hardware acceleration optical flow in a subsequent frame. And the tracking result of the S3 enters the S4 for bidirectional optical flow consistency check, feature points which fail in tracking are eliminated, and a high-reliability feature trajectory is output. S4, taking the verified feature track as input of S5 feature management, and performing dynamic density control and quality evaluation. And S5, taking the managed feature distribution as a candidate set for feature selection in S6, and selecting an optimal subset for back-end optimization based on information gain evaluation in S6. S7, an adaptive mechanism adjusts the selection strategy in the step S6; and S8, overall system parameters are adjusted according to the state of the computing resource.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of unmanned aerial vehicle inertial navigation, in particular to an optimization and acceleration method and system for a heterogeneous unmanned aerial vehicle visual-inertial navigation system. BACKGROUND

[0002] With the diversification of unmanned aerial vehicle application scenarios, the autonomous navigation capability of unmanned aerial vehicles is facing increasingly severe challenges, especially on resource-constrained unmanned aerial vehicle platforms. At present, the visual-inertial navigation system (VINS) has the following technical problems and defects: Low computational efficiency: although the VINS system based on nonlinear optimization (such as VINS-Mono and ORB-SLAM) has high accuracy and robustness, the computational complexity is large, and it is difficult to run in real time on the resource-limited unmanned aerial vehicle platform. The feature extraction and optical flow calculation modules occupy a large amount of computing resources, resulting in slow system response speed.

[0003] Poor environmental adaptability: the existing VINS system performs poorly in low-texture environments and fast motion scenarios, initialization is prone to failure, feature point tracking is unstable, and the application of the system in complex environments is limited.

[0004] Unreasonable resource allocation: the traditional VINS system does not consider the difference in the contribution of feature points to positioning accuracy, and simply uses all the tracked feature points for back-end optimization, resulting in waste of computing resources and low processing efficiency. SUMMARY

[0005] The technical problem to be solved by the application is how to improve the performance of the unmanned aerial vehicle navigation system.

[0006] The application solves the above technical problems by the following technical means: The application provides an optimization and acceleration method for a heterogeneous unmanned aerial vehicle visual-inertial navigation system, comprising the following steps: S1. Fusion of optical flow and descriptor: track feature points between consecutive frames through an optical flow algorithm to obtain dense motion information, extract descriptors for key points and match and verify the motion information, and then output a reliable feature point set; S2. Initial scale estimation and optimization: use the set relationship of the reliable feature point set among multiple frames, combine the IMU pre-integration data, solve the scale factor through visual-inertial joint optimization, and construct motion geometric constraints; S3. Feature tracking: based on the reliable feature point set output in step S1, identify key frames to form seed features for tracking, and track in subsequent frames based on the seed features; S4. Bidirectional optical flow consistency check: based on the tracking results of step S3, perform bidirectional optical flow consistency check, and according to a preset tracking error threshold, screen high-reliability feature tracks; S5. Feature management: based on the feature tracks output in step S4, perform dynamic density control and quality assessment of feature points, and output the managed feature points; S6. Feature selection: based on information gain evaluation, calculate the contribution of each feature point output in step S5 to system state estimation, and select a subset with the largest amount of information for backend optimization; S7. Adaptive feature screening: adaptive feature screening is performed on the subset selected in step S6, and the feature selection strategy is dynamically adjusted; S8. Trade-off strategy of computing resources and precision: dynamically adjust system parameters according to system resources.

[0007] Further, the step S1 is to perform multi-level feature fusion on the ARS45, first use the hardware accelerated KLT optical flow algorithm for preliminary tracking, then verify through the BRIEF descriptor matching, finally introduce geometric consistency constraint to remove outliers, and output 200 high-quality feature points per frame.

[0008] Further, the step S2 specifically comprises: obtaining the optimal scale and initial state by solving the following optimization objective function:

[0009] The visual re-projection residual can be represented as:

[0010] The IMU pre-integration residual can be represented as:

[0011] Wherein is a scale factor, is a system state variable, and are visual re-projection residual and IMU pre-integration residual respectively, is a robust kernel function, is the covariance matrix of IMU measurement, is the image feature point observation value, is the camera projection function, is the camera rotation matrix, is the camera translation vector, is the 3D road point coordinate. is the rotation of the kth IMU frame to the world coordinate system, is the position of the kth IMU frame in the world coordinate system, is the velocity of the kth IMU frame in the world coordinate system, The pre-integral value for the position from frame k to frame (k+1) is... For time intervals, This represents the gravity vector in the world coordinate system. For the velocity pre-integral quantity from frame k to frame (k+1), Let k be the rotation quaternion from the world coordinate system to the k-th IMU frame. For the rotation pre-integral quantity from frame k to frame (k+1), This is the quaternion multiplication operator.

[0012] Furthermore, step S5 specifically includes: (1) Dynamic density control: The feature point density is adaptively adjusted according to the scene complexity and system state; the density is appropriately reduced in feature-rich areas and the sampling density is increased in feature-sparse areas to achieve uniform feature distribution; (2) Multi-scale feature selection: Extract features at different levels of the image pyramid to ensure that there are sufficient feature coverage for both near and far targets, and improve the system’s adaptability to scale changes; (3) Feature quality scoring mechanism: Taking into account factors such as corner response value, gradient distribution and tracking stability, a quality score is assigned to each feature point, and high-quality features are retained first; (4) Predictive feature management: Combined with camera motion prediction, feature points that may leave the field of view are identified in advance, and new features are extracted in the corresponding areas in a timely manner to avoid feature shortage.

[0013] Furthermore, the characteristic is that step S6 specifically comprises: Feature point information gain evaluation is based on the following mathematical model: For feature points Its information gain Defined as:

[0014] in It is a matrix eigenvalues, It is the dimension of the state vector. For feature points The corresponding Jacobian matrix, To observe the noise covariance matrix, `trace()` represents the trace of the matrix.

[0015] Furthermore, step S7 dynamically adjusts the feature selection strategy: (1) Scene complexity awareness: By evaluating the number, distribution and quality of feature points in the current scene, the feature selection threshold is automatically adjusted. In feature-rich environments, stricter screening criteria are adopted, and selection conditions are relaxed in feature-sparse scenes. (2) Motion pattern adaptation: adjusting feature selection preference according to the motion state of the UAV; (3) Multi-objective optimization strategy: modeling the feature selection problem as a multi-objective optimization problem, considering both maximum information gain and feature distribution uniformity; using a weighted method to convert the multi-objective optimization problem into a single-objective optimization problem:

[0016] where the uniform distribution metric is defined as:

[0017] where is the set of feature points, is the i-th feature point, is the information gain of the feature point, is the weight coefficient of information gain and distribution uniformity (0≤ ≤1), is the number of image partitions, is the number of feature points in the k-th grid.

[0018] (4) Computing resource-aware adjustment: real-time monitoring of system computing load, automatically reducing the number of optimized feature points when computing resources are tight, and appropriately increasing the number of feature points when resources are sufficient, to achieve adaptive resource allocation.

[0019] Further, the trade-off strategy of S8 is: (1) Multi-stage optimization strategy: dividing the back-end optimization into two stages of coarse optimization and fine optimization; coarse optimization uses fewer feature points to quickly converge and provides preliminary estimates; fine optimization uses more feature points for fine adjustment when computing resources allow; (2) Keyframe dynamic management: designing a keyframe selection strategy based on information theory, retaining only keyframes that significantly contribute to system state estimation; introducing a keyframe redundancy evaluation mechanism to remove information-redundant historical keyframes in time, controlling the size of the optimization window; (3) Variable optimization window: dynamically adjusting the size of the sliding window according to the system state and computing resources; in high dynamic scenes or unstable feature tracking, the window is appropriately expanded to improve robustness; in stable scenes, the window is reduced to reduce the computational burden; (4) Precision guarantee mechanism: designing a precision monitoring module to evaluate the uncertainty of system state estimation in real time; when the uncertainty exceeds the preset threshold, automatically increasing the number of feature points and the number of optimization iterations to ensure that the system precision does not decrease significantly.

[0020] The application also provides a heterogeneous-based UAV visual-inertial navigation system optimization and acceleration system, comprising: Optical flow and descriptor fusion module: track feature points between consecutive frames by optical flow algorithm to obtain dense motion information, and extract descriptors of key points and match and verify the motion information to output a reliable feature point set; Initial scale estimation and optimization module: use the set relationship of the reliable feature point set between multiple frames, combine IMU pre-integration data, solve the scale factor through visual-inertial joint optimization, and construct motion geometric constraints; Feature tracking module: based on the reliable feature point set output by the optical flow and descriptor fusion module, identify key frames to form seed features for tracking, and track in subsequent frames based on the seed features; Bidirectional optical flow consistency verification module: based on the tracking results of the feature tracking module, perform bidirectional optical flow consistency verification, and filter high-reliability feature trajectories according to a preset tracking error threshold; Feature management module: based on the feature trajectories output by the bidirectional optical flow consistency verification module, perform dynamic density control and quality assessment of feature points, and output managed feature points; Feature selection module: based on information gain evaluation, calculate the contribution of each feature point output by the feature management module to system state estimation, and select a subset with the largest amount of information for back-end optimization; Adaptive feature screening module: adaptive feature screening is performed on the subset to dynamically adjust the feature selection strategy; Trade-off strategy module of computing resources and precision: dynamically adjust system parameters according to system resources.

[0021] Further, the optical flow and descriptor fusion module performs multi-level feature fusion on ARS45, first uses a hardware-accelerated KLT optical flow algorithm for preliminary tracking, then performs matching verification through a BRIEF descriptor, finally introduces geometric consistency constraints to remove outliers, and outputs 200 high-quality feature points per frame.

[0022] Further, the initial scale estimation and optimization module specifically obtains the optimal scale and initial state by solving the following optimization objective function:

[0023] The visual re-projection residual can be represented as:

[0024] The IMU pre-integration residual can be represented as:

[0025] wherein is a scale factor, is a system state variable, and respectively are visual re-projection residual and IMU pre-integration residual, is a robust kernel function, is a covariance matrix of IMU measurement, is an image feature point observation, is a camera projection function, is a camera rotation matrix, is a camera translation vector, is a 3D landmark point coordinate. is the k-th IMU frame to world coordinate system rotation, is the k-th IMU frame position in world coordinate system, is the k-th IMU frame velocity in world coordinate system, is the k-th frame to k+1 frame position pre-integration quantity, is a time interval, is a gravity vector in world coordinate system, is the k-th frame to k+1 frame velocity pre-integration quantity, is the k-th IMU frame to world coordinate system rotation quaternion, is the k-th frame to k+1 frame rotation pre-integration quantity, is a quaternion multiplication operator.

[0026] The present application has the advantages of: 1. An initialization method of fusing optical flow and descriptors: the prior art usually adopts a serial process of "detection-description-matching", and the optical flow tracking and descriptor matching are independently operated and mainly executed on the CPU. We use a hardware-accelerated KLT optical flow algorithm for preliminary tracking in the first layer; perform matching verification through BRIEF descriptors in the second layer; and introduce geometric consistency constraints to eliminate outliers through fundamental matrix estimation in the third layer. This multi-level fusion strategy realizes the complementary advantages of the efficiency of optical flow and the robustness of descriptors, and significantly improves the initialization success rate in low-texture environments and motion blur scenes.

[0027] 2. A feature point selection algorithm based on information gain: the contribution of feature points to system state estimation is evaluated and screened, and the subset with the largest amount of information is selected to participate in the back-end optimization, so as to realize the optimal balance between computing resources and positioning accuracy.

[0028] 3. An adaptive feature management strategy: combined with scene complexity perception, motion pattern adaptation and computing resource monitoring, the feature point density, distribution and selection standard are dynamically adjusted to improve the robustness and efficiency of the system in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is an algorithm architecture corresponding to the method in the embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] like Figure 1 As shown in the figure, this embodiment provides a method for optimizing and accelerating a heterogeneous UAV visual inertial navigation system, which includes the following steps: Step 1: Optical Flow and Descriptor Fusion This embodiment proposes an initialization method combining optical flow tracking and descriptor matching. The core idea of ​​this method is to leverage the high efficiency of optical flow algorithms and the high robustness of descriptor matching to achieve a complementary advantage. Specifically, the system first tracks feature points across consecutive frames using a hardware-accelerated optical flow algorithm to acquire dense motion information; simultaneously, it extracts descriptors from key points and performs matching to verify the optical flow tracking results and eliminate erroneous matches. In this embodiment, dense motion information refers to the detailed motion vector field obtained at the image pixel level through the optical flow algorithm. Specifically, the optical flow algorithm, based on the assumption of gray-level invariance, calculates the motion velocity and direction of each pixel in the image across consecutive frames. Compared to traditional feature point matching, which can only obtain motion information from a few hundred sparse feature points, the optical flow algorithm can provide motion estimation for tens of thousands of pixels in the image, forming a dense motion vector field. This dense information helps to more accurately understand the overall motion pattern of the scene, especially providing more motion constraints in textured areas. Optical flow tracking is based on the assumption that the gray level of the same point on the image remains unchanged over a short period of time, i.e.:

[0032] Performing a Taylor expansion and rearranging the equation, we obtain the Lucas-Kanade optical flow equation:

[0033] in x, y For image plane coordinates, t Let Δx, Δy, and Δt be time, and Δt be the corresponding changes. For pixel motion speed (optical flow), , and These are the partial derivatives of the image in space and time, respectively.

[0034] The fusion strategy adopts a hierarchical method based on consistency check: the first layer uses a hardware-accelerated KLT optical flow algorithm for preliminary tracking; the second layer verifies the matching through BRIEF descriptors; the third layer introduces geometric consistency constraints and eliminates outliers through fundamental matrix estimation. This multi-level fusion strategy significantly improves the robustness and accuracy of initialization, especially in low-texture and motion blur scenes.

[0035] Step two, initial scale estimation and optimization The monocular vision system has a scale uncertainty problem, which needs to use IMU information for scale recovery. The embodiment improves the scale estimation method of the traditional VINS-Mono, and proposes an initial scale estimation algorithm based on visual-inertial joint optimization. The algorithm uses the IMU pre-integration result to construct an optimization problem containing a scale factor, and combines visual geometric constraints to obtain the optimal scale and initial state by solving the following optimization objective function:

[0036] The visual re-projection residual can be expressed as:

[0037] The IMU pre-integration residual can be expressed as:

[0038] wherein is the scale factor, is the system state variable, and are the visual re-projection residual and the IMU pre-integration residual, respectively, is the robust kernel function, is the covariance matrix of the IMU measurement, is the image feature point observation value, is the camera projection function, is the camera rotation matrix, is the camera translation vector, is the 3D road point coordinate. is the rotation of the kth IMU frame to the world coordinate system, is the position of the kth IMU frame in the world coordinate system, is the velocity of the kth IMU frame in the world coordinate system, is the pre-integration quantity of the kth frame to the k+1th frame position, is the time interval, is the gravity vector in the world coordinate system, is the pre-integration quantity of the kth frame to the k+1th frame velocity, is the rotation quaternion of the kth IMU frame to the world coordinate system, For the k-th frame to the k+1-th frame rotation pre-integration quantity, For quaternion multiplication operator.

[0039] To improve the efficiency of optimization, the system adopts a step-by-step refinement strategy, first performs rough scale estimation, and then fine adjustment through nonlinear optimization.

[0040] The main output of step two is the scale factor and the initial state estimate, which provides the correct scale reference for the entire VINS system. The scale factor estimated in step two will affect the subsequent state estimation and optimization process, establishing the correct metric reference for the system.

[0041] Step three, feature tracking This embodiment proposes a feature tracking scheme based on hardware-accelerated optical flow, replacing the traditional "detection-description-matching" process. The core idea of this scheme is: comprehensive feature detection on key frames, and direct tracking of existing feature points on non-key frames through optical flow method, greatly reducing the frequency of feature extraction.

[0042] The identification strategy of key frames can use the average disparity, which refers to the average disparity of feature points calculated between consecutive frames. If the average disparity of the tracked features exceeds the preset threshold, the current frame will be marked as a key frame. This method ensures that the system can capture significant motion changes, thereby improving the accuracy of tracking. Comprehensive feature detection is performed on key frames, while existing feature points are directly tracked through optical flow method on non-key frames.

[0043] Adaptive feature detection: improved FAST corner detection algorithm is used on key frames, combined with image gradient information for feature point quality evaluation, ensuring uniform distribution of feature points and high discrimination.

[0044] Picture gradient information acquisition is specifically: get gradient input image through picture → hardware acceleration unit → parallel gradient calculation → gradient information output feature point acquisition method: 1. Hardware parallel calculation of image gradient (I x , I y ); 2. Evaluate the quality of the corner based on the gradient amplitude and direction; 3. FAST detection + gradient quality score → final feature point.

[0045] The hardware accelerator in this embodiment utilizes the Lucas-Kanade optical flow algorithm with hardware acceleration (the image signal processor (ISP) built in ARS45 is responsible for image preprocessing, the vector processing unit (VPU) is specially used for feature extraction and optical flow calculation, multi-point parallel processing is realized by using the SIMD instruction set, and zero-copy data transmission is realized by the DMA controller) to track feature points on a multi-scale image pyramid, effectively dealing with large parallax and motion blur. The core formula of optical flow tracking is:

[0046] wherein is an optical flow vector, which is obtained by solving the following minimization problem:

[0047] The equation is solved iteratively:

[0048] wherein, is the image gray value of the t-th frame at coordinates (x, y), is the image coordinates, represents a pixel window around the feature point, respectively represent the partial derivatives of the image gray value with respect to the x direction, the y direction and the time t, is the increment of the optical flow vector, and the iterative solving process is: n is the number of iterations.

[0049] It should be noted that the optical flow in step one is mainly used for obtaining and verifying the initial feature points, and is part of the "fusion of optical flow and descriptor" process, and the purpose is to establish a reliable set of feature points. The optical flow in step three is specially used for feature point tracking, and is based on the feature points established in step one for continuous tracking. The Lucas-Kanade optical flow algorithm is used to track feature points on a multi-scale image pyramid. Step one is used for feature point initialization, step three is used for feature point tracking, and step one processes the original image frame, and step three processes the tracking based on the known feature points.

[0050] Step four, bidirectional optical flow consistency test and feature point cycle This step implements forward and reverse (compared to the traditional single direction to reduce error) optical flow calculation, screens reliable points through a tracking error threshold, effectively eliminates occluded and mismatched points. A feature point aging mechanism is introduced to track the life cycle of each feature point, and feature points that have been stably tracked for a long time are preferentially retained to improve system stability (the system screens reliable points by setting a tracking error threshold, considers the difference between forward and reverse optical flow calculation, can more accurately judge the reliability of feature points, effectively removes points with large errors, and thus improves the accuracy of the entire navigation system).

[0051] It should be noted that step three uses the optical flow algorithm to track the feature points, and outputs the tracked feature point positions. Step four performs a "two-way optical flow consistency check" on the tracking results of step three. Two-way optical flow calculation specifically refers to comparing the consistency of the results of calculating the forward optical flow (from the previous frame to the current frame) and the reverse optical flow (from the current frame back to the previous frame), setting a tracking error threshold, and removing feature points with poor consistency, and outputting high-reliability feature trajectories.

[0052] Step five, balancing the number and quality of feature points The number and quality of feature points directly affect the accuracy and computational efficiency of the VINS system. This embodiment designs a dynamic feature management strategy to optimize the use of computing resources while ensuring positioning accuracy: 1. Dynamic density control: adjust the feature point density adaptively according to the scene complexity and system state. Reduce the density in areas with rich features, and increase the sampling density in areas with sparse features to achieve uniform distribution of features. Specifically, if (regional feature point density > high density threshold), then increase the FAST corner detection threshold, and if (regional feature point density <= low density threshold), then decrease the FAST corner detection threshold. 2. Multi-scale feature selection: extract features at different levels of the image pyramid to ensure that both near and far targets are covered with enough features, improving the system's adaptability to scale changes. (Through hierarchical detection + cross-layer matching, robust feature coverage of near and far targets is achieved).

[0053] 3. Feature quality scoring mechanism: consider factors such as corner response value, gradient distribution, and tracking stability to assign a quality score to each feature point, and prioritize high-quality features.

[0054] 4. Predictive feature management: combine camera motion prediction to identify features that may leave the field of view in advance, and extract new features in the corresponding area in time to avoid feature shortages.

[0055] Removing mis-matched points can reduce computational load while ensuring adequate coverage, thereby ensuring the accuracy of the calculation results while reducing computational load.

[0056] The image pyramid of step five extracts features at different levels to ensure feature coverage of near and far targets, and achieves uniform feature distribution through "hierarchical detection + cross-layer matching". The image pyramid of step three performs Lucas-Kanade algorithm at multiple scales to achieve multi-scale optical flow calculation. The purpose and function of the pyramid are different, but the pyramid itself and step three can be shared.

[0057] The feature management stage of step five is continuously executed. The specific execution timing is: 1. In each frame processing: based on the current camera motion state (provided by IMU and visual odometry); 2. Predict the camera pose of the next frame: using uniform motion model or IMU pre-integration results; 3. Project existing feature points: project the current feature points to the predicted next frame position; 4. Identify the feature points about to leave the field of view: judge whether the projected position exceeds the image boundary; 5. Extract new features in the corresponding area: force to detect new feature points in the area where the feature will be lost.

[0058] Step six, feature selection criteria and algorithm The computational complexity of the VINS system backend optimization is in square or cubic relationship with the number of feature points. On the resource-limited UAV platform, it is crucial to effectively manage the number of feature points participating in optimization. The traditional VINS-Mono system simply uses all the tracking feature points for backend optimization, without considering the contribution difference of feature points to positioning accuracy and the limitation of computing resources. This embodiment proposes a feature point selection algorithm based on information gain, which is to evaluate the contribution of each feature point to system state estimation and select the subset with the largest information to participate in the backend optimization.

[0059] The feature point information gain evaluation is based on the following mathematical model: for a feature point , its information gain is defined as:

[0060] where is the eigenvalue of the matrix , and is the dimension of the state vector. is the Jacobian matrix corresponding to the feature point , and is the observation noise covariance matrix, and trace() represents the trace of the matrix. This model considers factors such as the number of observations, observation angle diversity, and re-projection error of the feature point, and quantifies the contribution of the feature point to system accuracy.

[0061] In algorithm implementation, a greedy strategy is adopted to gradually select the feature points with the largest information gain until the preset number threshold or the information gain increment is lower than the specified threshold (limiting the total number of feature points participating in optimization). To improve efficiency, the feature selection process is designed to be incremental, only evaluating new feature points when each new frame arrives, avoiding complete recalculation.

[0062] Step seven, adaptive feature screening mechanism To address the dynamic changes in scene and motion state during UAV flight, this embodiment designs an adaptive feature selection mechanism, dynamically adjusting the feature selection strategy: 1. Scene complexity awareness: By evaluating the number, distribution, and quality of feature points in the current scene, automatically adjust the feature selection threshold. In feature-rich environments, use stricter selection criteria, and in feature-scarce scenes, relax the selection conditions.

[0063] 2. Motion pattern adaptation: Adjust feature selection preferences based on UAV motion state (still, uniform speed, acceleration, rotation, etc.). For example, prioritize feature points in the center region of the image during rapid rotation; during rapid forward motion, retain more distant feature points to maintain scale consistency.

[0064] 3. Multi-objective optimization strategy: Model the feature selection problem as a multi-objective optimization problem, considering both information gain maximization and feature distribution uniformity. Use a weighted method to convert the multi-objective problem into a single-objective optimization problem:

[0065] Where the uniform distribution metric is defined as:

[0066] Where is the set of feature points, is the i-th feature point, is the information gain of the feature point, is the weight coefficient of information gain and distribution uniformity (0 ≤ ≤ 1), is the number of image partitions, is the number of feature points in the k-th grid.

[0067] 4. Computing resource-aware adjustment: Monitor system computing load in real time, automatically reduce the number of optimized feature points when computing resources are tight, and appropriately increase the number of feature points when resources are sufficient, achieving adaptive resource allocation.

[0068] In terms of real-time monitoring system computing load, a multi-level monitoring strategy can be adopted. First, processing time monitoring is performed by monitoring the processing time consumption of each frame of image and setting a threshold (e.g., 33 ms for 30 fps), monitoring the execution time of front-end visual processing, back-end optimization, etc. modules, and monitoring the backlog of the image processing queue. Second, hardware resource monitoring is performed by obtaining CPU occupancy rate through system API, monitoring available memory and memory fragmentation, monitoring the usage rate and temperature of the acceleration unit, and monitoring power consumption changes on the UAV platform. In addition, system state indicators are monitored, such as cache hit rate of data access, scheduling efficiency of multi-threaded system, and data transmission load between CPU and acceleration unit.

[0069] In terms of reducing optimization feature standard setting, information gain calculation and quality score mechanism mentioned in the previous steps need to be combined. Based on the information gain grading standard, feature points can be divided into three priorities: high priority features with information gain greater than threshold A are always retained, medium priority features with information gain between threshold B and A are retained when resources are sufficient, and low priority features with information gain less than threshold B are first removed when resources are scarce. At the same time, feature quality score standards need to be considered, including tracking stability, observation frequency, and geometric distribution. Feature points that track stably over a long period, are observed by multiple key frames, and can ensure uniform distribution in the image should be given higher weight.

[0070] Dynamic adjustment strategy can implement different retention strategies according to the load level: 95% of candidate feature points are retained when the load is light, 80% of high-quality feature points are retained when the load is medium, and only 60% of core feature points are retained when the load is severe, and system real-time performance is prioritized. The adaptive threshold mechanism in step seven can be designed to reduce the number of feature points by 20% when the processing time exceeds 1.2 times the real-time threshold, and increase the number of feature points by 10% when the processing time is less than 0.8 times the real-time threshold and resources are sufficient.

[0071] It should be noted that the automatic adjustment of feature selection threshold in step seven is different from that in step five: the dynamic density control in step five is to adaptively adjust the feature point density according to the scene complexity and system state. In feature-rich areas, the sampling density is appropriately reduced, and in feature-sparse areas, the sampling density is increased to achieve uniform feature distribution. The dynamic density control in step five occurs in the front-end feature detection stage, i.e., dynamic density control and quality assessment are performed on the feature trajectories output by step four. The feature distribution managed by step five is used as the candidate set for step six feature selection, and step six selects the optimal subset based on information gain evaluation for back-end system parameter optimization. The optimal subset selected by steps five and six is used for back-end system parameter optimization, which can reduce the computational load while improving the optimization effect.

[0072] In contrast, the scenario complexity perception in step seven occurs in the feature selection stage of the backend optimization, which aims to select which existing feature points participate in the optimization calculation. It plays a role in the screening link before backend optimization, balancing computing resources and accuracy by adjusting the threshold criteria for feature selection, and the object of action is the existing feature point set, and finally a feature subset for optimization is selected.

[0073] From the timing relationship, the whole process is as follows: first, system initialization (steps one and two), then consistency check (step four), then density control in the feature tracking process combined with step five, and finally information gain selection (step six) and adaptive screening mechanism (step seven) before backend optimization. Therefore, step seven is an adaptive enhancement based on step six and occurs in the backend optimization stage.

[0074] Overall, step five controls "how many feature points are produced" in the front end, while step seven controls "which feature points are used" in the backend. Both consider scenario complexity, but serve different links and goals of the system.

[0075] Step eight, trade-off strategy between computing resources and accuracy On the UAV platform, the balance between computing resources and positioning accuracy is the core challenge of system design. Step eight is based on the subset selected in step six and dynamically adjusts system parameters in combination with system resources. Using the screened feature subset for backend optimization can reduce the amount of calculation. Specifically, the feature selection in step six relies on information gain evaluation, which quantifies the contribution of feature points to system state estimation, and selects the feature subset with the most information. The information gain evaluation formula of the feature point involves the calculation of Jacobian matrix, covariance matrix, etc. The computational complexity and memory consumption will increase exponentially with the increase of the number of feature points. Therefore, feature selection not only considers information gain, but also considers the consumption of computing resources, memory occupation and real-time demand.

[0076] The backend optimization is carried out through the nonlinear least squares method (Gauss-Newton method), and the goal is to update the state variables by minimizing the optimization objective function (including visual re-projection residual and IMU pre-integration residual). The selected feature subset is used as the input of the optimization objective, and through the visual-inertial joint optimization (visual re-projection and IMU pre-integration residual), the state variables of the system (such as camera pose, IMU bias, etc.) are updated. During the optimization process, the information gain of the feature points directly affects their weight in the optimization. Feature points with high information gain will occupy a larger proportion in the optimization, improving the final estimation accuracy. In the case of resource shortage, the size of the optimization window or the number of optimization iterations is dynamically adjusted to avoid excessive consumption of computing resources. Feature points with high accuracy will be given priority to participate in the backend optimization, while other low-contribution feature points may be excluded to reduce the amount of calculation.

[0077] By selecting only the most informative feature subset for backend optimization, the computational load is reduced, the memory consumption is lowered, and the system's response speed and real-time performance are improved. Especially on resource-constrained embedded platforms, this strategy ensures efficient operation of the system under limited hardware resources. Despite the reduction in the number of feature points involved in optimization, the system's positioning accuracy can be effectively guaranteed by selecting feature points with high information gain. In low-texture or dynamic environments, the system can avoid positioning errors caused by insufficient feature points through reasonable feature selection strategies.

[0078] The following trade-off strategies are proposed in step eight of the embodiment: 1. Multi-stage optimization strategy: The backend optimization is divided into two stages, coarse optimization and fine optimization. Coarse optimization uses fewer feature points to quickly converge and provides a preliminary estimate. Fine optimization uses more feature points to make fine adjustments under the condition that computing resources are allowed, making rational use of resources.

[0079] 2. Keyframe dynamic management: Based on information theory, a keyframe selection strategy is designed to retain only keyframes that significantly contribute to system state estimation. A keyframe redundancy evaluation mechanism is introduced to remove information-redundant historical keyframes in a timely manner, controlling the size of the optimization window.

[0080] 3. Variable optimization window: The size of the sliding window is dynamically adjusted according to the system state and computing resources. In high dynamic scenes or unstable feature tracking, the window size is appropriately expanded to improve robustness; in stable scenes, the window size is reduced to reduce computational burden.

[0081] 4. Precision guarantee mechanism: A precision monitoring module is designed to evaluate the uncertainty of system state estimation in real time. When the uncertainty exceeds the preset threshold, the number of feature points and the number of optimization iterations are automatically increased to ensure that the system accuracy does not decrease significantly.

[0082] So far, steps one to eight above complete the feature selection from the front end to the system optimization in the back end. For ease of understanding, steps one to eight above can be summarized as: Step one → Step two: The reliable feature point set output by step one is the direct input object for initial scale estimation in step two. Specifically, step two uses the geometric relationship of these feature points between multiple frames, combined with IMU pre-integration data, to solve the scale factor through visual-inertial joint optimization. Without the reliable feature points of step one, step two cannot establish accurate geometric constraints.

[0083] The main output of Step Two is the scale factor and the initial state estimation, which provides the correct scale reference for the whole VINS system. Although Step Three directly uses the seed feature points provided by Step One, the scale factor estimated by Step Two will affect the subsequent state estimation and optimization process, establishing the correct metric reference for the system.

[0084] Step One → Step Three: The feature points established by Step One become the seed features for Step Three tracking. Instead of re-detecting features, Step Three uses hardware-accelerated optical flow to track in subsequent frames based on the feature point positions determined by Step One.

[0085] Step Three → Step Four: The tracking results from Step Three enter Step Four for bidirectional optical flow consistency verification, eliminating failed feature points and outputting high-reliability feature tracks.

[0086] Step Four → Step Five: The feature tracks verified by Step Four serve as input for Step Five feature management, which controls the dynamic density and quality assessment. This further controls the number of feature points extracted by Step One.

[0087] Step Five → Step Six: The managed feature distribution from Step Five serves as the candidate set for Step Six feature selection, which selects the optimal subset based on information gain evaluation for backend optimization.

[0088] Step Six → Step Seven → Step Eight: Forming a strategy optimization chain, Step Seven's adaptive mechanism adjusts the selection strategy of Step Six, and Step Eight adjusts the overall system parameters based on the feature subset selected by Step Six and the state of the computing resources.

[0089] Compared to the traditional VINS-MONO implementation, the feature extraction and optical flow tracking method proposed in this embodiment achieves a 3.2 times performance improvement on a UAV platform, reducing the average feature tracking time from 35ms to 11ms while maintaining the same level of tracking accuracy. In particular, in dynamic scenes, the improved optical flow tracking algorithm shows higher robustness, with a 18.7% increase in feature point tracking success rate.

[0090] The VINS system based on the heterogeneous computing architecture for implementing the above method mainly consists of three core modules: front-end visual processing, back-end state estimation, and loop detection. Compared with the traditional VINS-Mono system, this system restructures the front-end, transferring computationally intensive tasks such as feature extraction and optical flow calculation to dedicated processing units through heterogeneous zero-copy data channels, while maintaining complex nonlinear optimization tasks on the main processor.

[0091] The heterogeneous zero-copy data channel of the embodiment adopts a Cool Chip ARS45, which is used as a special visual processor and internally has an image signal processor, a neural network acceleration unit, and a vector processing unit, and is usually applied to general image processing or deep learning inference tasks. However, the heterogeneous computing interface designed for the VINS algorithm features of the embodiment is essentially different from traditional general hardware acceleration. The embodiment deeply integrates the ARS45 with eight core steps of VINS and designs a three-layer architecture of the acceleration interface: the application algorithm layer is responsible for the execution of the VINS algorithm in steps 1 to 8, the hardware abstraction layer provides a unified visual processing interface, and the Cool Chip ARS45 hardware driver adaptation layer implements a zero-copy data transmission mechanism.

[0092] The algorithm acceleration core principle is reflected in the intelligent task separation strategy. The system allocates the computing-intensive tasks such as the fusion of optical flow and descriptors in step 1, feature tracking in step 3, and bidirectional optical flow verification in step 4 to the image signal processor and vector processing unit of the ARS45, while retaining the logically complex tasks such as the initial scale estimation in step 2 and the feature selection based on information gain in step 6 on the CPU for execution. This task separation is not a simple load distribution, but based on a deep analysis of the computing characteristics of the VINS algorithm, it realizes the optimal utilization of computing resources, and through the software and hardware collaborative design, the computing-intensive tasks (feature extraction, optical flow calculation) are allocated to the special acceleration unit, and the complex nonlinear optimization tasks are retained on the host processor for execution, realizing the overall acceleration of the system. The zero-copy mechanism and lightweight data structure of the Cool Chip ARS45 minimize data transmission overhead.

[0093] The multi-level feature fusion in step 1 of the embodiment involves three sub-processes of KLT optical flow algorithm, BRIEF descriptor matching, and geometric consistency constraint, and its execution time on the ARS45 must be accurately matched with the visual-inertial joint optimization time on the CPU in step 2 to avoid any processing unit from becoming a performance bottleneck. The information gain evaluation in step 6 involves Jacobian matrix calculation, whose computational complexity is proportional to the square of the number of feature points, and an accurate time prediction model needs to be established to guide the allocation of hardware resources. At the same time, there is a strict data dependency relationship between steps, and the reliable feature point set output by step 1 is the direct input object for the initial scale estimation in step 2, step 3 performs tracking based on these seed feature points, and step 4 performs consistency verification on the tracking results. The timing coordination of this data flow requires the design of an accurate buffer management mechanism.

[0094] More complex is the guarantee of real-time constraints. The VINS system requires a 30Hz image processing frequency, and each frame must complete all eight steps within 33ms. The adaptive filtering mechanism in step seven requires real-time adjustment of algorithm parameters based on scene complexity and UAV motion state. The resource trade-off strategy in step eight requires dynamic monitoring of ARS45 and CPU load states to optimize computing resource allocation while ensuring positioning accuracy.

[0095] In terms of data transmission, the system uses the DMA controller of ARS45 to implement the zero-copy mechanism. Image data is directly transmitted from the camera sensor to the memory of ARS45, avoiding multiple copies of image data between CPU and accelerator in traditional solutions. After processing, the feature point coordinates and descriptors are passed to the CPU through the shared memory area for subsequent state estimation and optimization calculations. ARS45 includes a multi-layer cache structure, which can effectively reduce memory access delay and improve feature extraction throughput. The Lucas-Kanade optical flow algorithm in step three is executed in parallel on a multi-scale image pyramid, utilizing the vector computing power of VPU to achieve efficient optical flow tracking, with processing speed improved by more than 5 times compared to pure CPU implementation.

[0096] The difficulty of the hardware configuration of this embodiment lies in the coordination and scheduling between ARS45 and CPU, as well as the dynamic management of heterogeneous resources. Due to the different processing capabilities, memory architectures and delay characteristics of the two processing units, a complex load balancing mechanism needs to be designed. When ARS45 processes steps one and three, CPU can execute steps two and six in parallel, but must ensure the timing consistency of data flow and correct handling of dependency relationships. The adaptive mechanism formed by steps seven and eight requires real-time monitoring of resource utilization of the two processing units, and dynamically adjusting algorithm parameters according to UAV flight state and scene complexity. The complexity of memory management lies in the need to uniformly manage on-chip memory of ARS45, CPU system memory and shared memory area, while handling memory fragmentation, access conflicts and cache consistency issues.

[0097] The innovation difficulty of the embodiment mainly lies in the deep integration and reconstruction of algorithm and hardware. The traditional VINS-Mono system sequentially executes all modules on the CPU, while the embodiment fundamentally reconstructs the entire computing process, decomposes and maps the eight steps to heterogeneous processing units. This is not just a simple task allocation, but requires the redesign of data structures, memory layout, execution order and error handling mechanism. Steps 1 to 8 form a complete closed-loop optimization system. The scene complexity perception of step 7 can adjust the feature selection strategy of step 6 according to the motion state of the unmanned aerial vehicle. The calculation resource perception adjustment of step 8 can dynamically control the feature density distribution of step 5. The system also designs a perfect fault-tolerant mechanism. When ARS45 has a hardware failure, the functions of steps 1 to 4 can automatically switch to CPU execution, although the performance is reduced, but the continuity and reliability of the system are guaranteed. The fault-tolerant design of heterogeneous computing further increases the complexity and innovation value of the system.

[0098] The specific process is as follows: Data acquisition stage: The camera carried by the unmanned aerial vehicle collects 640x480 resolution images at a frequency of 30Hz, and the IMU provides inertial measurement data at a frequency of 170Hz. The two types of data are accurately synchronized through hardware timestamps.

[0099] Front-end hardware acceleration processing: Image data is directly transmitted to the Cool Chip ARS45 hardware accelerator. Step one performs multi-level feature fusion on ARS45. First, a hardware-accelerated KLT optical flow algorithm is used for preliminary tracking, then a BRIEF descriptor matching verification is performed, and finally geometric consistency constraints are introduced to eliminate outliers, outputting about 200 high-quality feature points per frame. Based on these seed feature points, step three uses the SIMD processing unit of ARS45 to perform Lucas-Kanade optical flow tracking on a multi-scale image pyramid. Step four implements forward and backward optical flow calculation, and eliminates mis-matches through consistency checking to ensure the robustness of feature tracking.

[0100] Hardware-CPU data transmission: The zero-copy mechanism of step six ensures efficient transmission of feature point data from ARS45 to the CPU host processor. DMA direct memory access is used to avoid repeated copying of data between user space and kernel space, and the data transmission delay is less than 1ms.

[0101] CPU logic optimization process: step two performs visual-inertial joint optimization on CPU, using the feature point trajectories and IMU pre-integration results provided by the front-end, to estimate the initial scale and system state by solving a nonlinear least squares problem. Step five implements dynamic feature management, which adaptively adjusts the feature density according to the complexity of the scene to ensure uniform distribution of feature points in the image. Step six selects the feature subset that contributes most to positioning accuracy based on information gain theory, limiting the number of feature points involved in the back-end optimization to 100-200. Steps seven and eight form an adaptive adjustment mechanism that adjusts algorithm parameters in real time according to the motion state of the UAV and computing resources.

[0102] Navigation data output: the back-end Bundle Adjustment optimizer performs nonlinear optimization based on the selected feature subset, outputs 6DOF pose estimates and velocity estimates, and updates data at a frequency of 30Hz, meeting the real-time navigation control requirements of the UAV.

[0103] The above examples are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solution to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A heterogeneous-based optimization and acceleration method for unmanned aerial vehicle visual-inertial navigation system, characterized in that, The method comprises the following steps: S1. Optical flow and descriptor fusion: track feature points between consecutive frames by an optical flow algorithm to obtain dense motion information, extract descriptors for key points and match and verify the motion information, and then output a reliable feature point set; S2. Initial scale estimation and optimization: use the set relationship of the reliable feature point set between multiple frames, combine IMU pre-integration data, solve the scale factor by visual-inertial joint optimization, and construct a motion geometric constraint; S3. Feature tracking: based on the reliable feature point set, identify key frames to form seed features for tracking, and track in subsequent frames based on the seed features; S4. Bidirectional optical flow consistency check: based on the tracking results of S3, perform bidirectional optical flow consistency check, and according to a preset tracking error threshold, select feature trajectories with high reliability; S5. Feature management: based on the feature trajectories, perform dynamic density control and quality evaluation of the feature points, and output the managed feature points; S6. Feature selection: based on information gain evaluation, calculate the contribution of each feature point output by step S5 to system state estimation, and select a subset with the largest information amount for backend optimization; S7. Adaptive feature screening: adaptive feature screening is performed on the subset selected in step S6, and the feature selection strategy is dynamically adjusted; S8. Trade-off strategy between computing resources and accuracy: based on the subset selected in step S6, and combined with system resources, dynamically adjust system parameters.

2. The heterogeneous-based optimization and acceleration method for UAV visual-inertial navigation system according to claim 1, wherein, The step S1 is to perform multi-level feature fusion on the ARS45. First, a hardware-accelerated KLT optical flow algorithm is used for preliminary tracking, then BRIEF descriptor matching verification is performed, and finally geometric consistency constraints are introduced to remove outliers, and 200 high-quality feature points are output for each frame.

3. The heterogeneous-based optimization and acceleration method for UAV visual-inertial navigation system according to claim 1, wherein, The step S2 specifically is to obtain the optimal scale and initial state by solving the following optimization objective function: wherein the visual re-projection residual can be expressed as: The IMU pre-integration residual error can be expressed as: wherein is a scale factor, is a system state variable, and are visual re-projection error and IMU pre-integration error, respectively, is a robust kernel function, is a covariance matrix of IMU measurements, is an image feature point observation, is a camera projection function, is a camera rotation matrix, is a camera translation vector, is a 3D landmark point coordinate; is a k-th IMU frame to world coordinate system rotation, is a k-th IMU frame position in world coordinate system, is a k-th IMU frame velocity in world coordinate system, is a k-th frame to k+1-th frame position pre-integration quantity, is a time interval, is a world coordinate system gravity vector, is a k-th frame to k+1-th frame velocity pre-integration quantity, is a k-th IMU frame to world coordinate system rotation quaternion, is a k-th frame to k+1-th frame rotation pre-integration quantity, is a quaternion multiplication operator.

4. The heterogeneous-based optimization and acceleration method for UAV visual-inertial navigation system according to claim 1, wherein, The step S5 specifically is: (1) Dynamic density control: adaptively adjust the feature point density according to the scene complexity and system state; appropriately reduce the density in feature-rich areas, and increase the sampling density in feature-sparse areas to achieve uniform distribution of features; (2) Multi-scale feature selection: extract features at different levels of the image pyramid to ensure that both distant and near targets are covered with sufficient features, and improve the adaptability of the system to scale changes; (3) Feature quality scoring mechanism: consider factors such as corner response value, gradient distribution, and tracking stability to assign a quality score to each feature point, and preferentially retain high-quality features; (4) Predictive feature management: in combination with camera motion prediction, identify feature points that may leave the field of view in advance, extract new features in the corresponding area in a timely manner, and avoid feature shortages.

5. The heterogeneous-based optimization and acceleration method for UAV visual-inertial navigation system according to any one of claims 1 to 4, characterized in that, The step S6 specifically is: The feature point information gain evaluation is based on the following mathematical model: for a feature point its information gain is defined as: where is an eigenvalue of the matrix , is the dimension of the state vector, is a feature point corresponding Jacobian matrix, is the observation noise covariance matrix, and trace() denotes the trace of a matrix.

6. The heterogeneous-based optimization and acceleration method for UAV visual-inertial navigation system according to any one of claims 1 to 4, characterized in that, The step S7 dynamically adjusts the feature selection strategy: (1) Scene complexity awareness: automatically adjust the feature selection threshold by evaluating the number, distribution, and quality of feature points in the current scene, use stricter screening criteria in feature-rich environments, and relax the selection conditions in feature-scarce scenes; (2) Motion pattern adaptation: adjust the feature selection preference according to the motion state of the unmanned aerial vehicle; (3) Multi-objective optimization strategy: Model the feature selection problem as a multi-objective optimization problem, considering both maximum information gain and feature distribution uniformity; use a weighted method to convert the multi-objective problem into a single-objective optimization problem: wherein the uniformity metric is defined as: wherein, is a set of feature points, is the i-th feature point, is the information gain of a feature point, is a weight coefficient of information gain and distribution uniformity, is the number of image partitions, is the number of feature points in the k-th grid; (4) Computing resource-aware adjustment: Monitor system computing load in real time, automatically reduce the number of optimized feature points when computing resources are tight, and appropriately increase the number of feature points when resources are sufficient, to achieve adaptive resource allocation.

7. The heterogeneous-based optimization and acceleration method for UAV visual-inertial navigation system according to any one of claims 1 to 4, characterized in that, The trade-off strategy of S8 is: (1) Multi-stage optimization strategy: Divide the backend optimization into two stages, coarse optimization and fine optimization. Coarse optimization uses fewer feature points to quickly converge and provides preliminary estimates. Fine optimization uses more feature points for fine adjustment when computing resources allow; (2) Keyframe dynamic management: Design a keyframe selection strategy based on information theory, only keep keyframes that significantly contribute to system state estimation; introduce a keyframe redundancy evaluation mechanism to remove information redundant historical keyframes in time, and control the size of the optimization window; (3) Variable optimization window: Dynamically adjust the size of the sliding window according to the system state and computing resources; in high dynamic scenes or unstable feature tracking, appropriately expand the window to improve robustness; in stable scenes, reduce the window to reduce the computational burden; (4) Precision guarantee mechanism: Design a precision monitoring module to evaluate the uncertainty of system state estimation in real time; when the uncertainty exceeds the preset threshold, automatically increase the number of feature points and the number of optimization iterations to ensure that the system accuracy does not decrease significantly.

8. A heterogeneous based unmanned aerial vehicle visual-inertial navigation system optimization and acceleration system, comprising: It includes: Optical flow and descriptor fusion module: track feature points between consecutive frames through optical flow algorithm to obtain dense motion information, and extract descriptors for key points and match them with motion information to output reliable feature point set; Initial scale estimation and optimization module: use the set relationship of reliable feature point set in multiple frames, combined with IMU pre-integration data, to solve the scale factor through visual-inertial joint optimization and construct motion geometric constraints; Feature tracking module: based on the reliable feature point set output by the optical flow and descriptor fusion module, identify keyframes and form seed features for tracking in subsequent frames; Bidirectional optical flow consistency verification module: based on the tracking results of the feature tracking module, perform bidirectional optical flow consistency verification, and select high-reliability feature trajectories according to the preset tracking error threshold; Feature management module: based on the feature trajectories output by the bidirectional optical flow consistency verification module, dynamically control the density and quality of feature points, and output the managed feature points; Feature selection module: based on information gain evaluation, calculate the contribution of each feature point output by the feature management module to system state estimation, and select the subset with the most information for backend optimization; Adaptive feature screening module: adaptive feature screening for the subset, dynamically adjusting the feature selection strategy; Computing resource and precision trade-off strategy module: based on the subset selected by the feature selection module, and combined with system resources to dynamically adjust system parameters.

9. The heterogeneous based unmanned aerial vehicle visual-inertial navigation system optimization and acceleration system of claim 8, wherein, The light flow and descriptor fusion module performs multi-level feature fusion on ARS45, first uses a hardware-accelerated KLT light flow algorithm for preliminary tracking, then verifies through a BRIEF descriptor match, finally introduces a geometric consistency constraint to eliminate outliers, and outputs 200 high-quality feature points per frame.

10. The heterogeneous based unmanned aerial vehicle visual-inertial navigation system optimization and acceleration system of claim 8, wherein, The initial scale estimation and optimization module is specifically: the optimal scale and initial state are obtained by solving the following optimization objective function: wherein the visual re-projection residual can be expressed as: The IMU pre-integration residual error can be expressed as: wherein is a scale factor, is a system state variable, and are visual re-projection error and IMU pre-integration error, respectively, is a robust kernel function, is a covariance matrix of IMU measurements, is an image feature point observation, is a camera projection function, is a camera rotation matrix, is a camera translation vector, is a 3D landmark point coordinate; is a k-th IMU frame to world coordinate system rotation, is a k-th IMU frame position in world coordinate system, is a k-th IMU frame velocity in world coordinate system, is a k-th frame to k+1 frame position pre-integration quantity, is a time interval, is a world coordinate system gravity vector, is a k-th frame to k+1 frame velocity pre-integration quantity, is a k-th IMU frame to world coordinate system rotation quaternion, is a k-th frame to k+1 frame rotation pre-integration quantity, is a quaternion multiplication operator.

Images