Method and apparatus for edge optimization based on visual-inertial residual dynamic adjustment

Abstract

The application provides an edge optimization method and device based on visual-inertial residual dynamic adjustment, and belongs to the technical field of computer vision and robot navigation, and comprises the following steps: a weighted visual re-projection residual vector is spliced with an IMU pre-integral residual vector and a corresponding Jacobian matrix into an information matrix; the information matrix is blocked according to to-be-edged variables and reserved variables, a new information matrix and an information vector are generated by using a Schur complement formula; eigenvalue decomposition is performed, a linearized Jacobian matrix and a linearized residual vector are constructed, finally, an edge prior is generated, and a corresponding prior linearization point is recorded; and the updated prior constraint parameter and the updated prior linearization point are used for next round optimization. The application effectively suppresses abnormal observation interference, and takes into account prior consistency and calculation efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and robot navigation technology, specifically relating to an edge optimization method and device based on dynamic adjustment of vision-inertial residuals. Background Technology

[0002] In sliding window-based SLAM / VIO systems, for real-time performance, the oldest keyframe is typically "marginalized." This involves calculating the Jacobian between the visual reprojection residual vector and the IMU pre-integration residual vector at the current linearization point; using Schur complement to eliminate the states to be removed, resulting in a priori information matrix and vector containing only the retained states; this priori information continues to function as a "ghost" constraint for the next round of nonlinear optimization. To suppress the consistency problem caused by linearization point drift, existing methods generally employ First-Estimate Jacobian (FEJ) to fix the first linearization point, or use dynamic marginalization, information gain filtering, or other methods to remove low-quality observations. These methods still exhibit three major drawbacks in highly dynamic, weakly textured scenes, becoming bottlenecks restricting the long-term robustness of the system.

[0003] 1. Low-quality visual observations contaminate priors: During periods of rapid motion or image blurring, if incorrectly matched reprojection residual vectors are still included in the information matrix with uniform weights, noise will be amplified into the priors, causing subsequent optimization divergence. A mechanism is needed to evaluate the quality of visual observations online and adaptively reweight them to reduce the impact of abnormal observations on the prior noise distribution without compromising real-time performance.

[0004] 2. Disconnect between prior linearization points and the current state: After marginalization, the retained variables will continue to be updated in subsequent sliding windows, and their true state may be far from the original linearization point. Continuing to use the old prior will introduce model inconsistency errors, leading to changes in observability, estimation drift, or even oscillations. A lightweight, on-demand relinearization strategy is needed: when the linearization point offset is detected to exceed a threshold, only the affected prior terms are locally incrementally updated, rather than recalculating the entire graph in every iteration, thus balancing consistency, computational cost, and memory usage.

[0005] 3. Existing compensation schemes are costly and have poor versatility: In order to maintain consistency, existing methods either relinearize all priors in each iteration or maintain a set of auxiliary factor graphs, which are computationally and memory-intensive and difficult to implement on embedded platforms. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an edge-optimization method and apparatus based on dynamic adjustment of visual-inertial residuals. During the edge-optimization process, weights are dynamically calculated for each observed residual vector, suppressing the influence of low-quality visual residual vectors. Simultaneously, the deviation between the current state and the prior linearization point is monitored in real time during the sliding window iteration. When the deviation exceeds a preset threshold, only the prior residual vector and the Jacobian matrix are updated online without reconstructing the complete factor graph, thus balancing prior consistency and computational efficiency, and improving the robustness and positioning accuracy of the VIO system in dynamic scenarios.

[0007] To achieve the above objectives, this invention discloses an edge optimization method based on dynamic adjustment of visual-inertial residuals, the steps of which are as follows: S1. Linearize the visual reprojection residual and the inertial measurement unit pre-integration residual within the sliding window to obtain the visual residual vector, the inertial measurement unit residual vector, and their respective Jacobian matrices. S2. Calculate the weights dynamically based on the current norm of the visual residual vector and the inertial measurement unit residual vector, and construct the information matrix and information vector based on the Jacobian matrix and the residual vector to suppress the influence of low-quality visual observation on the prior. S3. Divide the information matrix into blocks according to the variables to be marginalized and the variables to be retained. Use Schur complement to eliminate the variables to be marginalized, generate marginalization priors and record the corresponding prior linearization points. S4. In subsequent sliding window optimization, continuously compare the deviation between the current state and the prior linearized point. When any deviation exceeds the preset threshold, only perform incremental updates on the prior residual vector and Jacobian matrix without reconstructing the factor graph, and use the updated prior constraints and prior linearized points for the next round of optimization; otherwise, continue using the original prior constraints.

[0008] Furthermore, the dynamic calculation of weights includes: a preset residual norm threshold. and attenuation factor Calculate the residual vector norm The weights are calculated using the following formula. : , in, It is an exponential function with weights. It decreases monotonically as the residual norm increases.

[0009] Furthermore, the process of constructing the information matrix and information vector is as follows: for each residual block Construct a weighted sub-information matrix and weighted sub-information vector The superscript (i) represents the local information matrix or sub-information vector calculated separately for the i-th residual block; where For the first The residual vector of each residual block. For the first The Jacobian matrix corresponding to each residual block For the first The weights of each residual block, with the superscript T indicating matrix transpose; are accumulated across blocks to assemble the global information matrix. and global information vector At that time, if the first The visual residual block and the first The IMU pre-integration residual blocks are mixed and accumulated, using symmetric weighting coefficients. As a coupling coefficient For the first The weights of each IMU pre-integration residual block are adjusted to keep the information matrix symmetric.

[0010] Furthermore, the process of generating marginalized priors and recording prior linearized points is as follows: Increment the global state According to the variables to be marginalized With reserved variables Block partitioning, using Schur complement to eliminate variables to be marginalized. This yields only the retained variables. New information matrix and information vector ; right Perform eigenvalue decomposition , where U is an orthogonal matrix and Σ is a diagonal matrix; let Construct the linearized Jacobian matrix and linearized residual vector , for The transpose of the matrix; diagonal matrix The square root linear factor; Constructing marginalized prior residuals: , To preserve the increment of variables in the next round of optimization, To marginalize prior residuals; save , And the corresponding prior linearization point.

[0011] Furthermore, the prior linearization points include the state values ​​of the variables preserved during marginalization: camera position. Quaternion rotation ,speed IMU bias and the corresponding prior residual vector And Jacobi matrix .

[0012] Furthermore, comparing the deviation between the current state and the prior linearized point includes: obtaining the camera position in the current state. Quaternion rotation ,speed IMU bias The subscript "cur" indicates the latest estimate. , in, , , , These are the position difference threshold, rotation difference threshold, velocity difference threshold, and offset difference threshold, respectively. for The reverse.

[0013] Furthermore, the incremental update includes: calculating the incremental vector between the current state and the prior linearized point. ; Using a first-order Taylor expansion on the original a priori residual vector Update: , In the current state Relinearize and compute the new Jacobian matrix. : , in, This is the overall state vector. Showing partial derivatives, the prior constraints are updated to... And update the prior linearized point to the current state value: Indicates in The value is taken at the location; If no prior failure is detected, then maintain It remains unchanged and will continue to be used in the next iteration.

[0014] An apparatus for an edge optimization method based on dynamic adjustment of visual-inertial residuals includes the following modules: The assembly module linearizes the visual reprojection residual and the IMU pre-integration residual within the sliding window to obtain the residual vector and the Jacobian matrix, respectively. It dynamically calculates the weights based on the norm of the residual vector and constructs the information matrix and information vector based on the Jacobian matrix and the residual vector. The recording module divides the information matrix into blocks according to variables to be marginalized and retained variables, uses Schur complement to eliminate variables to be marginalized, generates marginalization priors, and records the corresponding prior linearization points. The optimization module compares the deviation between the current state and the prior linearized point in subsequent optimizations. When the deviation exceeds the threshold, it only performs incremental updates on the prior residual vector and Jacobian matrix without reconstructing the factor graph, and uses the updated prior constraints and prior linearized points for the next round of optimization.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an edge optimization method based on dynamic adjustment of visual-inertial residuals.

[0016] A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements an edge optimization method based on dynamic adjustment of vision-inertial residuals.

[0017] Beneficial Effects: This invention exhibits strong anti-contamination performance. Through adaptive weighting based on the residual vector norm, it automatically weakens abnormal visual observations in blurred / sparse scenes, significantly reducing the probability of prior contamination by low-quality data and improving system robustness. This invention replaces full-image recalculation with threshold-triggered local relinearization, performing incremental updates only when the prior linearization point deviates significantly from the current state. This maintains consistency between the prior and the true state while minimizing computational load. This invention eliminates the need to maintain auxiliary factor graphs and avoids global linearization in each round, reducing overall algorithm complexity and enabling real-time operation on ARM or MCU-level embedded platforms. The dynamic weighting and local updates of this invention are implemented as plug-ins, not bound to specific visual front-ends or IMU pre-integration models. They can be seamlessly embedded into mainstream sliding window VIOs such as OKVIS and VINS-Mono, achieving "plug and play." Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the edge optimization method based on dynamic adjustment of visual-inertial residuals according to the present invention.

[0019] Figure 2 This is a schematic diagram of the edge optimization device based on dynamic adjustment of visual-inertial residual vector according to the present invention.

[0020] Figure 3 This is a schematic diagram showing the performance comparison of the present method, the original trajectory, comparison algorithm 1, and comparison algorithm 2 for the test sequence set in this embodiment of the invention.

[0021] Figure 4 This is a schematic diagram showing the performance comparison of the present method, the original trajectory, comparison algorithm 1, and comparison algorithm 2 for the test sequence set in this embodiment of the invention.

[0022] Figure 5This is a schematic diagram showing the performance comparison of the present method, the original trajectory, comparison algorithm 1, and comparison algorithm 2 for the test sequence set 3 in this embodiment of the invention.

[0023] Figure 6 This is a schematic diagram showing the performance comparison of the present method, the original trajectory, comparison algorithm 1, and comparison algorithm 2 for the test sequence set 4 in this embodiment of the invention.

[0024] Figure 7 This is the test sequence set 1 collected in an office setting in this embodiment of the invention.

[0025] Figure 8 This is the test sequence set 2 collected in an office setting in this embodiment of the invention.

[0026] Figure 9 This is the test sequence set 3 collected in an office setting in this embodiment of the invention.

[0027] Figure 10 This is the test sequence set 4 collected in an office setting in this embodiment of the invention. Detailed Implementation

[0028] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the present invention provides an edge optimization method based on dynamic adjustment of visual-inertial residuals, which is executed by a computing device and specifically includes the following steps: Step 1: Preprocessing and dynamic weighted assembly of residual vectors This step aims to suppress the contamination of marginalization priors by low-quality visual observations at the source. Specifically, the computing device first extracts features from the input left-eye image (no processing is required for the right-eye image here), and simultaneously evaluates the image using a Laplacian transform to obtain quality information (such as blur level and texture richness). This image quality information will serve as an important basis for subsequent residual weight allocation.

[0030] Subsequently, the residuals within the sliding window are linearized and dynamically weighted, specifically including: Step 1.1, Residual Vector Linearization and Norm Calculation: For each frame of visual observation within the sliding window, calculate the reprojection residual vector. and the corresponding Jacobian matrix For each IMU pre-integration observation segment, calculate the pre-integration residual vector. and the corresponding Jacobian matrix .

[0031] Among them, superscript as well as These represent visual reprojection observations and IMU pre-integration observations, respectively; subscript Indicates the first Frame visual observation, Indicates the first Segment IMU pre-integration observations ( ); For the first The residual dimension (i.e., the number of feature points) of frame visual observation. For the first The residual dimension of the segment IMU pre-integration. This is a rotation matrix used for rotational transformations between coordinate systems. is the dimension of the global state increment vector.

[0032] Calculate the norm of each residual vector block: ; in, To find the norm, Let k be the k-th element in the residual vector, where k is the index value.

[0033] Step 1.2, Calculate weights: Preset residual norm threshold. and attenuation factor ( Set based on image quality experience, for example, set it to 1.5 times the average residual value; (used to control the decay rate). For the first... The visual residual vector or the first The weights of the IMU residual vectors are calculated according to the following formula. : ; in, This represents an exponential function. The weight... It decreases monotonically as the residual norm increases, thus automatically weakening the contribution of anomalous observations such as fuzziness and occlusion.

[0034] Step 1.3, Weighted Information Matrix and Vector Assembly: For each residual vector block, weights are introduced. Then construct the weighted information submatrix and weighted information subvector: ; Where the superscript T denotes the transpose of the matrix, and The first Scalar weights of the j-th visual residual block and the j-th IMU pre-integration residual block. For the first One residual block, For the first The residual vector of each residual block. Let be the Jacobian matrix of the residual block with respect to the global state increment. For the first The weighted information subvector of the block. Superscript This represents the local (sub)information matrix or sub-information vector calculated individually for the i-th visual residual block. (Superscript) This represents the local (sub)information matrix or sub-information vector calculated separately by the j-th IMU pre-integration residual block.

[0035] When assembling the global information matrix A and the global information vector b across blocks, if the i-th visual residual block and the k-th IMU pre-integration residual block are mixed and accumulated, and symmetric weighting is used... As a coupling coefficient, it ensures the symmetry of the information matrix. Geometric mean is used in this process. This is to ensure that the final assembled information matrix A maintains strict symmetry, which is a necessary condition for subsequent nonlinear optimization solutions.

[0036] Finally, the complete weighted information matrix is ​​obtained. with weighted information vector ,in For the dimensions of variables to be marginalized, To preserve the dimensions of variables.

[0037] Step 2, Weighted Schur Complement Marginalization: This step eliminates the variables to be marginalized using Schur complement, generating marginalization priors. Specifically, the global state increment is... According to the parts to be marginalized (Pose, velocity, offset, etc. of the oldest frame) and the reserved portion The state of subsequent frames is divided into blocks, and the corresponding information matrix A and information vector b are represented as follows: ; in, , , and In accordance with and Sub-information matrix after block division. , For the corresponding and The segmented information vector.

[0038] Set the objective function as: Set the objective function ;about Finding the extreme values ​​yields a system of linear equations: ; Thus, the optimal solution is obtained: ; Among them, superscript Indicates that in a given The optimal solution at that time for The inverse matrix.

[0039] Will Substituting back into the objective function and eliminating the variables to be marginalized, we obtain only the variables to be retained. The related minimization problem corresponds to the new global information matrix. With information vector ;right Perform eigenvalue decomposition: ; in, The orthogonal matrix formed by all eigenvectors To include all eigenvalues diagonal matrix, To include all eigenvalues diagonal matrix, This number of records is a characteristic.

[0040] Here, diag() represents a diagonal matrix.

[0041] set up , S is The square root linear factor is defined as the linearized Jacobian. With linearized residual vector : ; This leads to the construction of marginalized prior residuals: ; in, To preserve the increment of variables in the next round of optimization, To marginalize prior residuals, the computing device will... It is saved together with the corresponding linearization point (i.e., the state value at the edge) as a prior constraint for subsequent sliding window optimization.

[0042] Step 3, Prior Consistency Detection and Online Update: Since subsequent sliding window iterations will cause changes in the state estimate, continuing to use the old linearization points will lead to a severe disconnect between the prior constraints and the current state. This step achieves real-time monitoring and low-overhead updating of the prior.

[0043] Step 3.1, Prior linearization point saving: After completing the first step... After frame marginalization, all retained variables at the prior linearization points (e.g., from frame 2 to frame 3) will be removed. The current state value of each frame is saved one by one, denoted as: ; in, Indicates the camera position. Represents quaternion rotation, Indicates linear velocity. This indicates the IMU bias, and the subscript `prior` indicates prior data. This section stores not only numerical values ​​but also the corresponding prior residual vectors. Jacobian matrix .

[0044] Step 3.2, Real-time Consistency Check: In the subsequent sliding window nonlinear optimization iteration, the latest estimated value of the currently retained variable is obtained after each update, denoted as: ; In this context, the subscript "cur" indicates the latest estimated value.

[0045] Compare the deviation between the current state and the prior linearization point. If any of the following conditions are met, the prior linearization point is considered invalid, and the prior residual vector and Jacobian matrix need to be updated online: ; in, , , , These are the position difference threshold, rotation (quaternion angle increment) difference threshold, velocity difference threshold, and offset difference threshold, respectively. This is a special multiplication symbol for quaternions. The above thresholds can be adaptively adjusted according to specific application scenarios (such as high dynamic or weak texture environments).

[0046] Step 3.3, Online Update of Prior Residual Vector and Jacobian Matrix: When prior failure is detected (specifically, when the current estimate of the retained variable deviates too much from the prior linearization point (the state recorded during marginalization), the increment vector between the current state and the prior linearization point is calculated. Specifically, the differences for camera pose, velocity, and offset are: , , in, , , and These are the interpolations for the current state and the prior linearized point record states, including position, attitude, velocity, and velocity bias.

[0047] The differences of all retained variables are stacked vertically in a fixed order to form the overall increment. ; Using a first-order Taylor expansion on the original a priori residual vector Update: ; in, This is the updated marginalized linear residual.

[0048] Meanwhile, in the current state At this point, the prior residual vector is relinearized, and a new Jacobian matrix is ​​calculated. : ; in, This is the overall state vector. This represents the partial derivative.

[0049] The updated prior constraints are And update the prior linearized point to the current state value: If no prior failure is detected, then maintain It remains unchanged and will continue to be used in the next iteration.

[0050] Step 3.4, Subsequent sliding window iterations and next marginalization: The updated prior constraints are used as new marginalization constraints, and together with the visual observation residual vector and IMU observation residual vector in the current sliding window, they participate in the next nonlinear optimization.

[0051] When the sliding window slides again and the oldest frame needs to be marginalized, the weighted Schur marginalization is performed again according to steps 3.1-3.2, and a new prior is generated.

[0052] Through steps 1 to 3 above, the present invention can effectively suppress the interference of high residual vector observations (such as blurring or occlusion) on the prior; when the state drift is large, only the prior residual vector and Jacobian matrix are updated with low overhead, without the need to reconstruct the complete factor graph, thus taking into account both prior consistency and real-time performance.

[0053] Example: Testers walked along a closed path within an office, holding a D455 camera equipped with a vision sensor and an IMU. During some tests, varying degrees of left-right or up-down shaking were applied to create visual degradation conditions such as motion blur, weak texture, repetitive texture, and partial occlusion. The camera images were acquired at a frequency of 20Hz, and the IMU measurement data was synchronized with the camera image data.

[0054] Four sets of indoor sequences were obtained, namely Test 1, Test 2, Test 3 and Test 4.

[0055] Test 1 is an indoor sequence with rich texture and local motion blur, such as Figure 7 As shown; Figure 7 The sequence contains six photographs taken by a tester holding a D455 camera and moving along a closed office route while subjecting the camera to left-right shaking. The images were taken at intervals of one frame every ten frames. The table, chairs, display screen, and object outlines in this scene are relatively clear, and the overall texture is rich, making it a suitable benchmark test scene for a typical indoor environment. However, some images suffer from localized motion blur due to camera shake, which may increase visual feature matching errors and reprojection residuals. This sequence is used to examine the ability of the dynamic weighting mechanism based on the residual norm of this invention to reduce the impact of abnormal visual observations on marginalized priors under conditions of both normal texture and localized visual degradation.

[0056] Test 2 consists of weak and repetitive texture indoor sequences, such as... Figure 8 As shown; Figure 8 The sequence contains six photographs taken by a tester using a handheld D455 camera while moving through an office area containing weakly textured planes and repetitive structures. The images were taken at intervals of one frame every ten frames. The scene includes weakly textured areas such as white walls and desktops, along with locally repetitive structures like partitions and chairs, which can easily lead to a reduction in the number of usable visual features, ambiguity in feature matching, and an increase in visual reprojection residuals. This sequence is used to examine the ability of this invention to reduce the weight of low-quality visual observations based on the residual norm under conditions of weak and repetitive textures, and to reduce the impact of anomalous observations on the prior information matrix during the marginalization process.

[0057] Test 3 is an indoor sequence involving partial occlusion and visual disturbance, such as... Figure 9 As shown; Figure 9 The six included images are continuously captured by a tester holding a D455 camera and moving it left-right and up-down while facing a door frame or obvious structural boundary area. The images were taken at intervals of one frame every 10 frames. The scene includes local occlusion relationships formed by door frames, workstation partitions, and object boundaries, accompanied by repetitive textures and significant viewpoint changes, which easily lead to short-term mismatches and large visual reprojection residuals at occlusion edges and repetitive structural areas. This sequence is used to examine whether the residual dynamic weighting mechanism of this invention can mitigate the impact of mismatched observations on marginalized priors, and whether the prior consistency detection and incremental update mechanism can suppress trajectory drift caused by state shifts.

[0058] Test 4 consists of an indoor sequence with strong motion and large angular changes, such as... Figure 10 As shown; Figure 10The sequence includes six images captured by a tester using a handheld D455 camera while rotating and translating around a fixed workstation, simultaneously subjecting the tester to significant left-right shaking. The images were captured consecutively at intervals of one frame every ten frames. This sequence, acquired during rotation and translation around a fixed workstation scene, is prone to motion blur, short-term observation instability, and increased deviation between the current state and the prior linearized point due to significant camera shaking and viewpoint changes. This sequence is used to examine the invention's ability to reduce the impact of abnormal visual observations under strong camera perturbation conditions through dynamic residual weighting, and to incrementally update the prior residual vector and Jacobian matrix when the state deviation exceeds a threshold, thereby maintaining marginalized prior consistency and trajectory stability.

[0059] The trajectory results corresponding to the four indoor test sequence sets (Test 1, Test 2, Test 3, and Test 4) are as follows: Figure 3 , Figure 4 , Figure 5 and Figure 6 As shown, Figure 3 , Figure 4 , Figure 5 and Figure 6 All coordinates are local Cartesian coordinates, with the horizontal axis as the X-axis and the vertical axis as the Y-axis to represent direction. The origin 0 in the figure is taken as the initial position of movement, and the coordinate unit is meters. The coordinates of each point in the figure represent the directional offset of the trajectory point relative to the initial position 0, rather than the movement distance. The distance between any two points is calculated by the square root of the sum of the squares of the coordinate differences, and is always non-negative. The result after dynamic edge-optimization (the method of this application) is compared with the original trajectory (the trajectory without edge-optimization), Algorithm 1 (traditional fixed linearization point edge-optimization method), and Algorithm 2 (dynamic edge-optimization method based on auxiliary factor map). The performance is evaluated by the magnitude of the final closure error. Through four tests, the optimized algorithm obtained the minimum closure error in all cases. The positioning accuracy is evaluated by calculating the closure error between the end point and the start point of the closed path. The Euclidean distance between the end point and the start point of the trajectory in the same coordinate system is used as the closure error. The smaller the closure error, the smaller the cumulative positioning error on the closed path. The trajectory comparison results of the four test sequence sets are as follows: Figure 3 , Figure 4 , Figure 5 and Figure 6 As shown in the image.

[0060] Based on the accompanying drawings and actual test results, the technical solution of this application has achieved the following significant beneficial effects: 1. Strong anti-pollution performance and significantly improved system robustness in complex dynamic scenarios: In the four sets of tests, due to the violent shaking of the test personnel, visual observation was prone to blurring and mismatches. For example... Figure 4 and Figure 5As shown, Algorithm 2 (purple trajectory) exhibited severe divergence due to its failure to effectively suppress contamination from anomalous observations. In Test 2, the Y-axis drifted to over 3.5 meters, and in Test 3, it even drifted to 6 meters. Algorithm 1 (yellow trajectory) also showed obvious spikes on some paths. In contrast, the method in this application (orange-red trajectory) automatically weakens the influence of low-quality visual observations on marginalized priors through an adaptive weighting mechanism based on the residual vector norm. Figures 3 to 6 It can be seen that the trajectory obtained by the method in this application maintains a good overall closed shape, the distance between the trajectory endpoint and the starting point is smaller than that of the corresponding comparative method, and no obvious divergence was observed in tests with obvious motion blur, weak texture, or occlusion as shown by comparative algorithm 2. The results show that dynamic weighting based on residual norm can reduce the impact of abnormal visual observations on marginalized priors, and threshold-triggered incremental prior updates help reduce the cumulative error caused by the disconnect between prior linearized points and the current state. 2. Threshold-triggered local incremental update achieves a balance between high accuracy and lightweight real-time: This application abandons the state disconnect problem caused by traditional fixed linearized points, and also avoids the high computational cost of global recalculation or maintenance of auxiliary factor graphs. Observation Figure 3 , Figure 4 , Figure 5 and Figure 6 As can be seen, the dynamic edge-forming trajectory (orange-red) of the proposed method achieved the best closure effect in all four sets of tests, with a closure error significantly smaller than that of comparative algorithms 1 and 2. Particularly in test 3, where the shaking was most severe (severe image blurring), comparative algorithm 2 exhibited significant drift, and the closure error of comparative algorithm 1 also increased significantly (the trajectory lines became thicker and deviated from the start and end points), while the proposed method still smoothly returned to the vicinity of the starting point. This proves that the proposed method only performs local relinearization when the deviation exceeds the threshold, maintaining a high degree of consistency between the prior and the actual state while reducing the overall algorithm complexity. This allows for real-time operation on ARM or MCU-level embedded platforms, achieving a win-win situation in terms of both accuracy and computational power.

[0061] 3. Plug-in architecture design with excellent engineering portability: The dynamic weighting and local incremental update mechanisms in this application are implemented as plug-ins, not bound to a specific visual front-end or IMU pre-integration model, and do not require maintaining a complete auxiliary factor graph. The algorithm framework used in the experiments can be seamlessly embedded into mainstream sliding window VIO frameworks such as OKVIS and VINS-Mono. Figure 3 , Figure 4 , Figure 5 and Figure 6As can be seen, the method of this application maintains extremely high trajectory smoothness under different test scenarios, and has a significant positioning advantage compared with the original trajectory and the comparison algorithm. This proves that the method has good versatility and scalability, and can achieve "plug and play", which greatly reduces the difficulty of engineering implementation under different hardware platforms and application scenarios.

[0062] In summary, the comparative experiments in four office scenarios fully demonstrate that this application effectively solves the technical pain points of low-quality observation contamination of priors, large computational overhead, and poor prior consistency in existing edge computing technologies, and has achieved significant technical progress in terms of positioning accuracy, system robustness, and computational real-time performance.

[0063] like Figure 2 As shown, the present invention also provides an edge optimization device based on dynamic adjustment of visual-inertial residual vectors to implement the above method, comprising the following modules: The assembly module linearizes all visual reprojection residuals and IMU pre-integration residual vectors within the sliding window, calculates their residual vectors and Jacobian matrices, dynamically calculates the corresponding weights based on the current norm of the visual reprojection residual vectors and IMU pre-integration residual vectors, and assembles the weighted visual reprojection residual vectors, IMU pre-integration residual vectors, and corresponding Jacobian matrices into an information matrix; IMU represents the inertial measurement unit. The recording module divides the information matrix into blocks according to the variables to be marginalized and the retained variables, and uses the Schur complement formula to generate new information matrices and information vectors; it performs eigenvalue decomposition, constructs linearized Jacobian matrices and linearized residual vectors, and finally generates marginalization priors and records their corresponding prior linearization points; the prior linearization points represent the state values ​​of the retained variables during marginalization. In the optimization module, the deviation between the current state and the prior linearized point is continuously compared during optimization. When any deviation exceeds the threshold, only the prior residual vector and Jacobian matrix are incrementally updated without reconstructing the factor graph. The updated prior constraint parameters and the updated prior linearized point are then used for the next round of optimization.

[0064] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

Claims

1. An edge optimization method based on dynamic adjustment of visual-inertial residuals, characterized in that, The steps are as follows: S1. Linearize the visual reprojection residual and the inertial measurement unit pre-integration residual within the sliding window to obtain the visual residual vector, the inertial measurement unit residual vector, and their respective Jacobian matrices. S2. Calculate the weights dynamically based on the current norm of the visual residual vector and the inertial measurement unit residual vector, and construct the information matrix and information vector based on the Jacobian matrix and the residual vector to suppress the influence of low-quality visual observation on the prior. S3. Divide the information matrix into blocks according to the variables to be marginalized and the variables to be retained. Use Schur complement to eliminate the variables to be marginalized, generate marginalization priors and record the corresponding prior linearization points. S4. In subsequent sliding window optimization, continuously compare the deviation between the current state and the prior linearized point. When the state deviation exceeds the preset threshold, only perform incremental updates on the prior residual vector and Jacobian matrix without reconstructing the factor graph, and use the updated prior constraints and prior linearized points for the next round of optimization; otherwise, continue using the original prior constraints.

2. The edge optimization method based on dynamic adjustment of visual-inertial residuals according to claim 1, characterized in that, Dynamic weight calculation includes: preset residual norm threshold and attenuation factor Calculate the residual vector norm The weights are calculated using the following formula. : ， in, It is an exponential function with weights. It decreases monotonically as the residual norm increases.

3. The edge optimization method based on dynamic adjustment of visual-inertial residuals according to claim 2, characterized in that, The process of constructing the information matrix and information vector is as follows: for each residual block Construct a weighted sub-information matrix and weighted sub-information vector ,in For the first The residual vector of each residual block. For the first The Jacobian matrix corresponding to each residual block For the first The weights of each residual block, with the superscript T indicating matrix transpose; are accumulated across blocks to assemble the global information matrix. and global information vector At that time, if the first The visual residual block and the first The IMU pre-integration residual blocks are mixed and accumulated, using symmetric weighting coefficients. As a coupling coefficient For the first The weights of each IMU pre-integration residual block are adjusted to keep the information matrix symmetric.

4. The edge optimization method based on dynamic adjustment of visual-inertial residuals according to claim 3, characterized in that, The process of generating marginalized priors and recording prior linearized points is as follows: Increment the global state According to the variables to be marginalized With reserved variables Block partitioning, using Schur complement to eliminate variables to be marginalized. This yields only the retained variables. New information matrix and information vector ; right Perform eigenvalue decomposition , where U is an orthogonal matrix and Σ is a diagonal matrix; let Construct the linearized Jacobian matrix and linearized residual vector , Constructing marginalized prior residuals: , To preserve the increment of variables in the next round of optimization, To marginalize prior residuals; save , And the corresponding prior linearization point.

5. The edge optimization method based on dynamic adjustment of visual-inertial residuals according to claim 4, characterized in that, Prior linearization points include the state values ​​of variables preserved during marginalization: camera position. Quaternion rotation ,speed IMU bias and the corresponding prior residual vector And Jacobi matrix .

6. The edge optimization method based on dynamic adjustment of visual-inertial residuals according to claim 5, characterized in that, Comparing the deviation between the current state and the prior linearized point includes: obtaining the camera position in the current state. Quaternion rotation ,speed IMU bias The subscript "cur" indicates the latest estimate. ， in, , , , These are the position difference threshold, rotation difference threshold, velocity difference threshold, and offset difference threshold, respectively. for The reverse.

7. The edge optimization method based on dynamic adjustment of visual-inertial residuals according to claim 6, characterized in that, Incremental updates include: calculating the increment vector between the current state and the prior linearized point. ; Using a first-order Taylor expansion on the original a priori residual vector Update: ， In the current state Relinearize and compute the new Jacobian matrix. : ， in, This is the overall state vector. Showing partial derivatives, the prior constraints are updated to... And update the prior linearized point to the current state value: If no prior failure is detected, then maintain It remains unchanged and will continue to be used in the next iteration.

8. An apparatus for the edge optimization method based on dynamic adjustment of visual-inertial residuals as described in any one of claims 1-7, characterized in that, Includes the following modules: The assembly module linearizes the visual reprojection residual and the IMU pre-integration residual within the sliding window to obtain the residual vector and the Jacobian matrix, respectively. It dynamically calculates the weights based on the norm of the residual vector and constructs the information matrix and information vector based on the Jacobian matrix and the residual vector. The recording module divides the information matrix into blocks according to variables to be marginalized and retained variables, uses Schur complement to eliminate variables to be marginalized, generates marginalization priors, and records the corresponding prior linearization points. The optimization module compares the deviation between the current state and the prior linearized point in subsequent optimizations. When the deviation exceeds the threshold, it only performs incremental updates on the prior residual vector and Jacobian matrix without reconstructing the factor graph, and uses the updated prior constraints and prior linearized points for the next round of optimization.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the edge optimization method based on dynamic adjustment of visual-inertial residuals as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the edge optimization method based on dynamic adjustment of visual-inertial residuals as described in any one of claims 1 to 7.

Images