State estimation method, device, and electronic device for visual inertial odometry

Abstract

The present disclosure discloses a state estimation method, device, and electronic device for a visual inertial odometry, the method comprising: obtaining a current image frame and inertial measurement data; when the current image frame is a key frame, adding the current image frame to a set of key image frames determined based on a sliding window; determining a first inertial residual based on the inertial measurement data; updating a first inertial constraint and a first visual constraint based on the first inertial residual and the first visual residual; estimating the visual inertial odometry state corresponding to the key image frame within the sliding window through iterative optimization; in each iteration, determining incremental change data of the visual inertial odometry state of the current iteration process relative to the previous iteration process based on marginalized prior constraints, the first visual constraints, and the first inertial constraints, and optimizing the visual inertial odometry state based on the incremental change data.

Description

Technical Field

[0001] The present disclosure relates to the field of visual positioning technology, and in particular to a state estimation method, device, and electronic device for a visual inertial odometry. Background Art

[0002] Visual-inertial odometry (VIO) utilizes visual information (images) acquired by a camera, inertial measurement data from an IMU (inertial measurement unit), and locality between consecutive image frames to determine the camera's pose and recover the scene's three-dimensional structure. This approach meets the requirements for simultaneous localization and 3D reconstruction for accuracy, robustness, and low cost. In practical applications, VIO state estimation typically employs graph optimization methods. Some VIO systems utilize open-source general-purpose nonlinear least-squares solvers. However, to accommodate diverse optimization problems, these solvers typically employ batch-based least-squares algorithms, sacrificing efficiency. While they can run in real time on some platforms, their practical application scenarios are limited, for example, they cannot be applied to devices such as terminals equipped with low-cost computing units. Therefore, efficiently and accurately determining the VIO state without consuming excessive computing resources remains a key technical challenge. Summary of the Invention

[0003] Embodiments of the present disclosure provide a state estimation method, device, and electronic device for a visual inertial odometry.

[0004] In a first aspect, an embodiment of the present disclosure provides a state estimation method for a visual inertial odometry, comprising:

[0005] Obtain the current image frame captured by the image acquisition device integrated with the visual inertial odometry, as well as the inertial measurement data output by the inertial measurement unit integrated with the visual inertial odometry;

[0006] When the current image frame is a key frame, adding the current image frame to a key image frame set determined based on a sliding window;

[0007] determining a first inertial residual between the current image frame and the previous adjacent image frame based on the inertial measurement data between the current image frame and the previous adjacent image frame;

[0008] updating a first inertial constraint and a first visual constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set;

[0009] For the key image frames within the sliding window, the visual-inertial odometry state corresponding to the key image frames is estimated through iterative optimization; wherein, in each iterative process, based on the marginalization prior constraint condition, the first visual constraint condition, and the first inertial constraint condition, the incremental change data of the visual-inertial odometry state in the current iterative process relative to the previous iterative process is determined, and the visual-inertial odometry state is optimized based on the incremental change data.

[0010] Furthermore, updating a first visual constraint and a first inertial constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set includes:

[0011] Calculating the first visual residual between the current image frame and other key image frames in the sliding window;

[0012] A first visual constraint condition and a first inertial constraint condition are updated based on the first visual residual and the first inertial residual, respectively.

[0013] Furthermore, when the current image frame is a key frame, adding the current image frame to a set of key image frames determined based on a sliding window includes:

[0014] determining the number of key image frames in the set of key image frames defined by the sliding window;

[0015] When the number of the key image frames is greater than or equal to a first preset threshold, updating the marginalization prior constraint condition based on inertial residuals and visual residuals between the key frame to be eliminated and other key image frames in the key image frame set, and eliminating the key frame to be eliminated from the key image frame set defined by the sliding window;

[0016] The current image frame is added to the key image frame set defined by the sliding window.

[0017] Furthermore, in each iteration, based on the marginalized prior constraint, the first visual constraint, and the first inertial constraint, determining incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration, and optimizing the visual-inertial odometry state based on the incremental change data, including:

[0018] Constructing a current state optimization incremental equation for the visual-inertial odometry state based on the marginalized prior constraint, the first visual constraint, the first inertial constraint, and the previous state optimization incremental equation constructed during the previous iteration of the visual-inertial odometry state; wherein the incremental portion of the current state optimization incremental equation relative to the previous state optimization incremental equation corresponds to an increment of a target state, and the target state includes the visual-inertial odometry state whose change value before and after the optimization of the visual-inertial odometry state during the previous iteration is greater than a second preset threshold;

[0019] The current state optimization incremental equation is solved to obtain an optimized value of the visual inertial odometry state in the current iteration process.

[0020] Furthermore, solving the current state optimization increment equation to obtain the optimized value of the visual inertial odometry state in the current iteration process includes:

[0021] Constructing a Schur complement equation for eliminating the three-dimensional space point state based on the current state optimization incremental equation;

[0022] Solving the Schur complement equation to obtain an increment of the camera pose state and an increment of the motion state in the visual inertial odometry state;

[0023] The increment of the three-dimensional space point is solved based on the increment of the camera posture state and the increment of the motion state.

[0024] Furthermore, when the number of the key image frames is greater than or equal to a first preset threshold, updating the marginalization prior constraint condition based on the inertial residual and the visual residual between the key frame to be eliminated and other key image frames in the key image frame set, and eliminating the key frame to be eliminated from the sliding window, including:

[0025] Constructing an intermediate incremental equation based on the second inertial constraint, the second visual constraint, and the previous condition update incremental equation; wherein the previous condition update incremental equation is the incremental equation constructed during the previous update of the marginalization prior constraint; the second inertial constraint and the second visual constraint are the constraints associated with the preset key frame in the first inertial constraint and the first visual constraint, respectively;

[0026] Removing the visual-inertial odometry state of the preset key frame from the intermediate incremental equation by a Schur complement operation to obtain a current conditional update incremental equation corresponding to the marginalized prior constraint condition;

[0027] The marginalization prior constraint condition is updated by using the current condition to update the incremental equation.

[0028] Furthermore, the method further comprises:

[0029] When the current image frame is a non-key frame, determining at least one matching key frame in the key image frame set that matches the current image frame, and three-dimensional space point coordinates corresponding to matching feature points between the current image frame and the at least one matching key frame;

[0030] calculating a second visual residual between the current image frame and the at least one matching key frame, and calculating a second inertial residual based on inertial measurement data between the current image frame and a previous adjacent image frame;

[0031] constructing a non-keyframe visual constraint condition based on the second visual residual, and constructing a non-keyframe inertial constraint condition based on the second inertial residual;

[0032] A target value of the video inertial odometry state corresponding to the current image frame is obtained based on the non-keyframe visual constraint condition, the non-keyframe inertial constraint condition, the known value of the video inertial odometry state corresponding to the at least one matching keyframe, and the three-dimensional space point coordinates.

[0033] In a second aspect, an embodiment of the present invention provides a state estimation device for a visual inertial odometry, comprising:

[0034] a first acquisition module configured to acquire a current image frame captured by an image acquisition device integrated with the visual inertial odometry and inertial measurement data output by an inertial measurement unit integrated with the visual inertial odometry;

[0035] a first determining module configured to determine a first inertial residual between the current image frame and the previous adjacent image frame based on the inertial measurement data between the current image frame and the previous adjacent image frame;

[0036] an adding module configured to, when the current image frame is a key frame, add the current image frame to a set of key image frames determined based on a sliding window;

[0037] an updating module configured to update a first inertial constraint and a first visual constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set;

[0038] An estimation module is configured to estimate, for the key image frames within the sliding window, the visual-inertial odometry state corresponding to the key image frames through iterative optimization; wherein, in each iteration, based on the marginalization prior constraint, the first visual constraint, and the first inertial constraint, the incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration is determined, and the visual-inertial odometry state is optimized based on the incremental change data.

[0039] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0040] In one possible design, the apparatus includes a memory and a processor. The memory is configured to store one or more computer instructions that enable the apparatus to perform the corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The apparatus may also include a communication interface for communicating with other devices or a communication network.

[0041] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising a visual inertial odometry, a memory, and a processor; wherein the memory is used to store one or more computer instructions, wherein, based on the image data and inertial measurement data output by the visual inertial odometry, the one or more computer instructions are executed by the processor to implement the method described in any one of the above aspects.

[0042] In a fourth aspect, an embodiment of the present disclosure provides a method for implementing augmented reality on the electronic device described in the third aspect, wherein the posture of the electronic device and the three-dimensional space around the electronic device are determined based on the method described in the first aspect.

[0043] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, which includes computer instructions involved in executing the method described in any of the above-mentioned aspects.

[0044] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0045] The present disclosure provides a state estimation method for a visual-inertial odometry, the method comprising: obtaining a current image frame captured by an image acquisition device integrated with the visual-inertial odometry, and inertial measurement data output by an inertial measurement unit integrated with the visual-inertial odometry; when the current image frame is a key frame, adding the current image frame to a set of key image frames determined based on a sliding window; determining a first inertial residual between the current image frame and a previous adjacent image frame based on the inertial measurement data between the current image frame and the previous adjacent image frame; updating a first visual constraint and a first inertial constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the set of key image frames; estimating the visual-inertial odometry state corresponding to the key image frames within the sliding window through iterative optimization; wherein, in each iteration, incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration is determined based on a marginalized prior constraint, the first visual constraint, and the first inertial constraint, and the visual-inertial odometry state is optimized based on the incremental change data. When iteratively optimizing the visual-inertial odometry state, the above method determines the incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration by marginalizing the prior constraints and the first visual constraints and first inertial constraints updated based on the current image frame during each iteration. The optimized value of the visual-inertial odometry state and the coordinates of the three-dimensional space point in the current iteration are then obtained based on this incremental change data. The disclosed embodiment reduces the update scale of the visual-inertial odometry state during the iterative optimization process by combining an incremental solution process with a sliding window marginalization operation, thereby improving the efficiency of visual-inertial odometry state estimation and enhancing the accuracy of visual-inertial odometry state estimation by using marginalized prior information.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0048] Figure 1 A flowchart showing a state estimation method of a visual inertial odometry according to an embodiment of the present disclosure is shown;

[0049] Figure 2 A schematic diagram illustrating a process flow of processing an input image frame during state estimation of a visual inertial odometry according to an embodiment of the present disclosure is shown;

[0050] Figure 3 A schematic diagram illustrating a framework of a mobile phone positioning and tracking application scenario according to an embodiment of the present disclosure is shown;

[0051] Figure 4 3 is a schematic structural diagram of an electronic device suitable for implementing a state estimation method of a visual inertial odometry according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0053] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts, or combinations thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, components, parts, or combinations thereof exist or are added.

[0054] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] The details of the embodiments of the present disclosure are described in detail below through specific examples.

[0056] Figure 1 FIG. 1 is a flow chart showing a state estimation method of a visual inertial odometer according to an embodiment of the present disclosure. Figure 1 As shown, the image processing method includes the following steps:

[0057] In step S101, a current image frame captured by an image acquisition device integrated with a visual inertial odometry and inertial measurement data output by an inertial measurement unit integrated with the visual inertial odometry are acquired;

[0058] In step S102, when the current image frame is a key frame, the current image frame is added to a key image frame set determined based on a sliding window;

[0059] In step S103, a first inertial residual between the current image frame and the previous adjacent image frame is determined based on the inertial measurement data between the current image frame and the previous adjacent image frame;

[0060] In step S104, a first inertial constraint condition and a first visual constraint condition are updated based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set;

[0061] In step S105, for the key image frames in the sliding window, the visual inertial odometry state corresponding to the key image frames is estimated by iterative optimization; wherein, in each iteration process, based on the marginalization prior constraint condition, the first visual constraint condition and the first inertial constraint condition, the incremental change data of the visual inertial odometry state in the current iteration process relative to the previous iteration process is determined, and the visual inertial odometry state is optimized based on the incremental change data.

[0062] In this embodiment, the visual inertial odometry (VIO) system uses an image acquisition device and an IMU (inertial measurement unit) provided on the same device to collect a series of image frames and inertial measurement data between the two image frames while the device is moving. Based on the series of image frames and the inertial measurement data, the system estimates the visual inertial odometry state corresponding to each image frame and the inertial measurement data, as well as the three-dimensional space point coordinates based on the image frames. The system then continuously updates the three-dimensional environment data based on the visual inertial odometry state. In some embodiments, the visual inertial odometry state may include, but is not limited to, a combination of one or more of a camera position state, a camera attitude state, a camera motion state (speed), a gyroscope bias state, an accelerometer bias state, and a three-dimensional space point state.

[0063] Currently, common visual inertial odometry state estimation methods can be divided into filtering-based methods, such as the Extended Kalman Filter (EKF) and Information Filter (IF), as well as graph optimization-based methods, such as the Gauss-Newton method (GN) and the Dog-Leg method (DL). Compared to filtering-based state estimation methods, graph optimization-based methods use more historical information and relinearize the objective function during the iteration process, significantly improving the accuracy of their state estimation. However, the disadvantages they bring are also quite obvious. As more historical information is used, the consumption of computing power also increases.

[0064] In order to solve the problem of excessive computing power consumption caused by the above-mentioned graph optimization method, and at the same time ensure the accuracy of state estimation, the present disclosure implements an incremental visual inertial odometry state estimation scheme. Compared with the traditional graph optimization method, the present disclosure embodiment introduces an incremental state estimation framework, which reduces the scale of re-linearization of the objective function in the graph optimization method, thereby reducing the computing power consumption of the graph optimization method. In addition, the present disclosure embodiment also limits the scale of the state to be optimized in the nonlinear optimization by introducing a sliding window, so that the time complexity is reduced from O(n) to O(1), further reducing the computing power consumption. That is, the present disclosure embodiment iteratively optimizes the visual inertial odometry state based on the marginalization prior constraint, the first visual constraint and the first inertial constraint for the key image frames in the sliding window through an incremental solution method, wherein the marginalization prior constraint is obtained based on the historical key frames eliminated from the sliding window.

[0065] In the disclosed embodiment, the length of the sliding window can be predetermined, and the first preset threshold can be determined based on the length of the sliding window. For example, the first preset threshold can be set to be equal to or less than the length of the sliding window. When the number of key image frames within the sliding window is greater than or equal to the first preset threshold, the key frames to be removed from the sliding window can be removed from the sliding window, and marginalization prior constraints are generated based on the removed key frames to constrain the remaining key frames within the sliding window.

[0066] The image acquisition device integrated in the visual inertial odometry device can continuously acquire image frames of the surrounding environment, and the IMU integrated in the visual inertial odometry device can also continuously acquire inertial measurement data of the visual inertial odometry device. After each image frame is acquired from the image acquisition device, that is, the current image frame, and the inertial measurement data between the previous image frame and the current image frame, an optimization process for the visual inertial odometry state can be performed. In the process of optimizing the visual inertial odometry state, it can be determined whether the current image frame is a key image frame, and when the current image frame is a key image frame, the visual inertial odometry state can be estimated based on the solution proposed in the embodiment of the present disclosure.

[0067] In some embodiments, the key image frame may be an image frame having a large change in image features compared to the previously captured image frame. Specifically, it may be determined by feature point matching or the like. The key image frame may be determined using existing methods, which will not be elaborated here.

[0068] When the current image frame is a key image frame, the key image frame can be added to the set of key image frames defined by the sliding window, and a single estimation of the visual-inertial odometry state can be performed based on the set of key image frames within the sliding window. During the single estimation of the visual-inertial odometry state, a first inertial residual between the current image frame and the previous adjacent image frame can be determined based on the inertial measurement data obtained by the IMU. The first visual constraint and the first inertial constraint obtained in the previous estimation process are then updated based on the first visual residual and the first inertial residual between the current image frame and the key image frames within the sliding window. Finally, an iterative optimization approach can be performed for the key image frames within the sliding window to gradually approximate the final estimation result of the visual-inertial odometry state in this estimation process. It should be noted that all image frames within the sliding window are key image frames to be observed, and non-key image frames are not included in the sliding window. It should also be noted that before adding the current image frame to the sliding window, it is necessary to ensure that the number of key image frames within the sliding window is less than a first preset threshold. This first preset threshold can be a value equal to or less than the length of the sliding window. That is, the number of key image frames in the sliding window cannot be greater than the first preset threshold.

[0069] In some embodiments, the first visual residual may be, for example, a visual reprojection error, including a point projection error and a line projection error, and may be calculated based on a visual residual model. The first inertial residual may be calculated based on an IMU residual model.

[0070] In some embodiments, after determining the first visual residual between the current image frame and all key image frames in the sliding window, and the first inertial residual between the current image frame and the previous adjacent image frame, the first visual residual and the first inertial residual can be used to update the original first visual constraint and the first inertial constraint, that is, the first visual constraint and the first inertial constraint obtained during the last estimation of the visual-inertial odometer state. It should be noted that the first visual constraint and the first inertial constraint before the update are constraints obtained without considering the current image frame. The updated first visual constraint includes the visual residual constraints between all key image frames (including the current image frame) in the sliding window, and the updated first inertial constraint includes the inertial residual constraints between the two adjacent frames before and after all key image frames (including the current image frame) in the sliding window.

[0071] In some embodiments, after the current image frame is added to the set of key image frames defined by the sliding window, the visual inertial odometry states corresponding to all key image frames within the sliding window can be solved through iterative optimization. It should be noted that the three-dimensional spatial point states in the visual inertial odometry state include the position coordinates of the matching feature points between the key image frames in the world coordinate system.

[0072] In some embodiments, during each iteration, the incremental change in the visual-inertial odometry state of the current iteration relative to the previous iteration can be determined based on the marginalized prior constraints, the first visual constraint, and the first inertial constraint. The optimized value of the visual-inertial odometry state for the current iteration is then obtained based on this incremental change. After the iterative optimization is complete, the final estimated visual-inertial odometry state for each key image frame within the sliding window during the current estimation process, as well as the final estimated three-dimensional space point coordinates for the key image frames within the sliding window, can be obtained.

[0073] It should be noted that the incremental change data can be determined by constructing the incremental equation corresponding to the normal equation of the visual-inertial odometry state based on the aforementioned marginalized prior constraints, the first visual constraints, and the first inertial constraints. In traditional graph optimization methods, after establishing the constraint equation based on the marginalized prior constraints, the first visual constraints, and the first inertial constraints, it is difficult to directly solve the constraint equation. Therefore, an iterative method is usually adopted. The initial value of the visual-inertial odometry state to be optimized is given, and it is continuously iterated and updated on this basis, so that the value of the objective function continues to decrease to obtain the final state estimation result. In other words, the solution is achieved by constructing a normal method of nonlinear least squares method.

[0074] Taking the typical Gauss-Newton method as an example, at each iteration, the objective function is relinearized based on the current state estimate. Therefore, each iteration calculates the Jacobian and residual of each factor in the objective function, and constructs the normal equations by summing each factor based on the solved Jacobian and residual. However, this is actually unnecessary because for the objective function corresponding to visual-inertial odometry states whose pose and three-dimensional space point coordinates have not changed or have changed slightly, it is not necessary to relinearize their Jacobian matrix and residual, as these states can be approximately considered unchanged. Therefore, the present embodiment relinearizes the objective function associated with the visual-inertial odometry states that have changed significantly and updates the constructed normal equations. In implementation, the present embodiment first determines the incremental change data of the visual-inertial odometry state based on the marginalized prior constraints, the first visual constraint, and the first inertial constraint, and superimposes this incremental change data on the normal equations from the previous iteration to solve the optimized value of the visual-inertial odometry state in the current iteration. The incremental change data can be understood as data that has undergone incremental changes compared to the results of the previous iteration.

[0075] When iteratively optimizing the visual-inertial odometry state, the disclosed embodiment determines the incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration by marginalizing the prior constraints and the first visual constraints and first inertial constraints updated based on the current image frame during each iteration. The optimized value of the visual-inertial odometry state and the coordinates of the three-dimensional space point in the current iteration are then obtained based on this incremental change data. By combining an incremental solution process with a sliding window marginalization operation, the disclosed embodiment reduces the update scale of the visual-inertial odometry state during the iterative optimization process, improves the efficiency of visual-inertial odometry state estimation, and improves the estimation accuracy of the visual-inertial odometry state by using marginalized prior information.

[0076] In an optional implementation of this embodiment, step S104, i.e., the step of updating the first visual constraint and the first inertial constraint based on the first inertial residual and the first visual residual between the current image frame and other key image frames in the key image frame set, further includes the following steps:

[0077] Calculating the first visual residual between the current image frame and other key image frames in the sliding window;

[0078] A first visual constraint condition and a first inertial constraint condition are updated based on the first visual residual and the first inertial residual, respectively.

[0079] In this optional implementation, after acquiring the current image frame, if the current image frame is a key image frame, a first visual residual can be calculated between the current image frame and each key image frame in the sliding window, and a first inertial residual can be calculated based on the inertial measurement data measured between the previous adjacent image frame and the current image frame. The first visual constraint and the first inertial visual constraint are updated using the first visual residual and the first inertial residual, respectively, so that after the current image frame is added to the sliding window, the visual constraint and inertial constraint of the current image frame can be introduced into the process of visual-inertial odometry state estimation for each key image frame in the sliding window.

[0080] In an optional implementation of this embodiment, step S102, i.e., when the current image frame is a key frame, adding the current image frame to a set of key image frames determined based on a sliding window, further includes the following steps:

[0081] determining the number of key image frames in the set of key image frames defined by the sliding window;

[0082] When the number of the key image frames is greater than or equal to a first preset threshold, updating the marginalization prior constraint condition based on inertial residuals and visual residuals between the key frame to be eliminated and other key image frames in the key image frame set, and eliminating the key frame to be eliminated from the key image frame set defined by the sliding window;

[0083] The current image frame is added to the key image frame set defined by the sliding window.

[0084] In this optional implementation, the first preset threshold can be set as needed. After the number of key image frames added to the sliding window is greater than or equal to the first preset threshold, the key frame to be eliminated can be eliminated from the key image frame set in the sliding window. The key frame to be eliminated can be the first key image frame added to the sliding window, or it can be the key image frame with the smallest difference from the previous and next adjacent key frames. The specific setting can be based on actual needs and is not specifically limited here. After removing the key frame to be eliminated, the visual constraints and inertial constraints associated with the key frame to be eliminated can be converted into constraints on other key image frames in the sliding window, that is, the marginalization prior constraints are updated through the visual residuals and inertial residuals between the key frame to be eliminated and the remaining key image frames in the sliding window, so that in the subsequent estimation process of the visual inertial odometry state, the marginalization prior constraints reflect the visual constraints and inertial constraints of the preset key frame to be eliminated on other key image frames in the sliding window.

[0085] In an optional implementation of this embodiment, in step S105, in each iteration process, based on the marginalization prior constraint, the first visual constraint, and the first inertial constraint, determining the incremental change data of the visual-inertial odometry state in the current iteration process relative to the previous iteration process, and optimizing the visual-inertial odometry state based on the incremental change data further includes the following steps:

[0086] Constructing a current state optimization incremental equation for the visual-inertial odometry state based on the marginalized prior constraint, the first visual constraint, the first inertial constraint, and the previous state optimization incremental equation constructed during the previous iteration of the visual-inertial odometry state; wherein the incremental portion of the current state optimization incremental equation relative to the previous state optimization incremental equation corresponds to an increment of a target state, and the target state includes the visual-inertial odometry state whose change value before and after the optimization of the visual-inertial odometry state during the previous iteration is greater than a second preset threshold;

[0087] The current state optimization incremental equation is solved to obtain an optimized value of the visual inertial odometry state in the current iteration process.

[0088] In this optional implementation, since the visual inertial odometry state is solved incrementally, during the process of estimating the visual inertial odometry state through iterative optimization, in the current iteration, the visual inertial odometry state that has changed significantly compared to the previous iteration can be first determined and determined as the target state. For example, the visual inertial odometry state whose difference between the previous and the next iteration is greater than a second preset threshold can be determined as the target state. The normal equation for the current iteration is constructed based on the normal equation constructed in the previous iteration, that is, the previous state optimization incremental equation, and the increment of the target state. That is, the difference between the normal equation constructed in the current iteration and the normal equation constructed in the previous iteration is the incremental part of the target state, which is equivalent to using the incremental part of the target state to update the normal equation constructed in the previous iteration to obtain the normal equation for the current iteration. Therefore, this normal equation can be called an incremental equation. For ease of distinction, the normal equation constructed in the previous iteration is called the previous state optimization incremental method, and the normal equation constructed in the current iteration is called the current state optimization incremental equation. In the current iteration, the increment of the target state in the current iteration can be obtained by solving the current state optimization incremental equation.

[0089] The following example illustrates the above process.

[0090] First, a constraint equation based on the above-mentioned marginalization prior constraint, the first visual constraint, and the first inertial constraint is established, as shown in the following formula (1):

[0091] The constraint equations are shown below

[0092]

[0093] Where χ=[x0,x1,…,x n ,λ0,λ1,…λ m ] is the visual inertial odometry state to be estimated, is the position and motion state of the kth key image frame in the sliding window, n is the sliding window size, is the position state, is the speed state, is the posture state, b a , b g are the gyroscope bias state and accelerometer bias state, λ l , l∈[0,m] is the inverse depth state of the lth three-dimensional space point, and there are m three-dimensional space points in total. m , H m} represents the marginalization prior constraint, represents the first inertial constraint condition, and B is the set of all inertial measurement data. represents the first visual constraint condition, and C is a set of feature points on at least two key image frames within the sliding window.

[0094] Since it is difficult to directly solve the above formula (1), an iterative optimization method is usually used. Given the initial value of the visual inertial odometry state to be optimized, it is continuously iterated and optimized on this basis, so that the value of the objective function continues to decrease to obtain the final state estimation result. In the typical Gauss-Newton method, each iteration of the objective function will be in the current state estimation result χ - Relinearization is performed based on , which can be expressed as follows:

[0095]

[0096]

[0097] Among them, r k is the objective function, χ - The current state estimation result is the Jacobian matrix, e k =r k (χ - ) is the residual term, represents the addition operation on the manifold. The increment δχ can be solved by the following formula:

[0098] Aδχ=b

[0099] [A|b]=Σ[A k |bk ]

[0100] [A k |b k ]=[J k T J k |-J k e k ] (4)

[0101] If the Jacobian and residual of each factor in the objective function are calculated for each iteration, and each factor is accumulated to construct the normal equation, a large amount of calculation will be generated. In fact, this is unnecessary, because for the objective function corresponding to the visual inertial odometry state whose pose and coordinates of the three-dimensional space points have not changed or have changed slightly, there is no need to re-linearize its Jacobian matrix and residual, because these states can be approximately considered to have not changed. Therefore, in the embodiment of the present disclosure, only the objective function associated with the state that has changed or changed significantly is re-linearized, and the normal equation is updated to obtain the incremental equation that can be expressed as follows:

[0102]

[0103] Where [A|b] + represents the incremental equation, that is, the current state optimization incremental equation in the current iteration process, [A|b] - Represents the normal equation in the previous iteration process, that is, the previous state optimization incremental equation, represents the incremental part, which corresponds to the incremental change of the target state, represents the set of objective functions that need to be relinearized, that is, the set of objective functions involving at least one target state.

[0104] By solving the above optimization increment equation for the current state, we can obtain the optimized value of the visual inertial odometry state in the current iteration. Through the above method, after multiple iterations of optimization, we can obtain the final estimated result of the visual inertial odometry state in this estimation process.

[0105] In an optional implementation of this embodiment, the step of solving the current state optimization increment equation to obtain the optimized value of the visual inertial odometry state in the current iteration process further includes the following steps:

[0106] Constructing a Schur complement equation for eliminating the three-dimensional space point state based on the current state optimization incremental equation;

[0107] Solving the Schur complement equation to obtain an increment of the camera pose state and an increment of the motion state in the visual inertial odometry state;

[0108] The increment of the three-dimensional space point is solved based on the increment of the camera posture state and the increment of the motion state.

[0109] In this optional implementation, in order to efficiently solve the increment of the visual inertial odometry state in the above-mentioned previous state optimization increment equation, the three-dimensional space point state can be first eliminated from the previous state optimization increment equation through the Schur elimination method, and the increment of the posture state (including position state and attitude state) and the motion state can be solved, and the increment of the posture state and the motion state can be solved by back substitution to obtain the increment of the three-dimensional space point state.

[0110] For example, the increment of the visual inertial odometry state can be expressed as follows:

[0111] δχ=[δχ c , δχ p ] (6)

[0112] Among them, δχ is the total increment of the visual inertial odometry state, δχ c Indicates the increment of the posture state and motion state in the visual inertial odometry state, δχ p Represents the increment of the 3D space point state in the visual inertial odometry state. According to the above formula (6), we can get:

[0113]

[0114] Eliminating the second line in the above formula (7) can obtain c The relevant Schur complement equation is expressed as follows:

[0115] Sδχ c =s

[0116] [S|s]=[U-WV -1 W T |uW / V -1 v] (8)

[0117] Where S represents the Schur complement matrix. As can be seen from the construction steps of the above Schur complement equation, changes in the equation are only related to the visual constraints. Therefore, the Schur complement equation only needs to be partially updated when the objective function corresponding to the 3D point state needs to be relinearized.

[0118] The update of the parameter block associated with the pose state (i1, i2) in S and the pose state i in s can be expressed as follows:

[0119]

[0120] Among them, P represents the set of three-dimensional space point states involved in the objective function that needs to be relinearized, Represents the set of all 3D space points tracked by the i-th key image frame.

[0121] By solving the above formula (9), the increment of the camera pose state and the increment of the camera motion state in the visual inertial odometry state can be obtained. Then, according to the increment of the camera pose state and the increment of the camera motion state, the increment of the three-dimensional space point state can be obtained. The solution process of the increment of the three-dimensional space point state is as follows:

[0122] δχ c =S -1 *s

[0123] δχ p =V -1 *(vW T δχ c ) (10)

[0124] In an optional implementation of this embodiment, when the number of key image frames is greater than or equal to a first preset threshold, the step of updating the marginalization prior constraint condition based on the inertial residual and the visual residual between the key frame to be eliminated and other key image frames in the key image frame set, and eliminating the key frame to be eliminated from the sliding window further includes the following steps:

[0125] Constructing an intermediate incremental equation based on the second inertial constraint, the second visual constraint, and the previous condition update incremental equation; wherein the previous condition update incremental equation is the incremental equation constructed during the previous update of the marginalization prior constraint; the second inertial constraint and the second visual constraint are the constraints associated with the preset key frame in the first inertial constraint and the first visual constraint, respectively;

[0126] Removing the visual-inertial odometry state of the preset key frame from the intermediate incremental equation by a Schur complement operation to obtain a current conditional update incremental equation corresponding to the marginalized prior constraint condition;

[0127] The marginalization prior constraint condition is updated by using the current condition to update the incremental equation.

[0128] In this optional implementation, since the key frame to be eliminated is reflected as a marginalization prior constraint in the next optimization process, and the first visual constraint and the first inertia constraint only reflect the visual constraints and inertia constraints between the key image frames in the sliding window, the second visual constraint and the second inertia constraint associated with the key frame to be eliminated can be removed from the first visual constraint and the first inertia constraint respectively, and then the second visual constraint and the second inertia constraint can be used to update the marginalization prior constraint.

[0129] Using the second visual constraint and the second inertial constraint to update the marginalization prior constraint can be understood as converting the second visual constraint and the second inertial constraint into constraints on other key image frames in the sliding window.

[0130] In some embodiments, the second visual constraint and the second inertial constraint are converted into constraints on other key image frames in the sliding window. This can be viewed as solving the problem of maximum a posteriori probability distribution of other key image frames in the sliding window when the key frame to be eliminated is known. The maximum a posteriori probability distribution is expressed as follows:

[0131]

[0132] Since the embodiment of the present disclosure adopts an incremental update method, the incremental equation corresponding to the above maximum a posteriori probability distribution can be expressed as follows:

[0133]

[0134] Among them, [A m |b m ] is the intermediate incremental equation constructed based on the previous conditional update incremental equation obtained in the previous update process based on the second inertial constraint, the second visual constraint and the marginalization prior constraint. last-marg |b last-marg ] represents the incremental equation before the update, that is, the incremental equation of the previous conditional update obtained by marginalizing the prior constraints in the previous update, [A I |b I ] represents the incremental equation constructed based on the inertia constraint between the key frame to be eliminated and its adjacent key frames, represents the incremental equation constructed based on the visual constraints between the key frame to be eliminated and other key image frames in the sliding window, is the set of all visual constraints between the keyframe to be removed and all other keyframes in the sliding window.

[0135] After obtaining the above intermediate incremental equation, the state of the key frame to be eliminated is marginalized, that is, the camera pose and motion state corresponding to the key frame to be eliminated, as well as the three-dimensional space point state corresponding to the key frame to be eliminated are removed from the above intermediate incremental equation. The process of removing the camera pose and motion state corresponding to the key frame to be eliminated, as well as the three-dimensional space point state corresponding to the key frame to be eliminated can be performed using the Schur complement operation shown in the following formula:

[0136]

[0137] Among them, the subscript m represents the visual inertial odometry state of the key frame to be eliminated, and the subscript s represents the visual inertial odometry state of other key image frames in the sliding window. m |b m ] is the incremental equation corresponding to the marginalization constraint, that is, the incremental equation for the current condition update corresponding to the updated marginalization prior constraint condition. Through the marginalization operation, the disclosed embodiment can transform the constraints associated with the key frames to be eliminated in the sliding window into constraints on each other key image frame in the sliding window, thereby ensuring the observability of each key image frame in the sliding window.

[0138] In an optional implementation of this embodiment, the method further includes the following steps:

[0139] When the current image frame is a non-key frame, determining at least one matching key frame in the key image frame set that matches the current image frame, and three-dimensional space point coordinates corresponding to matching feature points between the current image frame and the at least one matching key frame;

[0140] calculating a second visual residual between the current image frame and the at least one matching key frame, and calculating a second inertial residual based on inertial measurement data between the current image frame and a previous adjacent image frame;

[0141] constructing a non-keyframe visual constraint condition based on the second visual residual, and constructing a non-keyframe inertial constraint condition based on the second inertial residual;

[0142] A target value of the video inertial odometry state corresponding to the current image frame is obtained based on the non-keyframe visual constraint condition, the non-keyframe inertial constraint condition, the known value of the video inertial odometry state corresponding to the at least one matching keyframe, and the three-dimensional space point coordinates.

[0143] In this optional implementation, non-key frames are processed differently from key frames. Non-key frames do not need to be included in the sliding window. Instead, they are constrained based on key frames that match them within the sliding window, and the target value of the visual-inertial odometry state corresponding to the non-key frames is calculated.

[0144] In some embodiments, after obtaining a non-key image frame, the non-key image frame is matched with a key image frame in a sliding window to obtain one or more matching key frames and the coordinates of the three-dimensional space points corresponding to the matching feature points between these matching key frames. A non-key frame visual constraint condition is constructed based on the visual residual (such as visual reprojection error) between the non-key image frame and the matching key frame, and an inertial residual is determined based on the inertial measurement data measured between the previous adjacent image frame, and a non-key frame inertial constraint condition is constructed based on the inertial residual. A constraint equation is established based on the non-key frame visual constraint condition and the non-key frame inertial constraint condition, and the objective function of the constraint equation is constructed using the nonlinear least squares method, and the normal equation corresponding to the objective function is constructed by the graph optimization method. Then, the Schur complement equation is constructed based on the constructed normal equation, that is, when the matching key frame and the three-dimensional space point coordinates are known, the maximum posterior probability estimate of the posture state and motion state corresponding to the non-key frame, that is, the current image frame, is solved, as shown below:

[0145]

[0146] Among them, χ nk Indicates the pose and motion status of non-keyframes.

[0147] Figure 2 FIG. 1 is a schematic diagram showing a processing flow of an input image frame in the state estimation process of a visual inertial odometry according to an embodiment of the present disclosure. Figure 2 As shown, the input image frame includes a key image frame and a non-key image frame. After receiving the input image frame, a matching key frame that matches the input image frame and matching feature points between the input image frame and the matching image frame are searched within the sliding window.

[0148] When the input image is a key image frame, the input image frame is added to the sliding window and an iterative optimization process is performed. In this iterative optimization process, an updated normal equation (i.e., the current state optimization incremental equation mentioned above) is constructed by an incremental solution method in each iteration. The updated normal equation includes the original normal equation (i.e., the previous state optimization incremental equation mentioned above, which is the normal equation constructed in the previous iteration) and the incremental part in the current iteration. The updated normal equation is solved by the Schur complement operation. Since it is an incremental solution process, the updated Schur complement equation is also constructed by updating the original Schur complement equation in the Schur complement operation. The updated Schur complement equation includes the original Schur complement equation (i.e., the Schur complement equation in the previous iteration) and the incremental part. The posture state and motion state are solved by first eliminating the three-dimensional space point state from the updated Schur complement equation by the elimination method, and then the three-dimensional space point state is solved based on the solved posture state and motion state.

[0149] When the number of key image frames in the sliding window reaches a certain threshold, the key image frame that first enters the sliding window is removed, and the visual constraints and inertia constraints associated with the removed key image frame are removed from the visual constraints and inertia constraints in the updated normal equation and updated Schur complement equation. The visual constraints and inertia constraints of the removed key image frame are used as constraints for other key image frames in the sliding window, that is, the visual constraints and inertia constraints of the key image frame to be removed are added to the marginalization prior constraints. This addition process can be referred to the description process of formulas (11)-(13) above and will not be repeated here.

[0150] Figure 3 FIG. 1 shows a schematic diagram of a framework for a mobile phone positioning and tracking application scenario according to an embodiment of the present disclosure. Figure 3 As shown, the mobile phone is equipped with a camera and an inertial measurement unit (IMU). The mobile phone is also integrated with a visual inertial odometry system that executes the image processing method proposed in the embodiment of the present disclosure. The visual inertial odometry system can obtain the two-dimensional images captured by the camera and the inertial measurement data measured by the IMU in real time; the camera captures images at a fixed frame rate. When the mobile phone is in motion in indoor scenes such as offices, homes, exhibition halls, etc., the camera on the mobile phone captures the surrounding environment and provides the captured images to the visual inertial odometry system. The inertial measurement unit continuously measures the three-axis rotation angular velocity of the mobile phone relative to its own coordinate system and the accelerometer of the mobile phone relative to its own coordinate system, and provides the three-axis rotation angular velocity and accelerometer to the visual inertial odometry system.

[0151] The visual inertial odometry system is in an uninitialized state at startup. It initializes after receiving the camera's image sequence and the data collected by the IMU (gyroscope and accelerometer data). During initialization, it first creates a 3D spatial map of the surrounding environment and determines the system parameters.

[0152] Since the environment is unknown to the positioning system, the system needs to build a three-dimensional map of the surrounding environment. A typical three-dimensional map consists of several three-dimensional points with determined positions in space. The system can extract feature points from the images based on the obtained image sequence, match them pairwise, and then determine whether the parallax of the matched feature points in different images is large enough. When the parallax of the same three-dimensional point in the two images is large enough, the three-dimensional position of this feature point can be restored by triangulation method based on the relative position of the mobile phone in the camera coordinate system at two moments. Multiple feature points with known three-dimensional positions can be used as the initial spatial three-dimensional map. The position state and attitude state of the mobile phone output by the subsequent system are relative to the three-dimensional map coordinate system. The position of the mobile phone can include the spatial position coordinates of the mobile phone, and the attitude of the mobile phone can include the yaw angle θ, pitch angle ψ and roll angle φ of the mobile phone.

[0153] In addition to initializing the three-dimensional space map, it is also necessary to calculate other necessary parameters of the system, such as the transformation relationship between the camera coordinate system and the IMU coordinate system, so as to complete the unification of the camera coordinate system and the IMU coordinate system.

[0154] After initialization, the system receives the current image frame captured by the camera and the inertial measurement data between the current image frame and the previous adjacent image frame measured by the IMU. It then uses the current image frame, inertial measurement data, key image frames within the sliding window, and data generated during the previous iterative optimization process to estimate the phone's position and motion state, as well as the state of 3D points in the 3D map. The optimization process can be found in the optimization process for the visual inertial odometry state in the image processing method described above in the embodiments of this disclosure and will not be repeated here.

[0155] As the user moves their phone, the surrounding environment changes, and the camera constantly sees new objects. To ensure that the correspondence between feature points in the camera image and map points is always established, the 3D map created during initialization needs to expand as the phone moves. Therefore, the system can use the visual inertial odometry state obtained during the optimization process to update the 3D spatial map. By establishing a matching relationship between feature points in the current frame and the previous image frame, for matching point pairs that do not have corresponding 3D points, their 3D positions in space can be calculated through triangulation and added to the map.

[0156] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0157] According to an embodiment of the present disclosure, a state estimation device for a visual inertial odometry system can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The state estimation device for a visual inertial odometry system includes:

[0158] a first acquisition module configured to acquire a current image frame captured by an image acquisition device integrated with the visual inertial odometry and inertial measurement data output by an inertial measurement unit integrated with the visual inertial odometry;

[0159] a first determining module configured to determine a first inertial residual between the current image frame and the previous adjacent image frame based on the inertial measurement data between the current image frame and the previous adjacent image frame;

[0160] an adding module configured to, when the current image frame is a key frame, add the current image frame to a set of key image frames determined based on a sliding window;

[0161] an updating module configured to update a first inertial constraint and a first visual constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set;

[0162] An estimation module is configured to estimate, for the key image frames within the sliding window, the visual-inertial odometry state corresponding to the key image frames through iterative optimization; wherein, in each iteration, based on the marginalization prior constraint, the first visual constraint, and the first inertial constraint, the incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration is determined, and the visual-inertial odometry state is optimized based on the incremental change data.

[0163] In an optional implementation of this embodiment, the update module includes:

[0164] a first calculation submodule, configured to calculate the first visual residual between the current image frame and other key image frames in the sliding window;

[0165] The first updating submodule is configured to update a first visual constraint condition and a first inertial constraint condition based on the first visual residual and the first inertial residual, respectively.

[0166] In an optional implementation of this embodiment, the first determining module includes:

[0167] A first determining submodule is configured to determine the number of key image frames in a key image frame set defined by the sliding window;

[0168] a second updating submodule configured to, when the number of the key image frames is greater than or equal to a first preset threshold, update the marginalization prior constraint based on inertial residuals and visual residuals between the key frame to be eliminated and other key image frames in the key image frame set, and eliminate the key frame to be eliminated from the key image frame set defined by the sliding window;

[0169] The adding submodule is configured to add the current image frame to the key image frame set defined by the sliding window.

[0170] In an optional implementation of this embodiment, the estimation module includes:

[0171] a first construction submodule configured to construct a current state optimization incremental equation of the visual-inertial odometry state based on a marginalized prior constraint, the first visual constraint, the first inertial constraint, and a previous state optimization incremental equation constructed during a previous iteration of the visual-inertial odometry state; wherein an incremental portion of the current state optimization incremental equation relative to the previous state optimization incremental equation corresponds to an increment of a target state, and the target state includes the visual-inertial odometry state whose change value before and after the optimization of the visual-inertial odometry state during the previous iteration is greater than a second preset threshold;

[0172] The first solving submodule is configured to solve the current state optimization incremental equation to obtain an optimized value of the visual inertial odometry state in the current iteration process.

[0173] In an optional implementation of this embodiment, the first solution submodule includes:

[0174] A second construction submodule is configured to construct a Schur complement equation for eliminating the three-dimensional space point state based on the current state optimization incremental equation;

[0175] A second solving submodule is configured to solve the Schur complement equation to obtain an increment of a camera pose state and an increment of a motion state in the visual inertial odometry state;

[0176] The third solving submodule is configured to solve the increment of the three-dimensional space point based on the increment of the camera posture state and the increment of the motion state.

[0177] In an optional implementation of this embodiment, the second updating submodule includes:

[0178] a third construction submodule configured to construct an intermediate incremental equation based on a second inertial constraint, a second visual constraint, and a previous condition update incremental equation; wherein the previous condition update incremental equation is an incremental equation constructed during a previous update of the marginalization prior constraint; and the second inertial constraint and the second visual constraint are, respectively, constraints associated with the preset key frame in the first inertial constraint and the first visual constraint.

[0179] an acquisition submodule configured to remove the visual inertial odometry state of the preset key frame from the intermediate incremental equation through a Schur complement operation to obtain a current conditional update incremental equation corresponding to the marginalized prior constraint condition;

[0180] The third updating submodule is configured to update the marginalization prior constraint condition by using the current condition to update the incremental equation.

[0181] In an optional implementation of this embodiment, the apparatus further includes:

[0182] a second determining module configured to, when the current image frame is a non-key frame, determine at least one matching key frame in the key image frame set that matches the current image frame, and three-dimensional space point coordinates corresponding to matching feature points between the current image frame and the at least one matching key frame;

[0183] a calculation module configured to calculate a second visual residual between the current image frame and the at least one matching key frame, and calculate a second inertial residual based on inertial measurement data between the current image frame and its previous adjacent image frame;

[0184] a construction module configured to construct a non-keyframe visual constraint condition based on the second visual residual, and to construct a non-keyframe inertial constraint condition based on the second inertial residual;

[0185] The second acquisition module is configured to obtain the target value of the video inertial odometry state corresponding to the current image frame based on the non-key frame visual constraint condition, the non-key frame inertial constraint condition, the known value of the video inertial odometry state corresponding to the at least one matching key frame, and the three-dimensional space point coordinates.

[0186] The image processing device in this embodiment corresponds to the above-mentioned image processing method. For specific details, please refer to the above-mentioned description of the image processing method, which will not be repeated here.

[0187] Figure 4 3 is a schematic structural diagram of an electronic device suitable for implementing the method for estimating the state of a visual inertial odometry according to an embodiment of the present disclosure.

[0188] like Figure 4 As shown, the electronic device 400 includes a processing unit 401, which can be implemented as a processing unit such as a CPU, a GPU, an FPGA, an NPU, etc. The processing unit 401 can perform various processes in the embodiment of any of the above methods of the present disclosure according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0189] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 408 including a hard disk; and a communication section 409 including a network interface card such as a LAN card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed into the storage section 408 as needed.

[0190] In particular, according to embodiments of the present disclosure, any of the methods described above with reference to the embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for executing any of the methods described in the embodiments of the present disclosure. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 409 and / or installed from the removable medium 411.

[0191] In some embodiments, the electronic device 400 further includes a visual inertial odometry (not shown in the figure), which integrates an image acquisition device and an inertial measurement unit. During the state estimation process of the visual inertial odometry, the image acquisition device can continuously acquire image data in the environment, and the inertial measurement unit can continuously measure and obtain inertial measurement data. The above-mentioned image data and inertial measurement data can be output to a processor, and the processor executes the state estimation method of the above-mentioned visual inertial odometry proposed in the embodiment of the present disclosure based on the image data and inertial measurement data.

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the diagram or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0193] The units or modules involved in the embodiments described in this disclosure may be implemented in software or hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, constitute limitations on the units or modules themselves.

[0194] As another aspect, the present disclosure also proposes a method for implementing augmented reality, which is implemented on the above-mentioned electronic device, and determines the position and posture of the electronic device and the three-dimensional space around the electronic device based on the state estimation method of the above-mentioned visual inertial odometer proposed in the embodiment of the present disclosure. The electronic device can be a robot, a drone, an autonomous driving vehicle, etc. After obtaining the final estimation result of the visual inertial odometer state using the state estimation method of the above-mentioned visual inertial odometer proposed in the embodiment of the present disclosure, the position and posture of the electronic device and the surrounding three-dimensional space data can be determined based on the final estimation result of the visual inertial odometer state. The visual inertial odometer state includes the position and posture data of the image acquisition device integrated on the visual inertial odometer and the state of the three-dimensional space point. The position and posture of the electronic device can be determined based on the position and posture data of the image acquisition device and the relative position and posture change between the image acquisition device and the electronic device, and the three-dimensional space around the electronic device can be determined based on the state of the above-mentioned three-dimensional space point. The position and posture of the electronic device and the surrounding three-dimensional space data can be used to implement the augmented reality method.

[0195] As another aspect, the present disclosure further provides a computer-readable storage medium. This computer-readable storage medium may be included in the apparatus described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the apparatus. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods described in the present disclosure.

[0196] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

Claims

1. A state estimation method for a visual inertial odometry, wherein: include: Obtain the current image frame captured by the image acquisition device integrated with the visual inertial odometry, as well as the inertial measurement data output by the inertial measurement unit integrated with the visual inertial odometry; When the current image frame is a key frame, adding the current image frame to a key image frame set determined based on a sliding window; determining a first inertial residual between the current image frame and the previous adjacent image frame based on the inertial measurement data between the current image frame and the previous adjacent image frame; updating a first inertial constraint and a first visual constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set; For the key image frames within the sliding window, the visual inertial odometry state corresponding to the key image frames is estimated through iterative optimization; wherein, in each iterative process, based on the marginalization prior constraint condition, the first visual constraint condition, and the first inertial constraint condition, the incremental change data of the visual inertial odometry state in the current iterative process relative to the previous iterative process is determined, and the visual inertial odometry state is optimized based on the incremental change data.

2. The method according to claim 1, wherein Updating a first visual constraint and a first inertial constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set includes: Calculating the first visual residual between the current image frame and other key image frames in the sliding window; A first visual constraint condition and a first inertial constraint condition are updated based on the first visual residual and the first inertial residual, respectively.

3. The method according to claim 1 or 2, wherein When the current image frame is a key frame, adding the current image frame to a key image frame set determined based on a sliding window includes: determining the number of key image frames in the set of key image frames defined by the sliding window; When the number of the key image frames is greater than or equal to a first preset threshold, updating the marginalization prior constraint condition based on inertial residuals and visual residuals between the key frame to be eliminated and other key image frames in the key image frame set, and eliminating the key frame to be eliminated from the key image frame set defined by the sliding window; The current image frame is added to the key image frame set defined by the sliding window.

4. The method according to claim 1 or 2, wherein: In each iteration, based on the marginalized prior constraint, the first visual constraint, and the first inertial constraint, determining incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration, and optimizing the visual-inertial odometry state based on the incremental change data, including: Constructing a current state optimization incremental equation for the visual-inertial odometry state based on the marginalized prior constraint, the first visual constraint, the first inertial constraint, and the previous state optimization incremental equation constructed during the previous iteration of the visual-inertial odometry state; wherein the incremental portion of the current state optimization incremental equation relative to the previous state optimization incremental equation corresponds to an increment of a target state, and the target state includes the visual-inertial odometry state whose change value before and after the optimization of the visual-inertial odometry state during the previous iteration is greater than a second preset threshold; The current state optimization incremental equation is solved to obtain an optimized value of the visual inertial odometry state in the current iteration process.

5. The method according to claim 4, wherein Solving the current state optimization incremental equation to obtain the optimized value of the visual inertial odometry state in the current iteration process includes: Constructing a Schur complement equation for eliminating the three-dimensional space point state based on the current state optimization incremental equation; Solving the Schur complement equation to obtain an increment of the camera pose state and an increment of the motion state in the visual inertial odometry state; The increment of the three-dimensional space point is solved based on the increment of the camera posture state and the increment of the motion state.

6. The method according to claim 3, wherein: When the number of the key image frames is greater than or equal to a first preset threshold, updating the marginalization prior constraint condition based on the inertial residual and the visual residual between the key frame to be eliminated and other key image frames in the key image frame set, and eliminating the key frame to be eliminated from the sliding window, comprising: Constructing an intermediate incremental equation based on the second inertial constraint, the second visual constraint, and the previous condition update incremental equation; wherein the previous condition update incremental equation is the incremental equation constructed in the process of previously updating the marginalization prior constraint; the second inertial constraint and the second visual constraint are the constraints associated with the key frame to be eliminated in the first inertial constraint and the first visual constraint, respectively; Removing the visual-inertial odometry state of the key frame to be eliminated from the intermediate incremental equation by a Schur complement operation to obtain a current conditional update incremental equation corresponding to the marginalized prior constraint condition; The marginalization prior constraint condition is updated by using the current condition to update the incremental equation.

7. The method according to any one of claims 1-2, 5-6, wherein The method further comprises: When the current image frame is a non-key frame, determining at least one matching key frame in the key image frame set that matches the current image frame, and three-dimensional space point coordinates corresponding to matching feature points between the current image frame and the at least one matching key frame; calculating a second visual residual between the current image frame and the at least one matching key frame, and calculating a second inertial residual based on inertial measurement data between the current image frame and a previous adjacent image frame; constructing a non-keyframe visual constraint condition based on the second visual residual, and constructing a non-keyframe inertial constraint condition based on the second inertial residual; A target value of the visual-inertial odometry state corresponding to the current image frame is obtained based on the non-keyframe visual constraint condition, the non-keyframe inertial constraint condition, the known value of the visual-inertial odometry state corresponding to the at least one matching keyframe, and the three-dimensional space point coordinates.

8. A state estimation device for a visual inertial odometry, wherein: include: a first acquisition module configured to acquire a current image frame captured by an image acquisition device integrated with the visual inertial odometry and inertial measurement data output by an inertial measurement unit integrated with the visual inertial odometry; a first determining module configured to determine a first inertial residual between the current image frame and the previous adjacent image frame based on the inertial measurement data between the current image frame and the previous adjacent image frame; an adding module configured to add the current image frame to a set of key image frames determined based on a sliding window when the current image frame is a key frame; an updating module configured to update a first inertial constraint and a first visual constraint based on the first inertial residual and a first visual residual between the current image frame and other key image frames in the key image frame set; An estimation module is configured to estimate, for the key image frames within the sliding window, the visual-inertial odometry state corresponding to the key image frames through iterative optimization; wherein, in each iteration, based on the marginalization prior constraint, the first visual constraint, and the first inertial constraint, the incremental change data of the visual-inertial odometry state in the current iteration relative to the previous iteration is determined, and the visual-inertial odometry state is optimized based on the incremental change data.

9. An electronic device, wherein: It includes a visual inertial odometry, a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein, based on the image data and inertial measurement data output by the visual inertial odometry, the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

10. A method for implementing augmented reality on the electronic device according to claim 9, wherein: The posture of the electronic device and the three-dimensional space around the electronic device are determined based on the method described in any one of claims 1 to 7.

Images