Event camera / imu / natural landmark based odometry system and method

Abstract

The application provides an event camera / IMU / natural landmark-based odometer system and method, wherein the system comprises an event camera tracking module, a traditional camera tracking module and a joint optimization module. The application utilizes the complementary advantages of the event camera and the traditional camera, adopts a scheme combining the two with an IMU to construct a visual odometer, improves the robustness of the system in a high-speed high-dynamic scene, and calculates and updates the current pose through a scheme of presetting natural landmarks with known position information in the scene, so as to realize a function similar to loop detection. In the traditional camera tracking module, a step of dynamic point elimination is further added, so as to further improve the robustness of the system in a dynamic scene.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology, and relates to simultaneous localization and mapping technology, specifically to an odometry system and method based on event camera / IMU / natural landmarks. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) technology has important applications in various fields such as autonomous robot control and augmented / virtual reality. Visual inertial odometry (VIO) has made significant progress in recent years due to the excellent complementary performance of cameras and inertial sensors. However, due to some well-known limitations of traditional cameras (motion blur and low dynamic range), these VIOMA methods still struggle with certain situations, such as high-speed motion or high dynamic range scenes. The emergence of event cameras offers great potential to overcome these problems. Unlike traditional cameras that output pixel intensity at a fixed frame rate, event cameras (such as Dynamic Visual Sensors (DVS)) only output changes in pixel intensity. Compared to traditional cameras, event cameras have significant advantages in several aspects: microsecond-level latency and very high dynamic range (generally more than twice that of traditional cameras). Especially since all pixels independently acquire light information, event cameras are not affected by motion blur. Research on state estimation using event cameras has made some progress, with many schemes proposed using pure event cameras and combining event cameras with IMU elements for pose estimation. However, most of these schemes lack loop closure detection modules, making it difficult to guarantee the long-term performance of the system. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, and considering the complementary performance of event cameras and traditional cameras, as well as the excellent robustness and accuracy of visual inertial odometry, this invention proposes a new odometry system and method based on the combination of event camera / IMU / natural landmarks. By deploying preset natural landmarks with known location information, the system uses a target detection algorithm to identify the landmarks and calculate and update their poses to achieve a function similar to loop closure detection, thereby improving the system's performance.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] The odometry implementation method based on event camera / IMU / natural landmarks includes the following steps:

[0006] Step 1, Event Camera Tracking

[0007] Based on the event stream output by the event camera, feature points are extracted directly from the original events using SAE (Surface of Active Events). The extracted feature points are then tracked using the LK optical flow method, and their inverse depth is calculated to generate map points. The map points are then further tracked.

[0008] Step 2, Traditional Camera Tracking

[0009] While extracting and tracking feature points on standard image frames, the YOLOv5 target detection model is added to identify preset natural landmarks. When a preset natural landmark is detected, the PnP algorithm is used to calculate the pose and update the pose of the current frame.

[0010] Step 3, Joint Optimization

[0011] Visual-inertial fusion is achieved through nonlinear optimization, transforming the visual-inertial fusion problem into a minimization problem of an objective function. This objective function consists of four parts: the prior error after marginalization, the reprojection error between the event camera and the conventional camera, and the error term of inertial measurement. The accurate pose estimate is obtained by solving the problem using the G2O optimization library.

[0012] Furthermore, the events are represented using polarized TS (Time Surface): Where T p Let x and y represent the intensity of the event, t represent the time the event occurred, p represent the polarity of the pixel intensity change, S represent the timestamp of the last time the event was triggered on this pixel, and τ represent a constant decay rate.

[0013] Furthermore, step one specifically includes the following sub-steps:

[0014] Step 1: Generate an SAE based on the event stream provided by the event camera. The SAE records the timestamp of the last triggered event for each pixel in the event stream. It is a two-dimensional array, where each element corresponds to a pixel and its value corresponds to the timestamp of the last triggered event.

[0015] Step 2: When a new event stream is received, the Arc* algorithm is used to extract corner points on the generated SAE. The Arc* algorithm maintains two circular event sets and detects whether continuous arcs or their complementary arcs are within a certain range to extract corner points.

[0016] Step 3: Use the LK optical flow method to track the extracted feature points. If the tracking is successful, triangulate the points to generate map points and perform further tracking. If the tracking fails, discard the feature point. When the number of feature points is lower than the set threshold, extract new feature points.

[0017] Furthermore, the specific steps for extracting feature points using the Arc* algorithm in step 2 are as follows:

[0018] (1) Define two circular event sets, each containing the N most recent events;

[0019] (2) When a new event arrives, add it to the two ring event sets, calculate the radian between the event and other events in the ring, and record the range of the continuous arc or its supplementary arc;

[0020] (3) For each newly added event, check whether its corresponding continuous arc or supplementary arc is within a certain range. If so, mark the event as a corner point.

[0021] (4) Remove the oldest event from the two circular event sets to ensure that the set contains only the most recent N events.

[0022] Furthermore, in step 3, key frames are extracted during the tracking process, and the extracted key frames are sent to the joint optimization module for optimization calculation. The selection of key frames is based on the number of currently tracked features and the average disparity between the tracked features and two consecutive time stamps. A threshold is set for each. When the number of tracked features is lower than the set threshold or the average disparity between the tracked feature points and two consecutive time stamps is greater than the set threshold, a new key frame is selected.

[0023] Furthermore, step two specifically includes the following sub-steps:

[0024] Step 1: Pre-collect image data to train the YOLOv5 model.

[0025] Step 2: Perform target detection on standard image frames from a traditional camera to identify whether landmarks exist in the image.

[0026] Step 3: Extract feature points from standard image frames, identify potential dynamic objects based on the target detection results, and use static consistency to divide the feature points in the potential dynamic object region into static feature points and dynamic feature points, and remove the dynamic feature points.

[0027] Step 4: Use optical flow to track the extracted static feature points.

[0028] Step 5: If a landmark was identified in Step 2, calculate the pose using the PnP algorithm and update the camera pose at this time.

[0029] Furthermore, the specific steps for removing dynamic feature points in step 3 are as follows:

[0030] (1) Use the latest YOLOv5 algorithm to process each frame of the image and identify all objects in it;

[0031] (2) Save the rectangular region where the potential dynamic objects in the recognition results are located. The format of the saved region is (x1, y1, h, w), where x1 is the horizontal coordinate of the upper left corner of the region, y1 is the vertical coordinate of the upper left corner of the region, h is the height of the rectangular region, and w is the width of the rectangular region.

[0032] (3) By using static consistency, feature points in potential moving object regions are divided into static feature points and dynamic feature points, and finally dynamic feature points are eliminated; specifically, the following process is used to distinguish between static and dynamic feature points:

[0033] For feature points i,j in the image frame at time t and their corresponding feature points i,j in the image frame at time t+1, use This means that the distance between two feature points in an image frame at the same time is defined as:

[0034]

[0035] Where u and v are the x and y coordinates of the feature point in the pixel coordinate system; when the distance between corresponding feature points in image frames at different times is the same or the distance difference is within a certain threshold range, the two feature points are considered static feature points. The threshold is determined by feature points within a non-potentially moving object region, and its calculation formula is as follows:

[0036]

[0037] Where S is the set of feature points within the region of the non-potentially moving object, and d t (i,j) represents the distance between any two feature points in set S at time t, and N represents the number of feature points in set S.

[0038] Furthermore, the specific steps for updating the pose estimation by identifying artificial landmarks in steps 2 and 5 are as follows:

[0039] (1) Collect image data of pre-arranged natural landmarks in the scene from different angles;

[0040] (2) Mark the targets on the collected image data and mark the parts of the preset natural landmarks;

[0041] (3) Train the model using the labeled dataset;

[0042] (4) Use the trained model to identify landmarks in the image. After successful identification, obtain the 3D coordinates of the landmark in the world coordinate system and the 2D coordinates in the image. Use the PnP algorithm to solve for the pose estimation and update the pose in real time.

[0043] Furthermore, the objective function in step three is:

[0044]

[0045] Where, χ on the left side of the equation is the variable to be estimated, the first term on the right side is the prior residual remaining after marginalization, the second term is the residual of IMU pre-integration, the third term is the measurement residual of the event camera, and the fourth term is the measurement residual of the conventional camera; W (·) The weights are the values ​​corresponding to each residual.

[0046] This invention also provides a visual odometry system based on an event camera / traditional camera / IMU / natural landmarks, used to implement the aforementioned visual odometry implementation method based on an event camera / traditional camera / IMU / natural landmarks, including an event camera tracking module, a traditional camera tracking module, and a joint optimization module; wherein,

[0047] The event camera tracking module is used to extract feature points directly from the original events based on the event stream output by the event camera and SAE (Surface of Active Events). The extracted feature points are tracked using the LK optical flow method, and their inverse depth is calculated to generate map points. The map points are then tracked further.

[0048] Traditional camera tracking modules extract feature points on standard image frames and perform tracking while adding a target detection model YOLOv5 to identify preset natural landmarks. When a preset natural landmark is detected, the PnP algorithm is used to calculate the pose and update the pose of the current frame.

[0049] The joint optimization module is used to achieve visual-inertial fusion through nonlinear optimization. It transforms the visual-inertial fusion problem into a minimization problem of an objective function, which consists of four parts: the prior error after marginalization, the reprojection error of the event camera and the traditional camera, and the error term of the inertial measurement. After solving the problem using the G2O optimization library, an accurate pose estimate is obtained.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] 1. This invention leverages the complementary performance advantages of event cameras and traditional cameras, employing a combination of event camera / traditional camera / IMU / natural landmarks to construct a visual odometry system. This overcomes the problem that traditional visual inertial odometry methods cannot work in high-speed, high-dynamic scenarios, while also improving robustness and accuracy compared to a single event camera or an event camera combined with an IMU.

[0052] 2. By using landmarks at known locations in the scene, target detection algorithms are used for identification, the current camera pose is calculated and the pose is updated, achieving a function similar to loop closure detection without requiring loops in the trajectory.

[0053] 3. By identifying potential moving objects in a scene through object detection algorithms and using static consistency to find and remove dynamic feature points, false matches in feature matching can be reduced, thus improving the robustness of the system. Attached Figure Description

[0054] Figure 1 This is a system structure diagram of the present invention.

[0055] Figure 2 Steps for updating pose to identify landmarks.

[0056] Figure 3 This is the step to remove dynamic points. Detailed Implementation

[0057] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0058] This invention provides an odometry system based on an event camera / IMU / natural landmarks. A traditional camera, an event camera, and an IMU element are mounted on a mobile vehicle platform. The sensors are rigidly connected, and the intrinsic parameters of each sensor and their mutual extrinsic parameters are pre-calibrated. Known landmarks are deployed in the task scenario, and the image dataset required for training the target detection algorithm model is pre-collected.

[0059] Establish a world coordinate system with the starting point of the moving car as the origin, the direction the car's front faces as the positive x-axis, the left side of the car's front facing as the positive y-axis, and the top of the car as the positive z-axis. During the operation of the moving car, acquire real-time event streams from event cameras, image streams from traditional cameras, and acceleration and angular velocity measurements from IMU elements.

[0060] The system of this invention includes an event camera tracking module, a conventional camera tracking module, and a joint optimization module. The event camera tracking module extracts feature points directly from the original events based on the event stream output by the event camera using SAE (Surface of Active Events). It then tracks the extracted feature points using the LK optical flow method, calculates their inverse depth, and generates map points, which are subsequently tracked further. The conventional camera tracking module extracts and tracks feature points on standard image frames while incorporating a YOLOv5 object detection model to identify preset natural landmarks. When a preset natural landmark is detected, the PnP algorithm is used to calculate the pose and update the pose of the current frame. The joint optimization module achieves visual-inertial fusion through nonlinear optimization, transforming the visual-inertial fusion problem into a minimization problem of an objective function. This objective function includes four parts: the prior error after marginalization, the reprojection error of the event camera and the conventional camera, and the error term of inertial measurement. Solving this using the G2O optimization library yields an accurate pose estimate.

[0061] This invention defines an event representation method, using polarized TS (Time Surface) for representation: Where T p Let x and y represent the intensity of the event, t represent the time the event occurred, p represent the polarity of the pixel intensity change, S represent the timestamp of the last time the event was triggered on this pixel, and τ represent a constant decay rate.

[0062] The event camera tracking module is used for the following processing:

[0063] Step 1: Generate an SAE based on the event stream provided by the event camera. The SAE records the timestamp of the last time an event was triggered for each pixel in the event stream. It is a two-dimensional array, where each element corresponds to a pixel and its value corresponds to the timestamp of the last time an event was triggered.

[0064] Step 2: When a new event stream is received, the Arc* algorithm is used to extract corner points on the generated SAE. The Arc* algorithm maintains two circular event sets and detects whether consecutive arcs or their complementary arcs are within a certain range to extract corner points. The specific steps of the Arc* algorithm are as follows:

[0065] (1) Define two circular event sets, each containing the N most recent events;

[0066] (2) When a new event arrives, add it to the two ring event sets, calculate the radian between the event and other events in the ring, and record the range of the continuous arc or its supplementary arc;

[0067] (3) For each newly added event, check whether its corresponding continuous arc or supplementary arc is within a certain range. If so, mark the event as a corner point.

[0068] (4) Remove the oldest event from the two circular event sets to ensure that the set contains only the most recent N events.

[0069] Step 3: Use the LK optical flow method to track the extracted feature points. If the tracking is successful, triangulate the points to generate map points and perform further tracking. If the tracking fails, discard the feature point. When the number of feature points is lower than the set threshold, extract new feature points.

[0070] During tracking, keyframes are extracted and sent to the joint optimization module for optimization calculations. Keyframe selection is based on the number of currently tracked features and the average disparity between the current number of ...

[0071] Image data acquired by a traditional camera is processed, starting with preprocessing steps such as distortion correction. The image is then processed using a traditional camera tracking module.

[0072] Landmark recognition is performed on the preprocessed image using the YOLOv5 object detection algorithm (the YOLOv5 model needs to be pre-trained). The specific steps are as follows:

[0073] (1) Control a small car equipped with a traditional camera to collect image datasets in a scene with landmarks;

[0074] (2) Extract a portion of the images from the collected dataset for target labeling, and mark the landmarks;

[0075] (3) Divide the labeled images into training and validation sets, and train the network based on the pre-trained network;

[0076] (4) Once the model is trained, it can be used for landmark target detection.

[0077] FAST corner points are also extracted from the standard image frame. Among the extracted feature points, there may be feature points of dynamic objects, which may cause errors in subsequent tracking. It is necessary to remove dynamic feature points. The specific steps for removing dynamic points are as follows:

[0078] (1) Use the latest YOLOv5 algorithm to process each frame of the image and identify all objects in it;

[0079] (2) Save the rectangular region where the potential dynamic objects in the recognition results are located. The format of the saved region is (x1, y1, h, w), where x1 is the horizontal coordinate of the upper left corner of the region, y1 is the vertical coordinate of the upper left corner of the region, h is the height of the rectangular region, and w is the width of the rectangular region.

[0080] (3) By using static consistency, feature points in potential moving object regions are divided into static feature points and dynamic feature points, and finally, dynamic feature points are eliminated. For feature points i,j in the image frame at time t and their corresponding feature points i,j in the image frame at time t+1, the following is used: This means that the distance between two feature points in an image frame at the same time is defined as... Where u and v are the x and y coordinates of the feature point in the pixel coordinate system. Based on the consistency of static objects, if two feature points are static, the distance between corresponding feature points in image frames at different times is the same. Considering the existence of errors, the distance difference between corresponding feature points in image frames at different times only needs to be within a certain threshold range. The threshold is determined by feature points within the non-potentially moving object region, and its calculation formula is as follows: S is the set of feature points within the region of a non-potentially moving object, d t (i,j) represents the distance between any two feature points in set S at time t, and N represents the number of feature points in set S.

[0081] The LK optical flow method is used to track the obtained static feature points. During the tracking process, key frames are selected. The selection of key frames is based on the number of tracked feature points and the distance from the previous key frame. A threshold is set for each. When the number of tracked key frames is lower than the set threshold or the distance from the previous key frame is greater than the set threshold, a new key frame is selected.

[0082] If a landmark is identified during the landmark recognition step using the object detection algorithm, since the 3D position of the landmark itself is known, the 2D coordinates of the landmark in the image can be obtained after landmark recognition. This yields a set of corresponding 2D-3D point pairs. The PNP algorithm can then be used to solve for the pose estimate relative to the landmark at the current moment, which can then be converted into a pose estimate relative to the world coordinate system: T WC =T WL ·T LC The pose of the image at the current moment is updated using the calculated results.

[0083] Using information provided by the event camera tracking module and the traditional camera tracking module, a least-squares problem is constructed to minimize the objective function. The G2O optimization library is used to solve this problem, outputting a jointly optimized pose estimate. The entire optimization process is based on keyframes, employing a sliding window strategy that optimizes only the sliding window containing the last K keyframes. Between frames, IMU measurements are used to propagate predictions of the sensor state.

[0084] The objective function is:

[0085]

[0086] In this equation, χ on the left side is the variable to be estimated, the first term on the right side is the prior residual remaining after marginalization, the second term is the residual of IMU pre-integration, the third term is the measurement residual of the event camera, and the fourth term is the measurement residual of the conventional camera. (·) The weights are the values ​​corresponding to each residual.

[0087] This invention employs a combined approach of event camera / traditional camera / IMU / natural landmarks, overcoming the limitation of traditional visual inertial odometry in high-speed, high-dynamic scenarios. Furthermore, by pre-positioning landmarks with known location information within the scene, it achieves a loop closure detection-like function within the system. Additionally, a step to remove dynamic points from the environment is added to the traditional camera tracking module, improving the system's robustness.

[0088] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for odometry based on event camera / IMU / natural landmarks, characterized in that, Includes the following steps: Step 1, Event Camera Tracking Based on the event stream output by the event camera, feature points are extracted directly from the original event using SAE. The extracted feature points are then tracked using the LK optical flow method, and their inverse depth is calculated to generate map points. Further tracking of these map points is then performed. Specifically, the process includes the following sub-steps: Step 1: Generate an SAE based on the event stream provided by the event camera. The SAE records the timestamp of the last triggered event for each pixel in the event stream. It is a two-dimensional array, where each element corresponds to a pixel and its value corresponds to the timestamp of the last triggered event. Step 2: When a new event stream is received, the Arc* algorithm is used to extract corner points on the generated SAE. The Arc* algorithm maintains two circular event sets and detects whether continuous arcs or their complementary arcs are within a certain range to extract corner points. Step 3: Use the LK optical flow method to track the extracted feature points. If the tracking is successful, triangulate the points to generate map points and perform further tracking. If the tracking fails, discard the feature point. When the number of feature points is lower than the set threshold, extract new feature points. Step 2, Traditional Camera Tracking While extracting and tracking feature points on standard image frames, a YOLOv5 object detection model is added to identify preset natural landmarks. When a preset natural landmark is detected, the PnP algorithm is used to calculate the pose and update the pose of the current frame. The specific steps include the following: Step 1: Pre-collect image data to train the YOLOv5 model; Step 2: Perform target detection on standard image frames from a traditional camera to identify whether landmarks exist in the image; Step 3: Extract feature points from standard image frames, identify potential dynamic objects based on the target detection results, and use static consistency to divide the feature points in the potential dynamic object region into static feature points and dynamic feature points, and remove the dynamic feature points. Step 4: Use optical flow to track the extracted static feature points; Step 5: If a landmark was identified in Step 2, calculate the pose using the PnP algorithm and update the camera pose at this time. Step 3, Joint Optimization Visual-inertial fusion is achieved through nonlinear optimization, transforming the visual-inertial fusion problem into a minimization problem of an objective function. This objective function consists of four parts: the prior error after marginalization, the reprojection error between the event camera and the conventional camera, and the error term of inertial measurement. The accurate pose estimate is obtained by solving the problem using the G2O optimization library.

2. The odometry implementation method based on event camera / IMU / natural landmarks according to claim 1, characterized in that, The events are represented using polarized TS: ,in For the intensity of the event, For pixel coordinates, The time when the event occurred. This represents the polarity of the pixel intensity change. The timestamp of the last time an event was triggered for this pixel. This represents a constant decay rate.

3. The odometry implementation method based on event camera / IMU / natural landmarks according to claim 1, characterized in that, The specific steps for extracting feature points using the Arc* algorithm in step 2 are as follows: (1) Define two circular event sets, each containing the N most recent events; (2) When a new event arrives, add it to the two ring event sets, calculate the radian between the event and other events in the ring, and record the range of the continuous arc or its supplementary arc; (3) For each newly added event, check whether its corresponding continuous arc or supplementary arc is within a certain range. If so, mark the event as a corner point. (4) Remove the oldest event from the two circular event sets to ensure that the set contains only the most recent N events.

4. The odometry implementation method based on event camera / IMU / natural landmarks according to claim 1, characterized in that, In step 3, key frames are extracted during the tracking process and sent to the joint optimization module for optimization calculation. The selection of key frames is based on the number of currently tracked features and the average disparity between the tracked features and two consecutive time stamps. A threshold is set for each. When the number of tracked features is lower than the set threshold or the average disparity between the tracked feature points and two consecutive time stamps is greater than the set threshold, a new key frame is selected.

5. The odometry implementation method based on event camera / IMU / natural landmarks according to claim 1, characterized in that, The specific steps for removing dynamic feature points in step 3 are as follows: (1) Use the latest YOLOv5 algorithm to process each frame of the image and identify all objects in it; (2) Save the rectangular region containing the potential dynamic objects in the recognition results. The saved format is as follows: , The x-coordinate of the top-left corner of the region. The ordinate of the top-left corner of the region. The height of the rectangular area, The width of the rectangular area; (3) By using static consistency, feature points in the potential moving object region are divided into static feature points and dynamic feature points, and finally dynamic feature points are eliminated; specifically, the following process is used to distinguish between static and dynamic feature points: against Feature points in a time frame and its corresponding Feature points in a time frame ,use This means that the distance between two feature points in an image frame at the same time is defined as: in, Here, x and y represent the x and y coordinates of the feature points in the pixel coordinate system. When the distance between corresponding feature points in image frames at different times is the same or the distance difference is within a certain threshold range, the two feature points are considered static feature points. The threshold is determined by feature points within a non-potentially moving object region, and its calculation formula is: ， in, It is the set of feature points within the region of a non-potentially moving object. for The distance between any two feature points in set S at time t is given by N, where N is the number of feature points in set S.

6. The odometry implementation method based on event camera / IMU / natural landmarks according to claim 1, characterized in that, The specific steps for updating the pose estimation by identifying artificial landmarks in steps 2 and 5 are as follows: (1) Collect image data of pre-arranged natural landmarks in the scene from different angles; (2) Mark the targets on the collected image data and mark the parts of the preset natural landmarks; (3) Train the model using the labeled dataset; (4) Use the trained model to identify landmarks in the image. After successful identification, obtain the 3D coordinates of the landmark in the world coordinate system and the 2D coordinates in the image. Use the PnP algorithm to solve for the pose estimation and update the pose in real time.

7. The odometry implementation method based on event camera / IMU / natural landmarks according to claim 1, characterized in that, The objective function in step three is: ， Among them, the left side of the equation For the variable to be estimated, the first term on the right side of the equation is the prior residual remaining after marginalization, the second term is the residual of IMU pre-integration, the third term is the measurement residual of the event camera, and the fourth term is the measurement residual of the conventional camera. The weights are the values ​​corresponding to each residual.

8. An odometry system based on event camera / IMU / natural landmarks, characterized in that, A method for implementing visual odometry based on an event camera / traditional camera / IMU / natural landmarks as described in any one of claims 1-7, comprising an event camera tracking module, a traditional camera tracking module, and a joint optimization module; wherein... The event camera tracking module is used to extract feature points directly from the original events based on the event stream output by the event camera and SAE (Surface of Active Events). The extracted feature points are tracked using the LK optical flow method, and their inverse depth is calculated to generate map points. The map points are then tracked further. Traditional camera tracking modules extract feature points on standard image frames and perform tracking while adding a target detection model YOLOv5 to identify preset natural landmarks. When a preset natural landmark is detected, the PnP algorithm is used to calculate the pose and update the pose of the current frame. The joint optimization module is used to achieve visual-inertial fusion through nonlinear optimization. It transforms the visual-inertial fusion problem into a minimization problem of an objective function, which consists of four parts: the prior error after marginalization, the reprojection error of the event camera and the traditional camera, and the error term of the inertial measurement. After solving the problem using the G2O optimization library, an accurate pose estimate is obtained.

Images