Multi-sensor fusion state estimation system based on event camera

Abstract

The invention discloses a multi-sensor fusion state estimation system based on an event camera. In the invention, an event representation method based on the combination of an event accumulator setting strategy and an optimized motion correction compensation method is provided, and a multi-sensor data fusion processing pipeline of event flow, imu inertial measurement and standard images is constructed based on the event representation method. The most advanced feature tracker DeepEvTracker driven based on event data is introduced for the first time and is combined with a KLT tracker based on an event motion compensation frame. The feature tracking loss in high-speed motion blur and HDR scenes is effectively reduced, and the feature updating frequency and the prediction displacement precision are remarkably improved. A multi-sensor fusion state estimation framework based on event data driving is elaborately designed, robust and accurate state estimation can be provided in a challenging scene, dense manual parameter adjustment does not need to be depended on, real-time performance is guaranteed, and meanwhile the method is superior to a current most advanced state estimation algorithm based on events and frames.

Description

Technical Field

[0001] The present invention belongs to the technical field of state estimation, and in particular relates to a multi-sensor fusion state estimation system based on event cameras. Background Art

[0002] State estimation is a very important topic in the field of robotics, which is widely used in navigation, path planning, control, augmented / virtual reality (AR / VR), and other aspects. However, in challenging scenarios such as intense motion or high exposure and low illumination, due to the inherent limitations of standard cameras, motion blur caused by high-speed motion destroys visual information, forcing the pipeline to rely too much on the integration of IMU, resulting in a large amount of accumulated drift errors; or due to the limited dynamic range of standard cameras, most areas on the image are either overexposed or underexposed, which greatly reduces the amount of available information. These traditional visual inertial odometry (VIO) pipeline methods still have difficulty in coping with many scenarios, so how to achieve robust and accurate state estimation is crucial for real-time feedback control of robots and practical applications.

[0003] A new class of dynamic vision sensors (DVS), called event cameras, offers great potential for overcoming these issues. Event camera pixels independently detect positive or negative brightness changes with microsecond resolution. Event cameras can provide data at 1MHz and 120dB dynamic range, both orders of magnitude higher than what can be achieved with standard 60dB cameras. This can significantly reduce motion blur and enable operation in high dynamic range (HDR), low light, and fast motion conditions. These features make event cameras an ideal complement to standard cameras in challenging scenes, providing robustness to state estimation algorithms based on vision and inertial measurements, thereby completing accurate and robust ego-motion estimation.

[0004] However, introducing event cameras into the state estimation pipeline is a very challenging task. This is because the event stream consists of asynchronous events, which is fundamentally different from the synchronous standard intensity image, and traditional computer vision algorithms cannot be directly applied. Some existing works reconstruct intensity frames from event data, or aggregate a group of events in a short period of time into event frames with image properties. However, this will bring bottlenecks and reduce the natural advantages of event cameras; some directly use asynchronous raw events for feature detection and tracking, but the tracking performance is poor in the presence of noise, and it relies on iterative optimization of motion parameters, has very complex model parameters, limited scene migration scalability and difficult to deploy in practice. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-sensor fusion state estimation system based on event camera in order to solve the above-mentioned problems.

[0006] The technical solution adopted by the present invention is as follows: a multi-sensor fusion state estimation system based on an event camera, characterized in that: when the system is running, it includes:

[0007] Step 1: First, the event stream is grouped in time and space windows through a carefully designed accumulator strategy setting module;

[0008] Step 2: Fuse IMU data to perform event data correction and further optimize motion compensation to obtain optimized motion compensated event frames;

[0009] Step 3: Perform Shi-Tomasi feature point detection on the motion compensated event frame, then filter out erroneous features by locating and removing isolated events and then track based on the traditional KLT tracker;

[0010] Step 4: The extracted feature points are input as the initial image template block into DeepEvTracker, an event-driven feature tracker with the best performance.

[0011] Step 5: The standard image and event stream are simultaneously input into the feature extraction network of the DeepEvTracker tracker to process the generated events and template images, generate pixel-by-pixel feature maps, and use these feature maps to construct correlation maps to calculate pixel similarity;

[0012] Step 6: Through the self-attention mechanism module in DeepEvTracker, all feature points and image templates from motion compensated event frames and standard frames are combined to achieve cross-source sharing of feature information, thereby optimizing the prediction of image block displacement;

[0013] Step 7: Use the event frame KLT tracking method to track features in the blind area between two frames, and use the displacement prediction of the DeepEvTracker tracking model as the initial prediction of the KLT tracker after the new frame arrives;

[0014] Step 8: Determine candidate landmarks and persistent landmarks, the latter of which have been successfully triangulated in 3D space. The newly extracted features are initialized as candidate landmarks and tracked in the event frame. Once a candidate landmark can be reliably triangulated, it is converted to a persistent landmark and added to the local map;

[0015] Step 9: Landmarks that are not successfully tracked in the current frame are immediately discarded. Two-point RANSAC (using the relative direction between the current frame and the last keyframe) is used to further filter out outlier feature tracks;

[0016] Step 10: The fused back-end feature trajectory is provided to a nonlinear optimization framework based on a sliding window key frame to update the state of the entire sensor over time.

[0017] Step 11: Use the IMU basic kinematic and bias model to predict the current state based on the previous state. The IMU error term is then calculated as the difference between the prediction based on the previous state and the actual state.

[0018] Step 12: Use schurcomplement to marginalize out the old keyframes on the corresponding observations, making the old measurements a priori estimates. Between frames, the prediction of the sensor state is propagated using the IMU measurements.

[0019] Step 13: By minimizing the cost function of the visual term (including the reprojection error of events and images) and the inertial term, the camera trajectory and the position of the 3D landmarks are regularly refined, thereby effectively using events, standard frames and IMU to refine the camera pose and achieve accurate and robust state estimation. After that, the entire event-based camera multi-sensor fusion state process can be ended.

[0020] In a preferred embodiment, in step S1 and step S2, an event accumulator optimization module including a slicing adjustment method, a motion-free processing method, polarity usage, and an attenuation function and a motion compensation and correction module based on IMU measurement are respectively constructed to complete the preprocessing and characterization of the event stream.

[0021] At the same time, in order to alleviate the problem that the noise is much greater than the valid event point data due to the inherent motion properties of the event camera during slow motion or stillness, a simple but very effective method is adopted. A threshold is set. When the number of events in the interval is less than the threshold, the old event frame will be copied to the new event frame, that is, the camera remains still during this interval. In the end, a clear edge event frame with sufficient feature information and event data robustness can be obtained.

[0022] In the case of fast scene changes, the generated image will be somewhat blurred due to the compression of too many event points, which is manifested as coarse edges of objects in the image, which will also affect the subsequent state estimation. Therefore, the IMU measurement value is used here to complete the optimized version of motion compensation. Compared with other solutions, the response speed is faster and the compensation effect is better for subsequent feature extraction.

[0023] In a preferred embodiment, in steps S3, S4 and S9, a Shi-Tomasi corner detector is used on the motion compensated event frame, and a bucket grid (wherein the size of each grid unit is 32×32 pixels) is used, and a minimum distance is set between feature points of two adjacent event frames to ensure that the features are evenly distributed on the image.

[0024] It is worth noting that event cameras generate a lot of noise, which will inevitably affect the quality and stability of event frame imaging, thereby affecting the extraction of feature points and subsequent tracking. To alleviate this problem, a fast and efficient method is used here to discard incorrect feature points that may be generated due to event noise. The process is similar to image filtering, which seeks to eliminate erroneous features by locating and removing isolated events. The threshold of the filter depends on the configuration of the event camera, as it may give different noise profiles. At the same time, the filtering parameters are adjusted to emphasize event corner features detected in strong edges rather than large noise features in low-texture areas. Finally, although filtering tends to treat small details in edge images as noise and remove them, it actually helps to obtain more stable images, focusing more on extracting major edges from the scene and ignoring some redundant textures.

[0025] Two sets of landmarks are maintained: candidate landmarks and persistent landmarks, the latter of which have their 3D positions in space successfully triangulated. Newly extracted features are initialized as candidate landmarks and tracked in event frames. Once a candidate landmark can be reliably triangulated, it is converted to a persistent landmark and added to the local map. Two-point RANSAC (using the relative orientation between the current frame and the last keyframe) is also used to further filter out outlier feature tracks.

[0026] In a preferred embodiment, in steps S5 and S6, the DeepEvTracker tracker feature extraction network first uses an FPN encoder to process the event and template images to generate pixel-by-pixel feature maps. These feature maps are then used to construct correlation maps to calculate pixel similarity. Subsequently, the feature maps and correlation maps are fed into a subsequent encoder for refinement, in which temporal information and motion features are integrated.

[0027] In addition, DeepEvTracker proposes a frame attention module that fuses and refines feature information for the relevant motion of rigid bodies in the image. Therefore, this module combines feature points from event frames and standard frames to achieve cross-source sharing of feature information, thereby optimizing the prediction of image block displacement.

[0028] In a preferred embodiment, in steps S7 and S8, in order to make fuller use of the texture information from the grayscale image and the high-delay robust performance of the event, the event frame KLT tracking method is used here to expand the event-based tracking image template method. Specifically, the event frame KLT tracking method is used to track the features in the blind spot between the two frames, and the displacement prediction of the DeepEvTracker tracking model is used as the initial prediction of the KLT tracker after the new frame arrives. The advantage of doing so is that the negative impact of the excessive baseline between the two frames caused by high-speed motion can be effectively alleviated. In addition, the combination of the KLT tracking method based on the event motion compensation frame and the DeepEvTracker event tracking model provides more feature positions between the two frames, more accurately corrects the feature positions, and significantly increases the frequency of feature position updates.

[0029] In a preferred embodiment, in steps S10 to S13, the optimization variables are defined as:

[0030]

[0031] This visual-inertial localization and mapping problem involves the joint optimization of a cost function consisting of two weighted projection errors constructed from observations from the event camera and the standard camera, an IMU inertial error term, a marginalization error term, and a loop closure and relocalization error term.

[0032]

[0033] The reprojection error is defined as:

[0034]

[0035] Its i,j,k Denotes the jth landmark point measured by the i-th sensor in the k-th frame, and uses the basic kinematic and bias models of the IMU to predict the current state based on the previous state. Then, the IMU error term is calculated as the difference between the prediction based on the previous state and the actual state.

[0036] The IMU error term is:

[0037]

[0038] The optimization is not performed on all observed frames, but on a bounded frame set consisting of M keyframes and a sliding window containing the last K frames. Keeping all keyframes in the GN optimization system will affect the running speed of the system, and simply discarding past keyframes will lose some useful information. To overcome this problem, schur complement is used on the corresponding observations to marginalize the old keyframes, making the old measurements a priori estimates.

[0039] In a preferred embodiment, in steps S10 and S13, the initialization process of EVIO starts from a pure visual structure from motion (SfM) to build the upper scale structure of camera pose and event angle feature position. By loosely aligning SfM with pre-integrated IMU measurements, the system can be guided from an unknown initial state, rather than assuming that the sensor remains static during the initialization phase, or assuming that the local scene is planar.

[0040] In a preferred embodiment, in steps S10 and S13, in order to eliminate the accumulated drift and ensure the global consistency of long-term motion, the loop detection will also perform feature point detection in two channels in the motion compensation event frame and the standard image frame, and describe them by the Brief descriptor, and further feed them to the back end (as shown in the figure below). These additional features are used to achieve a better recall rate in loop detection.

[0041] In a preferred embodiment, in steps S10 and S13, motion compensated event frames will be triggered in scenes with strong edges, which can also help position identification and ensure global consistency. The correspondence is found by identifying the BRIEF descriptor match through the Hamming distance. When the number of correspondences of the event descriptor is greater than a certain threshold, a closed loop is detected. At the same time, a two-step geometric outlier is used to reject erroneous BRIEF descriptor matches, and DBoW2 is used for loop identification. After the loop is detected, the connection residual of the previous keyframe and the current keyframe is integrated into the pose graph optimization.

[0042] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0043] 1. In the present invention, first, the event stream is grouped in the spatiotemporal window and fused after correction using IMU data to construct a motion compensation frame (so that it has the properties of a grayscale image) through a carefully designed accumulator strategy setting module. Then, the event frame and the image frame are respectively subjected to Shi-Tomasi feature point detection and the traditional KLT tracker for tracking. At the same time, the extracted feature points are input as the initial image template block to the event data-driven feature tracker with the best performance at present. The event tracker tracks the features in the blind time between the two frames, and uses the displacement prediction of the event tracker as the initial value of the KLT tracker after the new frame arrives.

[0044] The feature trajectories of the two heterogeneous sources (event frames and standard frames) are then combined through an improved and extended frame attention module from the event data-driven feature tracker to achieve feature trajectory information sharing and fusion, and provided to a nonlinear optimization framework based on sliding window key frames. By minimizing the cost function of visual terms (including event and image reprojection errors) and inertial terms, the camera trajectory and the position of 3D landmarks are regularly refined, thereby effectively using events, standard frames and IMU to refine the camera pose and achieve more accurate and robust state estimation.

[0045] 2. In the present invention, an event characterization method based on the combination of event accumulator setting strategy and optimized motion correction compensation method is proposed, and based on this, a multi-sensor data fusion processing pipeline of event stream, imu inertial measurement, and standard image is constructed. For the first time, the most advanced event data-driven feature tracker DeepEvTracker is introduced and combined with the KLT tracker based on event motion compensation frame. It effectively reduces the feature tracking loss in high-speed motion blur and HDR scenes and significantly improves the feature update frequency and predicted displacement accuracy. A multi-sensor fusion state estimation framework driven by event data is carefully designed, which can provide robust and accurate state estimation in challenging scenarios, without relying on intensive manual parameter adjustment and ensuring real-time performance, while outperforming the most advanced event- and frame-based state estimation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the present invention;

[0047] Figure 2 The comparison experiment diagrams before and after optimization of each scene in the present invention are as follows;

[0048] Figure 3 This is a visualization result diagram showing event data in indoor and outdoor scenes of different sizes in the present invention;

[0049] Figure 4 This is a diagram showing the results of the DeepEvTracker event tracking model in the present invention;

[0050] Figure 5 It is a dual-channel loop detection diagram of the event frame and the standard image in the present invention;

[0051] Figure 6 is a comparison chart of the test accuracy of the benchmark data set of the present invention;

[0052] Figure 7 This is a diagram showing the ablation experiment results in the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Reference Figure 1-7 ,

[0055] A multi-sensor fusion state estimation system based on an event camera, the system includes:

[0056] Step 1: First, the event stream is grouped in time and space windows through a carefully designed accumulator strategy setting module;

[0057] Step 2: Fuse IMU data to perform event data correction and further optimize motion compensation to obtain optimized motion compensated event frames;

[0058] Step 3: Perform Shi-Tomasi feature point detection on the motion compensated event frame, then filter out erroneous features by locating and removing isolated events and then track based on the traditional KLT tracker;

[0059] Step 4: The extracted feature points are input as the initial image template block into DeepEvTracker, an event-driven feature tracker with the best performance.

[0060] Step 5: The standard image and event stream are simultaneously input into the feature extraction network of the DeepEvTracker tracker to process the generated events and template images, generate pixel-by-pixel feature maps, and use these feature maps to construct correlation maps to calculate pixel similarity;

[0061] Step 6: Through the self-attention mechanism module in DeepEvTracker, all feature points and image templates from motion compensated event frames and standard frames are combined to achieve cross-source sharing of feature information, thereby optimizing the prediction of image block displacement;

[0062] Step 7: Use the event frame KLT tracking method to track features in the blind area between two frames, and use the displacement prediction of the DeepEvTracker tracking model as the initial prediction of the KLT tracker after the new frame arrives;

[0063] Step 8: Determine candidate landmarks and persistent landmarks, the latter of which have been successfully triangulated in 3D space. The newly extracted features are initialized as candidate landmarks and tracked in the event frame. Once a candidate landmark can be reliably triangulated, it is converted to a persistent landmark and added to the local map;

[0064] Step 9: Landmarks that are not successfully tracked in the current frame are immediately discarded. Two-point RANSAC (using the relative direction between the current frame and the last keyframe) is used to further filter out outlier feature tracks;

[0065] Step 10: The fused back-end feature trajectory is provided to a nonlinear optimization framework based on a sliding window key frame to update the state of the entire sensor over time.

[0066] Step 11: Use the IMU basic kinematic and bias model to predict the current state based on the previous state. The IMU error term is then calculated as the difference between the prediction based on the previous state and the actual state.

[0067] Step 12: Use schurcomplement to marginalize out the old keyframes on the corresponding observations, making the old measurements a priori estimates. Between frames, the prediction of the sensor state is propagated using the IMU measurements.

[0068] Step 13: By minimizing the cost function of the visual term (including the reprojection error of events and images) and the inertial term, the camera trajectory and the position of the 3D landmarks are regularly refined, thereby effectively using events, standard frames and IMU to refine the camera pose and achieve accurate and robust state estimation. After that, the entire event-based camera multi-sensor fusion state process can be ended.

[0069] In step S1 and step S2, an event accumulator optimization module including a slicing adjustment method, a motion-free processing method, polarity usage, and an attenuation function and a motion compensation and correction module based on IMU measurement are respectively constructed to complete the preprocessing and characterization of the event stream.

[0070] At the same time, in order to alleviate the problem that the noise is much greater than the valid event point data due to the inherent motion properties of the event camera during slow motion or stillness, a simple but very effective method is adopted. A threshold is set. When the number of events in the interval is less than the threshold, the old event frame will be copied to the new event frame, that is, the camera remains still during this interval. In the end, a clear edge event frame with sufficient feature information and event data robustness can be obtained.

[0071] In the case of fast scene changes, the generated image will be somewhat blurred due to the compression of too many event points, which is manifested as coarse edges of objects in the image, which will also affect the subsequent state estimation. Therefore, the IMU measurement value is used here to complete the optimized version of motion compensation. Compared with other solutions, the response speed is faster and the compensation effect is better for subsequent feature extraction.

[0072] In steps S3, S4 and S9, the Shi-Tomasi corner detector is used on the motion compensated event frame, and a bucket grid (where the size of each grid unit is 32×32 pixels) is used, and a minimum distance is set between feature points of two adjacent event frames to ensure that the features are evenly distributed on the image.

[0073] It is worth noting that event cameras generate a lot of noise, which will inevitably affect the quality and stability of event frame imaging, thereby affecting the extraction of feature points and subsequent tracking. To alleviate this problem, a fast and efficient method is used here to discard incorrect feature points that may be generated due to event noise. The process is similar to image filtering, which seeks to eliminate erroneous features by locating and removing isolated events. The threshold of the filter depends on the configuration of the event camera, as it may give different noise profiles. At the same time, the filtering parameters are adjusted to emphasize event corner features detected in strong edges rather than large noise features in low-texture areas. Finally, although filtering tends to treat small details in edge images as noise and remove them, it actually helps to obtain more stable images, focusing more on extracting major edges from the scene and ignoring some redundant textures.

[0074] Two sets of landmarks are maintained: candidate landmarks and persistent landmarks, the latter of which have their 3D positions in space successfully triangulated. Newly extracted features are initialized as candidate landmarks and tracked in event frames. Once a candidate landmark can be reliably triangulated, it is converted to a persistent landmark and added to the local map. Two-point RANSAC (using the relative orientation between the current frame and the last keyframe) is also used to further filter out outlier feature tracks.

[0075] In steps S5 and S6, the DeepEvTracker feature extraction network first uses the FPN encoder to process the event and template images to generate pixel-by-pixel feature maps. These feature maps are then used to construct correlation maps to calculate pixel similarity. Subsequently, the feature maps and correlation maps are fed into the subsequent encoder for refinement, which incorporates temporal information and motion features.

[0076] In addition, DeepEvTracker proposes a frame attention module that fuses and refines feature information for the relevant motion of rigid bodies in the image. Therefore, this module combines feature points from event frames and standard frames to achieve cross-source sharing of feature information, thereby optimizing the prediction of image block displacement.

[0077] In steps S7 and S8, in order to make better use of the texture information from the grayscale image and the high-delay robust performance of the event, the event frame KLT tracking method is used here to expand the event-based tracking image template method. Specifically, the event frame KLT tracking method is used to track the features in the blind area between the two frames, and the displacement prediction of the DeepEvTracker tracking model is used as the initial prediction of the KLT tracker after the new frame arrives. The advantage of doing this is that it can effectively reduce the negative impact of the excessive baseline between the two frames caused by high-speed motion. In addition, the combination of the KLT tracking method based on the event motion compensation frame and the DeepEvTracker event tracking model provides more feature positions between the two frames, corrects the feature positions more accurately, and significantly increases the frequency of feature position updates.

[0078] In steps S10 to S13, the optimization variables are defined as:

[0079]

[0080] This visual-inertial localization and mapping problem involves the joint optimization of a cost function consisting of two weighted projection errors constructed from observations from the event camera and the standard camera, an IMU inertial error term, a marginalization error term, and a loop closure and relocalization error term.

[0081]

[0082] The reprojection error is defined as:

[0083]

[0084] where z i,j,k Denotes the jth landmark point measured by the i-th sensor in the k-th frame, and uses the basic IMU kinematic and bias models to predict the current state based on the previous state. The IMU error term is then calculated as the difference between the prediction based on the previous state and the actual state.

[0085] The IMU error term is:

[0086]

[0087] The optimization is not performed on all observed frames, but on a bounded frame set consisting of M keyframes and a sliding window containing the last K frames. Keeping all keyframes in the GN optimization system will affect the running speed of the system, and simply discarding past keyframes will lose some useful information. To overcome this problem, schur complement is used on the corresponding observations to marginalize the old keyframes, making the old measurements a priori estimates.

[0088] In steps S10 and S13, EVIO's initialization process starts with a pure visual structure from motion (SfM) to build the upper-scale structure of camera pose and event angular feature positions. By loosely aligning SfM with pre-integrated IMU measurements, the system can be guided from an unknown initial state, rather than assuming that the sensor remains static during the initialization phase, or assuming that the local scene is planar.

[0089] In steps S10 and S13, in order to eliminate accumulated drift and ensure global consistency of long-term motion, loop detection also performs feature point detection in two channels in motion compensated event frames and standard image frames, which are described by Brief descriptors and further fed to the backend (as shown in the figure below). These additional features are used to achieve better recall rates in loop detection.

[0090] In steps S10 and S13, motion compensated event frames are triggered in scenes with strong edges, which can also help position identification and ensure global consistency. Correspondences are found by identifying BRIEF descriptor matches through Hamming distance. When the number of correspondences of the event descriptor is greater than a certain threshold, a closed loop is detected. At the same time, a two-step geometric outlier is used to reject incorrect BRIEF descriptor matches, and DBoW2 is used for loop identification. After the loop is detected, the connection residual of the previous keyframe and the current keyframe is integrated into the pose graph optimization.

[0091] In the present invention, first, the event stream is grouped in a spatiotemporal window and fused after correction using IMU data to construct a motion compensation frame (so that it has the properties of a grayscale image) through a carefully designed accumulator strategy setting module. Then, the event frame and the image frame are respectively subjected to Shi-Tomasi feature point detection and a traditional KLT tracker for tracking. At the same time, the extracted feature points are input as the initial image template block to the event data-driven feature tracker with the best current performance. The event tracker tracks the features in the blind time between the two frames, and uses the displacement prediction of the event tracker as the initial value of the KLT tracker after the new frame arrives.

[0092] The feature trajectories of the two heterogeneous sources (event frames and standard frames) are then combined through an improved and extended frame attention module from the event data-driven feature tracker to achieve feature trajectory information sharing and fusion, and provided to a nonlinear optimization framework based on sliding window key frames. By minimizing the cost function of visual terms (including event and image reprojection errors) and inertial terms, the camera trajectory and the position of 3D landmarks are regularly refined, thereby effectively using events, standard frames and IMU to refine the camera pose and achieve more accurate and robust state estimation.

[0093] In this invention, an event characterization method based on the combination of event accumulator setting strategy and optimized motion correction compensation method is proposed, and based on this, a multi-sensor data fusion processing pipeline of event stream, imu inertial measurement, and standard image is constructed. For the first time, the most advanced event data-driven feature tracker DeepEvTracker is introduced and combined with the KLT tracker based on event motion compensation frame. It effectively reduces the feature tracking loss in high-speed motion blur and HDR scenes and significantly improves the feature update frequency and predicted displacement accuracy. A multi-sensor fusion state estimation framework driven by event data is carefully designed, which can provide robust and accurate state estimation in challenging scenarios, without relying on intensive manual parameter adjustment and ensuring real-time performance, while outperforming the most advanced event- and frame-based state estimation algorithms.

[0094] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion state estimation system based on event camera, characterized by: The system includes when it is running: Step 1: First, the event stream is grouped in time and space windows through a carefully designed accumulator strategy setting module; Step 2: Fuse IMU data to perform event data correction and further optimize motion compensation to obtain optimized motion compensated event frames; Step 3: Perform Shi-Tomasi feature point detection on the motion compensated event frame, then filter out erroneous features by locating and removing isolated events and then track based on the traditional KLT tracker; Step 4: The extracted feature points are input as the initial image template block into DeepEvTracker, an event-driven feature tracker with the best performance. Step 5: The standard image and event stream are simultaneously input into the feature extraction network of the DeepEvTracker tracker to process the generated events and template images, generate pixel-by-pixel feature maps, and use these feature maps to construct correlation maps to calculate pixel similarity; Step 6: Through the self-attention mechanism module in DeepEvTracker, all feature points and image templates from motion compensated event frames and standard frames are combined to achieve cross-source sharing of feature information, thereby optimizing the prediction of image block displacement; Step 7: Use the event frame KLT tracking method to track features in the blind area between two frames, and use the displacement prediction of the DeepEvTracker tracking model as the initial prediction of the KLT tracker after the new frame arrives; Step 8: Determine candidate landmarks and persistent landmarks, the latter of which have been successfully triangulated in 3D space; the newly extracted features are initialized as candidate landmarks and tracked in the event frame; once a candidate landmark can be reliably triangulated, it is converted to a persistent landmark and added to the local map; Step 9: Landmarks that are not successfully tracked in the current frame are immediately discarded; two-point RANSAC is used to further filter out outlier feature tracks using the relative direction between the current frame and the last keyframe; Step 10: The fused back-end feature trajectory is provided to a nonlinear optimization framework based on a key frame of a sliding window to update the state of the entire sensor over time; Step 11: Use the basic IMU kinematic and bias model to predict the current state based on the previous state; then, the IMU error term is calculated as the difference between the prediction based on the previous state and the actual state; Step 12: Use schurcomplement to marginalize the old keyframes on the corresponding observations, so that the old measurements become prior estimates; between frames, the prediction of the sensor state is propagated using the IMU measurements; Step 13: By minimizing the cost function of visual and inertial terms, the camera trajectory and the position of the 3D landmarks are regularly refined, thereby effectively using events, standard frames, and IMU to refine the camera pose and achieve accurate and robust state estimation, after which the entire event-based camera multi-sensor fusion state process is terminated.

2. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In the step S1 and step S2, an event accumulator optimization module including a slicing adjustment method, a motion-free processing method, polarity use, and an attenuation function and a motion compensation and correction module based on IMU measurement are respectively constructed to complete the preprocessing and characterization of the event stream; At the same time, in order to alleviate the problem of the inherent motion properties of the event camera causing the noise to be much greater than the valid event point data during slow motion or stillness, a simple but very effective method is adopted; A threshold is set. When the number of events in the interval is less than the threshold, the old event frame will be copied to the new event frame, that is, the camera remains still during this interval. Finally, a clear edge event frame with sufficient feature information and event data robustness can be obtained. When the scene changes quickly, the generated image will be blurred due to the compression of too many event points, which is manifested as coarser edges of objects in the image, which will also affect the subsequent state estimation. Therefore, the IMU measurement value is used here to complete the optimized motion compensation. Compared with other solutions, the response speed is faster and the compensation effect is better for subsequent feature extraction.

3. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In the steps S3, S4 and S9, a Shi-Tomasi corner detector is used on the motion compensated event frame, and a bucket grid is used, and a minimum distance is set between feature points of two adjacent event frames to ensure that the features are evenly distributed on the image; Also seeks to eliminate erroneous features by locating and removing isolated events; The filtering threshold depends on the configuration of the event camera, and the filtering parameters are adjusted to emphasize the detected event corner features located in strong edges rather than the large noise features in low-texture areas; Finally, although filtering tends to treat small details in edge images as noise and remove them; Two sets of landmarks are maintained: candidate landmarks and persistent landmarks, the latter of which have their 3D positions in space successfully triangulated. Newly extracted features are initialized as candidate landmarks and tracked in event frames. Once a candidate landmark can be reliably triangulated, it is converted to a persistent landmark and added to the local map. Two-point RANSAC is used to further filter out outlier feature tracks.

4. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In the steps S5 and S6, the DeepEvTracker tracker feature extraction network first uses the FPN encoder to process the event and template images to generate pixel-by-pixel feature maps; then uses these feature maps to construct correlation maps to calculate pixel similarity; then the feature maps and correlation maps are sent to the subsequent encoder for refinement, which integrates time information and motion features; DeepEvTracker proposes a frame attention module that fuses and refines feature information based on the relevant motion of rigid bodies in images. Therefore, this module is used to combine feature points from event frames and standard frames to achieve cross-source sharing of feature information.

5. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In the steps S7 and S8, the event-based tracking image template method is expanded using the event frame KLT tracking method; the event frame KLT tracking method is used to track the features in the blind area between two frames, and the displacement prediction of the DeepEvTracker tracking model is used as the initial prediction of the KLT tracker after the new frame arrives. The combination of the KLT tracking method based on the event motion compensation frame and the DeepEvTracker event tracking model provides more feature positions between the two frames, corrects the feature positions more accurately, and significantly increases the frequency of feature position updates.

6. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In the steps S10 to S13, the optimization variables are defined as: This visual-inertial localization and mapping problem involves the joint optimization of a cost function consisting of two weighted projection errors constructed from observations from the event camera and the standard camera, an IMU inertial error term, a marginalization error term, and a loop closure and relocalization error term. The reprojection error is defined as: where z i,j,k represents the jth landmark point measured by the i-th sensor in the k-th frame, and uses the basic kinematic and bias models of the IMU to predict the current state based on the previous state; then, the IMU error term is calculated as the difference between the prediction based on the previous state and the actual state; The IMU error term is: The optimization is not performed on all observed frames, but on a bounded frame set consisting of M key frames and a sliding window containing the last K frames; keeping all key frames in the GN optimization system will affect the running speed of the system, and simply discarding past key frames will lose some useful information. To overcome this problem, schur complement is used on the corresponding observations to partially marginalize the old key frames, making the old measurements a priori estimates.

7. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In steps S10 and S13, the initialization process of EVIO starts from the pure visual structure of motion to build the upper-scale structure of camera pose and event angle feature position; by loosely aligning SfM with pre-integrated IMU measurements, the system is guided from an unknown initial state, rather than assuming that the sensor remains static during the initialization phase, or assuming that the local scene is planar.

8. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In steps S10 and S13, in order to eliminate accumulated drift and ensure global consistency of long-term motion, loop detection also performs feature point detection in two channels in motion compensated event frames and standard image frames, and describes them with Brief descriptors, which are further fed to the back end; these additional features are used to achieve better recall rate in loop detection.

9. The multi-sensor fusion state estimation system based on event camera according to claim 1, characterized in that: In the steps S10 and S13, motion compensated event frames will be triggered in scenes with strong edges, which can also help position identification and ensure global consistency; the corresponding relationship is found by matching BRIEF descriptors through Hamming distance identification; when the number of corresponding relationships of the event descriptors is greater than a certain threshold, a closed loop is detected; at the same time, a two-step geometric outlier is used to reject erroneous BRIEF descriptor matching, and DBoW2 is used for loop identification; after the loop is detected, the connection residual between the previous key frame and the current key frame is integrated into the pose graph optimization.

Images