Visual inertial odometer method based on event assistance and related device

Abstract

The embodiment of the invention relates to the field of positioning algorithms, and provides a visual inertial odometer method based on event assistance, and the method comprises the following steps: extracting angular points in an event stream by adopting event stream asynchronous feature tracking, constructing a feature template according to the asynchronous angular points, updating a first feature template according to the event stream, and obtaining a second feature template according to the updated first feature template; obtaining any frame of image in the flight process of the unmanned aerial vehicle, carrying out feature tracking on the image frame according to a second feature template, carrying out feature extraction on the image frame by adopting image feature tracking, setting cubic B-spline state points on state trajectory sampling points, obtaining IMU information collected by an IMU, and calculating a pre-integration quantity corresponding to the IMU information; the first image feature information and the second image feature information are subjected to interpolation processing, the pose information and the point cloud position information are updated according to the motion priori factor, the IMU factor, the vision factor and the preset marginalization priori factor, marginalization processing is performed, and the accuracy of obtaining the flight path of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present application relates to the technical field of positioning algorithms, and in particular to an event-assisted visual inertial odometer method and related devices. Background Art

[0002] Rotary-wing drones have the advantages of simple operation, small size, flexible movement, and controllable cost. They can enter certain complex and dangerous environments to complete many tasks that are difficult for humans to complete. They have shown great application potential in the field of infrastructure inspection. However, due to the obstruction of GPS signals by the reinforced concrete structure of the infrastructure, it is difficult for drones to obtain positioning information. As a common means of positioning drones in indoor scenes, visual inertial odometers can provide continuous and high-precision relative positioning information. However, in infrastructure inspection scenes, due to the obstruction of buildings, there is often a large range of light changes and low light conditions. At the same time, some tasks require drones to fly quickly. The synchronous exposure and low dynamic range of traditional cameras result in low accuracy in obtaining the flight trajectory of drones. Summary of the invention

[0003] The embodiments of the present application provide an event-assisted visual inertial odometer method and related devices, which can improve the accuracy in obtaining the flight trajectory of a drone.

[0004] A first aspect of an embodiment of the present application provides an event-assisted visual inertial odometer method, the method comprising:

[0005] Event stream asynchronous feature tracking is used to extract corner points in the event stream to obtain asynchronous corner points;

[0006] Construct a feature template according to the asynchronous corner points to obtain a first feature template;

[0007] The first feature template is updated according to the event stream to obtain a second feature template;

[0008] Get any frame of image of the drone during flight to obtain the image frame;

[0009] Tracking features of the image frame according to the second feature template to obtain feature information of the first image;

[0010] Using image feature tracking to extract features from the image frame to obtain second image feature information;

[0011] By setting cubic B-spline state points on the state trajectory sampling points, the motion prior factor is obtained;

[0012] Obtain the IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor;

[0013] Performing interpolation processing on the first image feature information and the second image feature information to obtain a visual factor;

[0014] The UAV's posture information and point cloud position information are updated using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first posture information and the first point cloud position information;

[0015] Repeat the steps of setting cubic B-spline state points on the state trajectory sampling points to obtain the motion prior factor, and using the sliding window to update the UAV's pose information and point cloud position information according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first pose information and the first point cloud position information, until the sliding window ends and the UAV's trajectory information is obtained.

[0016] In this example, by using event stream asynchronous feature tracking to extract corner points in the event stream, asynchronous corner points are obtained, feature templates are constructed based on the asynchronous corner points to obtain a first feature template, the first feature template is updated according to the event stream to obtain a second feature template, any frame of the drone's image during flight is obtained to obtain an image frame, feature tracking is performed on the image frame according to the second feature template to obtain first image feature information, image feature tracking is used to extract features from the image frame to obtain second image feature information, a motion prior factor is obtained by setting a cubic B-spline state point on the state trajectory sampling point, the IMU information collected by the IMU is obtained, and the pre-integrated quantity corresponding to the IMU information is calculated to obtain the IMU factor, the first image feature information and the second The image feature information is interpolated to obtain the visual factor, and the UAV's pose information and point cloud position information are updated using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first pose information and the first point cloud position information, and then the steps of setting cubic B-spline state points on the state trajectory sampling points to obtain the motion prior factor, and then using the sliding window to update the UAV's pose information and point cloud position information according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first pose information and the first point cloud position information are repeated until the sliding window ends to obtain the UAV's trajectory information, thereby improving the accuracy in obtaining the UAV's flight trajectory.

[0017] In a possible implementation, a method for updating a first feature template according to an event stream to obtain a second feature template includes:

[0018] Acquire the feature position of the first feature template to obtain the first feature position;

[0019] Get the motion parameters of the drone;

[0020] Calculate the feature position corresponding to the updated feature template according to the motion parameters of the UAV to obtain a second feature position;

[0021] The first feature position is replaced with the second feature position to obtain a second feature template.

[0022] In a possible implementation, a method for tracking features of an image frame according to a second feature template to obtain feature information of a first image includes:

[0023] Acquire the feature position corresponding to the image frame to obtain the third feature position;

[0024] Extract the second feature position corresponding to the second feature template

[0025] Feature matching is performed on the image frame according to the third feature position and the second feature position. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

[0026] In a possible implementation, a method for extracting features from an image frame using image feature tracking to obtain second image feature information includes:

[0027] Image feature tracking is used to extract corner point data from image frames to obtain corner point data;

[0028] Perform optical flow tracking on the image frame to obtain optical flow change information;

[0029] Image feature information of the image frame is determined according to the corner point data and the optical flow change information to obtain second image feature information.

[0030] In a possible implementation, a method for obtaining a visual factor by performing interpolation processing based on first image feature information and second image feature information includes:

[0031] Triangulate the first image feature information and the second image feature information to obtain first point cloud depth information;

[0032] Determine whether the first point cloud depth information belongs to an element within a preset point cloud depth interval. If it does, perform interpolation processing on the first point cloud depth information to obtain a visual factor. If it does not, delete the first point cloud depth information.

[0033] A second aspect of an embodiment of the present application provides an event-assisted visual inertial odometer device, the device comprising:

[0034] An extraction unit, used for extracting corner points from an event stream by using event stream asynchronous feature tracking to obtain asynchronous corner points;

[0035] A construction unit, used for constructing a feature template according to the asynchronous corner points to obtain a first feature template;

[0036] A first updating unit, used for updating the first feature template according to the event stream to obtain a second feature template;

[0037] An acquisition unit is used to acquire any frame of image of the UAV during flight to obtain an image frame;

[0038] A first tracking unit, used for tracking features of the image frame according to the second feature template to obtain first image feature information;

[0039] A second tracking unit is used to extract features from the image frame by using image feature tracking to obtain second image feature information;

[0040] A first calculation unit is used to obtain a motion prior factor by setting a cubic B-spline state point on a state trajectory sampling point;

[0041] The second calculation unit is used to obtain IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor;

[0042] An interpolation unit, used for performing interpolation processing on the first image feature information and the second image feature information to obtain a visual factor;

[0043] The second updating unit is used to update the pose information and point cloud position information of the UAV using a sliding window according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor, so as to obtain the first pose information and the first point cloud position information;

[0044] The loop unit is used to repeatedly execute the steps of obtaining a motion prior factor by setting a cubic B-spline state point on a state trajectory sampling point, updating the pose information and point cloud position information of the UAV using a sliding window according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor, and obtaining the first pose information and the first point cloud position information, until the sliding window ends and the trajectory information of the UAV is obtained.

[0045] In a possible implementation, the first updating unit is specifically configured to:

[0046] Acquire the feature position of the first feature template to obtain the first feature position;

[0047] Get the motion parameters of the drone;

[0048] Calculate the feature position corresponding to the updated feature template according to the motion parameters of the UAV to obtain a second feature position;

[0049] The first feature position is replaced with the second feature position to obtain a second feature template.

[0050] In a possible implementation, the first tracking unit is specifically configured to:

[0051] Acquire the feature position corresponding to the image frame to obtain the third feature position;

[0052] Extract the second feature position corresponding to the second feature template

[0053] Feature matching is performed on the image frame according to the third feature position and the second feature position. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

[0054] In a possible implementation, the second tracking unit is specifically configured to:

[0055] Image feature tracking is used to extract corner point data from image frames to obtain corner point data;

[0056] Perform optical flow tracking on the image frame to obtain optical flow change information;

[0057] Image feature information of the image frame is determined according to the corner point data and the optical flow change information to obtain second image feature information.

[0058] In a possible implementation, the interpolation unit is specifically used for:

[0059] Triangulate the first image feature information and the second image feature information to obtain first point cloud depth information;

[0060] Determine whether the first point cloud depth information belongs to an element within a preset point cloud depth interval. If it does, perform interpolation processing on the first point cloud depth information to obtain a visual factor. If it does not, delete the first point cloud depth information.

[0061] A third aspect of an embodiment of the present application provides a terminal, comprising a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions and execute the step instructions as in the first aspect of the embodiment of the present application.

[0062] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in the first aspect of the embodiments of the present application.

[0063] A fifth aspect of the embodiments of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A flow chart of an event-assisted visual inertial odometer method is provided for an embodiment of the present application;

[0066] Figure 2 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0067] Figure 3 A structural schematic diagram of an event-assisted visual inertial odometer device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0070] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0071] In order to better understand the event-assisted visual inertial odometry method provided in the embodiment of the present application, the visual inertial odometry method in the existing scheme is briefly introduced below. In the existing scheme, feature extraction and tracking are usually performed by using image-like frames constructed in traditional image frames and event streams respectively, and then the information of the two is fused through back-end optimization. However, the high temporal resolution characteristics of the event stream itself are ignored by constructing image-like frames, and the rich motion information is lost, resulting in a loss of precision, which makes the accuracy of obtaining the flight trajectory of the drone low.

[0072] In order to solve the above technical problems, an embodiment of the present application provides an event-assisted visual inertial odometry method, which can obtain the first image feature information and the second image feature information of the image frame through event stream asynchronous feature tracking and image feature tracking, and calculate the motion prior factor, IMU factor, and visual factor according to the first image feature information, the second image information and the IMU information, and then update the posture information and point cloud position information of the drone according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor to obtain the first posture information and the first point cloud position information, thereby improving the accuracy in obtaining the flight trajectory of the drone.

[0073] See also Figure 1 , Figure 1 A flowchart of an event-assisted visual inertial odometer method is provided for an embodiment of the present application. Figure 1 As shown, the method includes:

[0074] 101. Event stream asynchronous feature tracking is used to extract corner points in the event stream to obtain asynchronous corner points.

[0075] The event stream may be acquired by an event camera carried by a drone, and then the corner points in the event stream may be extracted by a general corner point extraction method to obtain asynchronous corner points. Specifically, when the number of feature tracking in the event stream is less than a preset minimum number of features, the general corner point extraction method may be used to extract the corner points in the event stream to obtain asynchronous corner points.

[0076] 102. Construct a feature template according to the asynchronous corner point to obtain a first feature template.

[0077] The first feature template can be obtained by calculating the number of adaptive events in the spatial neighborhood near the asynchronous corner point, and then generating a feature template based on the number of adaptive events in the spatial neighborhood near the asynchronous corner point. Specifically, the number of adaptive events in the spatial neighborhood near the asynchronous corner point can be calculated by a formula, and it can be determined whether the number of adaptive events in the spatial neighborhood near the asynchronous corner point reaches the adaptive event number threshold. If the adaptive event number threshold is reached, a feature template is constructed. The formula used to calculate the number of adaptive events in the spatial neighborhood near the asynchronous corner point is as follows:

[0078]

[0079] In the formula, N represents the number of adaptive events in the spatial neighborhood near the asynchronous corner point; a represents the proportional coefficient, which is determined by user input or system default; Δt represents the time interval; and ΔN0 represents the rate of change of the number of adaptive time.

[0080] 103. Update the first feature template according to the event stream to obtain a second feature template.

[0081] The second feature template may be obtained by acquiring the motion parameters of the drone and updating the first feature template according to the motion parameters of the drone.

[0082] 104. Obtain any frame of image of the UAV during flight to obtain an image frame.

[0083] The image of the drone at any time during the flight can be obtained by using a traditional camera mounted on the drone to obtain an image frame.

[0084] 105. Perform feature tracking on the image frame according to the second feature template to obtain first image feature information.

[0085] The feature position of the second feature template may be matched with the feature position of the image frame, and a RANSAC test may be performed. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

[0086] 106. Use image feature tracking to extract features from the image frame to obtain second image feature information.

[0087] The image feature information may be obtained by performing corner point data extraction and optical flow tracking on the image frame through image feature tracking to obtain corner point data and optical flow change information, and then determining image feature information of the image frame based on the corner point data and optical flow change information to obtain second image feature information.

[0088] 107. The motion prior factor is obtained by setting the cubic B-spline state point on the state trajectory sampling point.

[0089] The method of Lie group Lie algebra can be used to set a cubic B-spline state point on the state trajectory sampling point, and a uniform acceleration prior is added to the cubic B-spline state point to obtain a motion prior factor. Among them, the B-spline state point can be expressed as follows:

[0090]

[0091] EXP in the formula represents the mutual conversion relationship between exponential mappings; b j (u j (t)) represents the constraint relationship between any time t and the local spline control point; Ω i+j-1 represents the local motion, which can be expressed as Log(·) represents the mutual conversion relationship between logarithmic mappings.

[0092] Specifically, the motion prior factor can be obtained by adding a uniform acceleration prior to the cubic B-spline state point through the formula shown below;

[0093]

[0094] The r in the formula prior represents the motion prior factor; represents the cubic B-spline state point corresponding to time i;

[0095] Δt i,i+1 represents the time interval between two adjacent cubic B-spline state points; ξ i,i+1 Represents relative motion parameters, determined by user input; represents the cubic B-spline state point corresponding to the i+1th moment; adj(·) represents the adjoint operation of the matrix; represents the uniform acceleration prior.

[0096] 108. Obtain IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor.

[0097] The IMU information collected by the IMU can be obtained by obtaining the readings uploaded by the IMU carried by the drone, and the pre-integrated quantity corresponding to the IMU information is calculated according to the IMU information collected by the IMU, and the IMU factor is obtained as a relative motion constraint. Among them, the pre-integrated quantity corresponding to the IMU information can be calculated by a general pre-integrated quantity calculation method.

[0098] 109. Perform interpolation processing on the first image feature information and the second image feature information to obtain a visual factor.

[0099] The visual factor may be obtained by initializing the first image feature information and the second image feature information, then calling a triangulation method to obtain the first point cloud depth information, and then performing interpolation and projection processing on the first point cloud depth information.

[0100] 110. The posture information and point cloud position information of the UAV are updated using a sliding window according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor to obtain the first posture information and the first point cloud position information.

[0101] The first pose information and the first point cloud position information can be obtained by updating the UAV's pose information and point cloud position information using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor by the method shown in the following formula:

[0102]

[0103] In the formula Represents the first pose information and the first point cloud position information; r v i sua l Represents visual factor; r imu represents the IMU factor; r pr i or represents the motion prior factor; r mar represents the preset marginalization prior factor, which can be determined by user input or by system default; Q1 represents the covariance matrix corresponding to the visual factor, which is determined by the system by default when the system initializes the parameters; Q2 Indicates the covariance matrix corresponding to the IMU factor, which is determined by the system by default when the system initializes the parameters; Q3 The covariance matrix corresponding to the motion prior factor is determined by the system by default when the system initializes the parameters; Q4 The covariance matrix corresponding to the preset marginalization prior factor is determined by the system by default when the system initializes the parameters.

[0104] 111. Repeat the steps of obtaining a motion prior factor by setting a cubic B-spline state point on the state trajectory sampling point, and updating the UAV's pose information and point cloud position information using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first pose information and the first point cloud position information, until the sliding window ends and the UAV's trajectory information is obtained.

[0105] This can be done by judging whether the first pose information and the first point cloud position information in each window meet the marginalization conditions. If the marginalization conditions are met, the first pose information and the first point cloud position information in the window are marginalized, and the original first pose information and the first point cloud position information in the window are deleted. The execution is repeated until the sliding window ends, and the trajectory information of the drone is obtained.

[0106] In this example, the first image feature information and the second image feature information of the image frame are obtained through event stream asynchronous feature tracking and image feature tracking, and the motion prior factor, IMU factor, and visual factor are calculated according to the first image feature information and the second image information and the IMU information. Then, the pose information and the point cloud position information of the UAV are updated according to the motion prior factor, the IMU factor, the visual factor, and the preset marginalization prior factor to obtain the first pose information and the first point cloud position information. Therefore, feature tracking according to the frequency change of the scene can be output by asynchronously processing the event stream, and the asynchronous and high-frequency characteristics of the event camera can be fully utilized to obtain richer visual motion information. The image frame is used to assist in updating the asynchronous feature tracking, and the event information and the image information are fused at the front end to obtain a more stable asynchronous event stream tracking. Then, the feature tracking of traditional images and the asynchronous event stream tracking are fused with the IMU information, thereby improving the accuracy in obtaining the flight trajectory of the UAV.

[0107] In a possible implementation, a method for updating a first feature template according to an event stream to obtain a second feature template includes:

[0108] A1. Acquire the feature position of the first feature template to obtain the first feature position;

[0109] A2. Obtain the motion parameters of the drone;

[0110] A3. Calculate the feature position corresponding to the updated feature template according to the motion parameters of the UAV to obtain a second feature position;

[0111] A4, using the second feature position to replace the first feature position to obtain a second feature template;

[0112] Specifically, the feature position of the first feature template may be obtained by a general feature position calculation method to obtain the first feature position.

[0113] After obtaining the first feature position, it is also necessary to obtain the translation and rotation speed of the pixel plane to obtain the motion parameters of the drone. The motion parameters of the drone can be expressed as follows:

[0114] p@[ω,v] T

[0115] In the formula, p represents the motion parameter of the drone, ω represents the rotation speed of the pixel plane; v represents the translation speed of the pixel plane; and T represents the transpose of the matrix.

[0116] After obtaining the motion parameters of the drone, it is also necessary to calculate the feature position corresponding to the updated feature template using the method shown in the following formula to obtain the second feature position:

[0117]

[0118] Where p is the motion parameter of the UAV, T′ i is the updated feature template, W(T i , p) is the transformation of the first feature template; K is the number of events used for template alignment.

[0119] After the second feature position is obtained, the first feature position in the first feature template is replaced by the second feature position to obtain the second feature template.

[0120] In this example, the first feature template is updated by the motion parameters of the drone, thereby enriching the richness of the information contained in the first feature template and avoiding the need to calculate the template position based on each event, thereby reducing the computational consumption of the processor and improving the efficiency in obtaining the flight trajectory of the drone.

[0121] In a possible implementation, a method for tracking features of an image frame according to a second feature template to obtain feature information of a first image includes:

[0122] B1. Obtain the feature position corresponding to the image frame to obtain the third feature position;

[0123] B2, extracting the second feature position corresponding to the second feature template;

[0124] B3. Perform feature matching on the image frame according to the third feature position and the second feature position. If the features match, determine the feature position corresponding to the image frame as the first image feature information. If the features do not match, delete the image frame.

[0125] The third feature position may be obtained by extracting corner points of the image frame and then performing KLT tracking on the corner points of the image frame.

[0126] After obtaining the third feature position, it is also necessary to extract the feature position corresponding to the second feature template by a general feature position extraction method to obtain the second feature position.

[0127] After obtaining the second feature position, it is also necessary to perform feature matching on the third feature position and the second feature position corresponding to the second feature template by using RANSAC test. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

[0128] In this example, by performing feature matching on the image frame at the third feature position and the second feature position to obtain the first image feature information, the asynchronous and high-frequency characteristics of the event camera can be fully utilized to obtain richer visual motion information, thereby improving the accuracy of the first image feature information and the accuracy in obtaining the flight trajectory of the drone.

[0129] In a possible implementation, a method for extracting features from an image frame using image feature tracking to obtain second image feature information includes:

[0130] C1. Use image feature tracking to extract corner point data from the image frame to obtain corner point data;

[0131] C2, perform optical flow tracking on the image frame to obtain optical flow change information;

[0132] C3. Determine image feature information of the image frame according to the corner point data and the optical flow change information to obtain second image feature information.

[0133] The corner point data may be obtained by performing traditional Harris corner point extraction on the image frame.

[0134] After obtaining the corner point data, it is also necessary to perform KLT optical flow tracking on the image frame to obtain the optical flow change information.

[0135] Then, the visual SFM of the system is initialized, and the image feature information of the image frame is determined according to the corner point data and the optical flow change information to obtain the second image feature information.

[0136] In this example, corner point data extraction and optical flow tracking are performed on the image frame through image feature tracking to obtain corner point data and optical flow change information, and then the image feature information of the image frame is determined based on the corner point data and optical flow change information to obtain the second image feature information, which can assist in updating asynchronous feature tracking, fuse event information and image information at the front end, obtain more stable asynchronous event stream tracking, and improve the accuracy in obtaining the flight trajectory of the drone.

[0137] In a possible implementation, a method for obtaining a visual factor by performing interpolation processing based on first image feature information and second image feature information includes:

[0138] D1, triangulating the first image feature information and the second image feature information to obtain first point cloud depth information;

[0139] D2. Determine whether the first point cloud depth information belongs to an element within a preset point cloud depth interval. If it does, perform interpolation processing on the first point cloud depth information to obtain a visual factor. If it does not, delete the first point cloud depth information.

[0140] The relative pose information of the drone can be obtained through the first image feature information and the second image feature information, and then a measurement and interpolation pose pair is formed according to the relative pose information of the drone, and the measurement and interpolation pose pair is triangulated by calling a triangulation method to obtain the first point cloud depth information. The measurement and interpolation pose pair can be expressed as follows:

[0141] {m fi ,T wBi}

[0142] In the formula, m fi Represents the relative position information of the UAV; T wBi Represents the interpolation of relative pose information.

[0143] After obtaining the first point cloud depth information, it is necessary to interpolate the first point cloud depth information using the method shown in the following formula to obtain the visual factor:

[0144] r visual =m fi -h(T wBi ,l i )

[0145] In the formula, r visual Represents visual factor; m fi Represents the relative position information of the UAV; T wBi Represents the interpolation of relative pose information; h(T,l) is the projection equation.

[0146] In this example, the first point cloud depth information is obtained through the first image feature information and the second image feature information, and the first point cloud depth information is interpolated, so that the visual constraint conditions of the UAV during the flight can be accurately obtained, thereby improving the accuracy in obtaining the flight trajectory of the UAV.

[0147] In a specific implementation, the embodiment of the present application also provides another risk assessment method for a substation based on artificial intelligence, specifically:

[0148] 1) Start the event stream asynchronous feature tracking thread, denoted as Thread1. When the number of feature tracking is less than the minimum number of features, extract the asynchronous corner point c from the event stream. i =[x i ,y i ,t i ] T The minimum number of features M is not less than 30 to ensure the stable output of the backend optimization results. It can be increased appropriately according to the pixel plane size of the event camera and the detection scene of the application. Since the corner point detection method is not the focus of this invention, it is assumed that the asynchronous corner point set C = {c1, c2, ..., c M}.

[0149] 2) Cache corner point c i Events arriving in the nearby spatial neighborhood (9x9 pixel block range), since the number of events generated is related to the environmental texture and relative motion speed, and the event stream can provide the spatial and temporal distribution of events, this method designs the following formula to calculate the number of adaptive events N. When the number of events in the neighborhood reaches the threshold number N, construct the feature template in For event data.

[0150]

[0151] Where a is the proportionality coefficient, is the average event generation rate when counting N0 events. This formula constrains the relationship between the event window size and the average event generation rate. When the relative motion speed changes greatly, the average event generation rate is large, and the window N is reduced to increase the tracking frequency.

[0152] 3) Update the feature template T according to the event stream i ,Since the information provided by a single event is limited, and the calculation of the template position based on each event is computationally intensive, it is difficult to run in real time on the onboard computer. ,This method adopts when the number of updated events accounts for more than 1 / 2, the original template is transformed and compared with the current template for similarity, alignment optimization, and the updated feature position of the feature template is calculated by formula (2), and the original template is updated to the current template.

[0153]

[0154] Where p is the motion parameter to be estimated, T′ i is the updated current template, W(T i ,p) is the transformation of the original template, and K is the number of events used for template alignment.

[0155] The motion parameter p@[ω,v] used in the present invention Tis the translation and rotation speed of the pixel plane, and it is assumed that the translation and rotation speed are constant in a short time. Under this assumption, the event space-time transformation in the original template is performed and the similarity is compared with the new template to optimize the alignment and solve the motion parameters. First N , select a set of sub-events with temporal consistency, that is, for template T i , T′ i Event Set and Find a subset S that satisfies the approximate equality of timestamp differences i ,Right now:

[0156]

[0157] ε t is the time consistency error, set to 0.01Δ i The above formula can select the corresponding event pairs with the same template time. Secondly, the event subset of formula (3) is transformed as follows:

[0158] W(T i ,p)=R(ωΔ i ) i +vΔ i (4)

[0159] Where R(ωΔ i ) is the two-dimensional plane rotation matrix, u i Template T i Since the event set contained in the template contains noise events, in order to ensure robustness, the present invention only performs noise filtering on the subset S i The events with a proportion of K in the template alignment operation of (2) are performed, and K is set to 0.8. Finally, the optimization solution of (2) is obtained, where the initial value ω,v is set to 0. After that, update the template location to And update the template, repeat (3), (2) to get asynchronous feature tracking, such as Figure 3 shown.

[0160] 4) Start the image feature tracking thread, denoted as Thread2, perform traditional Harris corner point extraction and KLT optical flow tracking, and initialize the visual SFM. There are many methods involved in this part of the technical content, so I will not describe it in detail. Assume that the image feature tracking module can provide stable feature tracking under the condition of good image quality, and initialize the scene point cloud of the drone with unknown posture and scale based on the image frame.

[0161] 5): When the j-th image frame arrives, Thread1 changes the specific data structure of the image frame cache image_buffer to a map structure containing a timestamp and an image pointer, and saves the template-related information to feature_buffer. This data structure is a custom structure that includes the image frame time, template serial number, the template itself, and the feature position. When the j+1-th image arrives, KLT tracking is performed on the cached corner feature position between this image and the j-th image frame to obtain a new feature position. A RANSAC test is performed on the feature matching on the two frames. If it meets the requirements, the feature position of the feature template will be updated, and if it does not meet the requirements, it will be eliminated. After completing the above steps, delete the old image pointer and the image it points to in image_buffer, and delete the old template-related information in feature_buffer.

[0162] After steps 1-5, stable image-assisted asynchronous event stream tracking and image feature tracking can be formed. The feature form is that the next step is to fuse IMU and the above two visual tracking results.

[0163] 6) Start the backend optimization thread, denoted as Thread3, and first initialize the parameters, including the following adaptive optimization output frequency, noise parameters, calibration parameters, etc. The IMU noise parameters come from the IMU static calibration results; the feature tracking noise parameters are independent Gaussian distributions, the image method parameters are set to 1 pixel per measurement, and the asynchronous feature tracking is set to 2 pixels per measurement; the noise parameters of the continuous time state estimation are appropriately adjusted according to the preset smoothness of the motion.

[0164] Since asynchronous feature tracking has a high output tracking measurement frequency when the relative motion changes quickly, if the traditional constant time interval is used to solve the motion state, it is not enough to use the state to reflect the rich measurement, and there is a possibility of accuracy loss and solution divergence. Therefore, this method adopts an adaptive back-end output frequency, which is mainly adjusted appropriately according to the frequency of front-end measurement. The formula is as follows:

[0165]

[0166] in When the number of measurements output by the asynchronous feature tracking front end reaches N fa When the time interval is greater than the minimum time interval, the time interval of the optimization state is set to Otherwise, set to ΔT min .

[0167] 7) Set cubic B-spline state points on the state trajectory sampling points. This method uses the representation method of Lie group Lie algebra, assuming that the position of the Body system relative to the world W system is a value that changes continuously with time, that is:

[0168]

[0169] Then a local continuous-time pose change can be expressed by the nearby B-spline control points as:

[0170]

[0171] Where Exp(·), Log(·) are exponential and logarithmic mappings representing the conversion relationship between se(3) and SE(3); b j (u j (t)) represents the constraint relationship between any time t and the local spline control point, as follows:

[0172]

[0173] in u j (t)@(tt j-1 ) / (t j -t j-1 );The last Ω i+j-1 The local motion is represented by

[0174] Through the state trajectory representation of the above B-spline, the state at any time can be queried through formula (7), which is represented as a combination of three local control points. Although the third-order B-spline ensures the continuity of the second-order derivative of the continuous-time trajectory, common drone flights are mostly based on local uniform acceleration motion. Therefore, this method adds a uniform acceleration prior to the continuous trajectory and constrains the motion through the following prior:

[0175]

[0176] in is the generalized velocity. By taking the derivative of formula (7), we can obtain ξ i,i+1 is the relative se(3) motion, and adj(·) is the adjoint operation of the matrix. In summary, the above formula establishes the relationship between the relative pose increment calculated by uniform acceleration and the increment between the control points themselves, adding a uniform acceleration constraint to the original continuous time trajectory, where the time interval between the control points is determined by formula (5).

[0177] 8) Update imu_buffer according to the IMU message, and calculate the IMU pre-integration amount of adjacent time according to the output time interval as the relative motion constraint. Note that the IMU integration interval is adjusted according to formula (5).

[0178] 9) Receive the asynchronous event stream tracking and image feature tracking results. For the uninitialized features, first query the pose of the feature observation time point through formula (7), and then form the measurement and interpolation pose pair. Then call the triangulation method to obtain the point cloud depth d i , when 0.5<d i When the value is less than 50, the feature is marked as triangulated and initialized successfully, and features with depths exceeding the range are discarded. When the feature is judged to be initialized successfully, the measurement is used to generate an interpolation projection factor, as shown in the following formula, and then added to the factor graph.

[0179]

[0180] in is the interpolated pose and measurement time t i Related, h(T,l) is the projection equation, which projects the spatial point cloud position l to the pixel plane.

[0181] 10) Add the IMU factor, motion prior factor, and marginalization prior factor to the factor graph in turn, perform optimization operations, and update the pose and point cloud position information in the sliding window. The specific optimization problem is:

[0182]

[0183] r visual ,r imu ,r pr i or ,r mar They are respectively visual factor (including asynchronous feature visual factor and image visual factor), IMU factor, motion prior factor, and marginalization prior factor. Q is the covariance matrix, which is obtained through the parameter initialization step.

[0184] After the initialization in step 4 is completed, the scale of the visual initialization is calibrated in combination with the IMU factor integrated in step 8, and the IMU bias and gravity direction are calibrated at the same time. After obtaining the above initial values, any nonlinear least squares optimizer (such as the GTSAM library) can be used to solve equation (11) to obtain the estimated value of T,l, that is, the drone's pose information and environmental point cloud information.

[0185] 11) Determine whether the marginalization condition is met, and directly perform marginalization if it is met, and delete the old state of the sliding window and the corresponding point cloud information. The present invention directly discards the asynchronous feature part measurement during marginalization, and only performs marginalization processing on the image features to ensure real-time performance and simplify the problem after marginalization, and finally repeats 7)-11).

[0186] For the above embodiments, please refer to Figure 2 , Figure 2A schematic diagram of the structure of a terminal provided in an embodiment of the present application, such as Figure 2 As shown, it includes a processor, an input device, an output device and a memory, which are connected to each other, wherein the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the program includes instructions for executing the following steps;

[0187] Event stream asynchronous feature tracking is used to extract corner points in the event stream to obtain asynchronous corner points;

[0188] Construct a feature template according to the asynchronous corner points to obtain a first feature template;

[0189] The first feature template is updated according to the event stream to obtain a second feature template;

[0190] Get any frame of image of the drone during flight to obtain the image frame;

[0191] Tracking features of the image frame according to the second feature template to obtain feature information of the first image;

[0192] Using image feature tracking to extract features from the image frame to obtain second image feature information;

[0193] By setting cubic B-spline state points on the state trajectory sampling points, the motion prior factor is obtained;

[0194] Obtain the IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor;

[0195] Performing interpolation processing on the first image feature information and the second image feature information to obtain a visual factor;

[0196] The UAV's posture information and point cloud position information are updated using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first posture information and the first point cloud position information;

[0197] Repeat the steps of setting cubic B-spline state points on the state trajectory sampling points to obtain the motion prior factor, and using the sliding window to update the UAV's pose information and point cloud position information according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first pose information and the first point cloud position information, until the sliding window ends and the UAV's trajectory information is obtained.

[0198] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that in order to realize the above functions, the terminal includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0199] The embodiment of the present application can divide the terminal into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0200] In line with the above, see Figure 3 , Figure 3 The present invention provides a schematic diagram of a visual inertial odometer device based on event assistance. Figure 3 As shown, the device comprises:

[0201] An extraction unit 301 is used to extract corner points from an event stream by using event stream asynchronous feature tracking to obtain asynchronous corner points;

[0202] A construction unit 302 is used to construct a feature template according to the asynchronous corner points to obtain a first feature template;

[0203] A first updating unit 303, configured to update the first feature template according to the event stream to obtain a second feature template;

[0204] The acquisition unit 304 is used to acquire any frame of image during the flight of the UAV to obtain an image frame;

[0205] A first tracking unit 305 is used to perform feature tracking on the image frame according to the second feature template to obtain first image feature information;

[0206] The second tracking unit 306 is used to extract features from the image frame by using image feature tracking to obtain second image feature information;

[0207] A first calculation unit 307 is used to obtain a motion priori factor by setting a cubic B-spline state point on a state trajectory sampling point;

[0208] The second calculation unit 308 is used to obtain IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor;

[0209] An interpolation unit 309 is used to perform interpolation processing on the first image feature information and the second image feature information to obtain a visual factor;

[0210] The second updating unit 310 is used to update the pose information and point cloud position information of the UAV using a sliding window according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor to obtain the first pose information and the first point cloud position information;

[0211] The loop unit 311 is used to repeatedly execute the steps of obtaining the motion prior factor by setting the cubic B-spline state point on the state trajectory sampling point, updating the UAV's posture information and point cloud position information using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor, and obtaining the first posture information and the first point cloud position information, until the sliding window ends and the trajectory information of the UAV is obtained.

[0212] In a possible implementation, the first updating unit 303 is specifically configured to:

[0213] Acquire the feature position of the first feature template to obtain the first feature position;

[0214] Get the motion parameters of the drone;

[0215] Calculate the feature position corresponding to the updated feature template according to the motion parameters of the UAV to obtain a second feature position;

[0216] The first feature position is replaced with the second feature position to obtain a second feature template.

[0217] In a possible implementation, the first tracking unit 305 is specifically configured to:

[0218] Acquire the feature position corresponding to the image frame to obtain the third feature position;

[0219] Extracting a second feature position corresponding to the second feature template;

[0220] Feature matching is performed on the image frame according to the third feature position and the second feature position. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

[0221] In a possible implementation, the second tracking unit 306 is specifically configured to:

[0222] Image feature tracking is used to extract corner point data from image frames to obtain corner point data;

[0223] Perform optical flow tracking on the image frame to obtain optical flow change information;

[0224] Image feature information of the image frame is determined according to the corner point data and the optical flow change information to obtain second image feature information.

[0225] In a possible implementation, the interpolation unit 309 is specifically configured to:

[0226] Triangulate the first image feature information and the second image feature information to obtain first point cloud depth information;

[0227] Determine whether the first point cloud depth information belongs to an element within a preset point cloud depth interval. If it does, perform interpolation processing on the first point cloud depth information to obtain a visual factor. If it does not, delete the first point cloud depth information.

[0228] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any event-assisted visual-inertial odometry method recorded in the above method embodiments.

[0229] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program enables a computer to execute part or all of the steps of any event-assisted visual inertial odometry method recorded in the above method embodiments.

[0230] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0231] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0232] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0235] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk, etc., and other media that can store program codes.

[0236] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0237] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An event-assisted visual inertial odometry method, characterized in that: Methods include: Event stream asynchronous feature tracking is used to extract corner points in the event stream to obtain asynchronous corner points; Construct a feature template according to the asynchronous corner points to obtain a first feature template; The first feature template is updated according to the event stream to obtain a second feature template; Get any frame of image of the drone during flight to obtain the image frame; Tracking features of the image frame according to the second feature template to obtain feature information of the first image; Extracting features from the image frame using image feature tracking to obtain second image feature information; By setting cubic B-spline state points on the state trajectory sampling points, the motion prior factor is obtained; Obtain the IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor; Performing interpolation processing on the first image feature information and the second image feature information to obtain a visual factor; The UAV's posture information and point cloud position information are updated using a sliding window according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first posture information and the first point cloud position information; Repeat the steps of setting cubic B-spline state points on the state trajectory sampling points to obtain the motion prior factor, and using the sliding window to update the UAV's pose information and point cloud position information according to the motion prior factor, IMU factor, visual factor and preset marginalization prior factor to obtain the first pose information and the first point cloud position information, until the sliding window ends and the UAV's trajectory information is obtained.

2. The event-assisted visual inertial odometry method according to claim 1, characterized in that: The first feature template is updated according to the event stream to obtain a second feature template, including: Acquire the feature position of the first feature template to obtain the first feature position; Get the motion parameters of the drone; Calculate the feature position corresponding to the updated feature template according to the motion parameters of the UAV to obtain a second feature position; The first feature position is replaced with the second feature position to obtain a second feature template.

3. The event-assisted visual inertial odometry method according to claim 2, characterized in that: Tracking the features of the image frame according to the second feature template to obtain the first image feature information includes: Acquire the feature position corresponding to the image frame to obtain the third feature position; Extracting a second feature position corresponding to the second feature template; Feature matching is performed on the image frame according to the third feature position and the second feature position. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

4. The event-assisted visual inertial odometry method according to claim 3, characterized in that: Image feature tracking is used to extract features from the image frame to obtain second image feature information, including: Image feature tracking is used to extract corner point data from image frames to obtain corner point data; Perform optical flow tracking on the image frame to obtain optical flow change information; Image feature information of the image frame is determined according to the corner point data and the optical flow change information to obtain second image feature information.

5. The event-assisted visual inertial odometry method according to claim 4, characterized in that: Interpolation processing is performed according to the first image feature information and the second image feature information to obtain a visual factor, including: Triangulate the first image feature information and the second image feature information to obtain first point cloud depth information; Determine whether the first point cloud depth information belongs to an element within a preset point cloud depth interval. If it does, perform interpolation processing on the first point cloud depth information to obtain a visual factor. If it does not, delete the first point cloud depth information.

6. An event-assisted visual inertial odometer device, characterized in that: The device includes: An extraction unit, used for extracting corner points from an event stream by using event stream asynchronous feature tracking to obtain asynchronous corner points; A construction unit, used for constructing a feature template according to the asynchronous corner points to obtain a first feature template; A first updating unit, used for updating the first feature template according to the event stream to obtain a second feature template; An acquisition unit is used to acquire any frame of image of the UAV during flight to obtain an image frame; A first tracking unit, used for tracking features of the image frame according to the second feature template to obtain first image feature information; A second tracking unit is used to extract features from the image frame by using image feature tracking to obtain second image feature information; A first calculation unit is used to obtain a motion prior factor by setting a cubic B-spline state point on a state trajectory sampling point; The second calculation unit is used to obtain IMU information collected by the IMU, and calculate the pre-integrated quantity corresponding to the IMU information to obtain the IMU factor; An interpolation unit, used for performing interpolation processing on the first image feature information and the second image feature information to obtain a visual factor; The second updating unit is used to update the pose information and point cloud position information of the UAV using a sliding window according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor, so as to obtain the first pose information and the first point cloud position information; The loop unit is used to repeatedly execute the steps of obtaining a motion prior factor by setting a cubic B-spline state point on a state trajectory sampling point, updating the pose information and point cloud position information of the UAV using a sliding window according to the motion prior factor, the IMU factor, the visual factor and the preset marginalization prior factor, and obtaining the first pose information and the first point cloud position information, until the sliding window ends and the trajectory information of the UAV is obtained.

7. The event-assisted visual inertial odometer device according to claim 6, characterized in that: The first updating unit is specifically used for: Acquire the feature position of the first feature template to obtain the first feature position; Get the motion parameters of the drone; Calculate the feature position corresponding to the updated feature template according to the motion parameters of the UAV to obtain a second feature position; The first feature position is replaced with the second feature position to obtain a second feature template.

8. The event-assisted visual inertial odometer device according to claim 6, characterized in that: The first tracking unit is specifically used for: Acquire the feature position corresponding to the image frame to obtain the third feature position; Extract the second feature position corresponding to the second feature template Feature matching is performed on the image frame according to the third feature position and the second feature position. If the features match, the feature position corresponding to the image frame is determined as the first image feature information. If the features do not match, the image frame is deleted.

9. A terminal, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the event-assisted visual inertial odometer method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the event-assisted visual-inertial odometry method according to any one of claims 1 to 5.

Images